From skyleach at gmail.com  Mon Aug  1 11:50:38 2016
From: skyleach at gmail.com (Matt Gregory)
Date: Mon, 1 Aug 2016 11:50:38 -0400
Subject: [Python-Dev] A question concerning named pipes,
 multiprocessing.Queue (multiprocessing.queues.JoinableQueue)
Message-ID: <9a58a65a-0600-7f60-16b3-fff08379b8ee@gmail.com>

I raised this issue and question on StackExchange and #python (FreeNode) 
and have received little or no feedback.  I fear that the only answer 
will lie in profiling the python interpreter itself, which is beyond the 
scope of my capabilities at present.

The original question can be found here: 
http://stackoverflow.com/questions/38637282/multiprocessing-queue-seems-to-go-away-os-pipe-destruction-vs-python

Cliffs: after about one to two hours of processing, the consumer process 
reading the queue terminates on timeout during queue.get(). The thread 
writing objects to the queue receives no exception continues writing to 
the queue.  The thread keeps track of each child process using a simple 
local list of processes, and each time an object is added to the queue, 
the processes objects (consumers) are checked with .is_alive().  When 
the child processes suicide on timeout, is_alive() continues to return 
True, so they are not garbage collected nor are new processes started.

I sincerely apologize if my own understanding of the documentation is at 
fault, but to the best of my knowledge this code should work according 
to the documentation.

From mdione at grulic.org.ar  Wed Aug  3 13:23:12 2016
From: mdione at grulic.org.ar (Marcos Dione)
Date: Wed, 3 Aug 2016 19:23:12 +0200
Subject: [Python-Dev] stuck issue 26826
Message-ID: <20160803172312.cmnao5y6r35gr4se@diablo.grulicueva.local>


    Hi pythonistas. A couple of moths ago I opened an issue in the bug
tracker for adding a new syscall to the os module. It's based on new
developments in the Linux kernel. Here's the link:

https://bugs.python.org/issue26826

    After two months and a half I managed to create a nice patch with
unit tests and documentation (yay!), but then the issue went cold. I
would like to know how to proceed to get the issue going again.

    Cheers,

	-- Marcos.

-- 
(Not so) Random fortune:
Terrorism isn't a crime against people or property. It's a crime against
our minds, using the death of innocents and destruction of property to
make us fearful.
	    -- Bruce Schneier

From guido at python.org  Wed Aug  3 13:46:13 2016
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Aug 2016 10:46:13 -0700
Subject: [Python-Dev] stuck issue 26826
In-Reply-To: <20160803172312.cmnao5y6r35gr4se@diablo.grulicueva.local>
References: <20160803172312.cmnao5y6r35gr4se@diablo.grulicueva.local>
Message-ID: <CAP7+vJJh2b9rRY=nzgcBh=-qwiAuQj+DF26hAw8-3d9SRUf5Jw@mail.gmail.com>

I wonder if the issue isn't that there are so many Linux syscalls that
we probably should have a process for deciding which ones are worth
supporting in the os module, and that process should not necessarily
start with a patch review. What fraction of Linux syscalls do we
currently support? What fraction of BSD syscalls? How much of this is
better served by a 3rd party module? Certainly it's not rocket science
to write a C extension module that wraps a syscall or a bunch of them.

On Wed, Aug 3, 2016 at 10:23 AM, Marcos Dione <mdione at grulic.org.ar> wrote:
>
>     Hi pythonistas. A couple of moths ago I opened an issue in the bug
> tracker for adding a new syscall to the os module. It's based on new
> developments in the Linux kernel. Here's the link:
>
> https://bugs.python.org/issue26826
>
>     After two months and a half I managed to create a nice patch with
> unit tests and documentation (yay!), but then the issue went cold. I
> would like to know how to proceed to get the issue going again.
>
>     Cheers,
>
>         -- Marcos.
>
> --
> (Not so) Random fortune:
> Terrorism isn't a crime against people or property. It's a crime against
> our minds, using the death of innocents and destruction of property to
> make us fearful.
>             -- Bruce Schneier
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From mdione at grulic.org.ar  Wed Aug  3 14:16:17 2016
From: mdione at grulic.org.ar (Marcos Dione)
Date: Wed, 3 Aug 2016 20:16:17 +0200
Subject: [Python-Dev] stuck issue 26826
In-Reply-To: <CAP7+vJJh2b9rRY=nzgcBh=-qwiAuQj+DF26hAw8-3d9SRUf5Jw@mail.gmail.com>
Message-ID: <20160803181617.zg2jwt6pnca5zg46@diablo.grulicueva.local>

On Wed, Aug 03, 2016 at 10:46:13AM -0700, Guido van Rossum wrote:
> I wonder if the issue isn't that there are so many Linux syscalls that
> we probably should have a process for deciding which ones are worth
> supporting in the os module, and that process should not necessarily
> start with a patch review. [...] Certainly it's not rocket science
> to write a C extension module that wraps a syscall or a bunch of them.

    I agree, but also notice that some of these syscalls, specially those
which are optimizations for certain situations like this one or
sendfile(), could also be used by the rest of python's core modules if
they're available. In this case in particular, it could be used to speed
up copyfile(), copy(), copy2() and probably copytree() from the shutil
module. In fact, if this patch goes in, I'm planning to implement such
optimizations.

-- 
(Not so) Random fortune:
Terrorism isn't a crime against people or property. It's a crime against
our minds, using the death of innocents and destruction of property to
make us fearful.
	    -- Bruce Schneier

From guido at python.org  Wed Aug  3 14:31:46 2016
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Aug 2016 11:31:46 -0700
Subject: [Python-Dev] stuck issue 26826
In-Reply-To: <20160803181617.zg2jwt6pnca5zg46@diablo.grulicueva.local>
References: <CAP7+vJJh2b9rRY=nzgcBh=-qwiAuQj+DF26hAw8-3d9SRUf5Jw@mail.gmail.com>
 <20160803181617.zg2jwt6pnca5zg46@diablo.grulicueva.local>
Message-ID: <CAP7+vJ+Xy5XaVw_4eH62qOd2yVnJSqyBpQzK-pJoW9yLKTJ3zQ@mail.gmail.com>

Then again are people really concerned about the speed of those file
copy functions? Or are we just offering a solution in search of a
problem?

(I honestly don't know. At Dropbox we don't use Python for scripting
much, we use it to write dynamic web servers. Static files are served
by a CDN so e.g. sendfile() is not interesting to us either.)

On Wed, Aug 3, 2016 at 11:16 AM, Marcos Dione <mdione at grulic.org.ar> wrote:
> On Wed, Aug 03, 2016 at 10:46:13AM -0700, Guido van Rossum wrote:
>> I wonder if the issue isn't that there are so many Linux syscalls that
>> we probably should have a process for deciding which ones are worth
>> supporting in the os module, and that process should not necessarily
>> start with a patch review. [...] Certainly it's not rocket science
>> to write a C extension module that wraps a syscall or a bunch of them.
>
>     I agree, but also notice that some of these syscalls, specially those
> which are optimizations for certain situations like this one or
> sendfile(), could also be used by the rest of python's core modules if
> they're available. In this case in particular, it could be used to speed
> up copyfile(), copy(), copy2() and probably copytree() from the shutil
> module. In fact, if this patch goes in, I'm planning to implement such
> optimizations.
>
> --
> (Not so) Random fortune:
> Terrorism isn't a crime against people or property. It's a crime against
> our minds, using the death of innocents and destruction of property to
> make us fearful.
>             -- Bruce Schneier



-- 
--Guido van Rossum (python.org/~guido)

From mdione at grulic.org.ar  Wed Aug  3 16:32:55 2016
From: mdione at grulic.org.ar (Marcos Dione)
Date: Wed, 3 Aug 2016 22:32:55 +0200
Subject: [Python-Dev] stuck issue 26826
In-Reply-To: <CAP7+vJ+Xy5XaVw_4eH62qOd2yVnJSqyBpQzK-pJoW9yLKTJ3zQ@mail.gmail.com>
Message-ID: <20160803203255.wcf6h42v364pjgqw@diablo.grulicueva.local>

On Wed, Aug 03, 2016 at 11:31:46AM -0700, Guido van Rossum wrote:
> Then again are people really concerned about the speed of those file
> copy functions? Or are we just offering a solution in search of a
> problem?

    At kernel level: clearly yes, otherwise their BDFL would noy allow
those[1] patches to go in.

    Now, should/could Python benefit from them? I personally think so,
that it should. That's why I developed the patch in the first place. As
for the cost, in terms of maintainability, I just noticed that copy() and
copy2() use copyfile(), and that copytree() uses copy2(), so only one
function should be modified. True, the code will be ~50% more complex
(it needs to check the availability of the function and the suitability
for the parameters given; copy_file_range() only works on files on the
same filesystem[1]). Hmm...

    Maybe you're right. Maybe, to keep Python's own code simple, we could
skip these optimizations, and leave them in a 3rd party module.

--
[1] I still don't understand why all these optimizations are exposed
separately for the specific cases in which they work; I would expect one
function that would take care of the details, but at least provide copy
functionality without using user space buffers but kernel ones. But then,
I'm not a kernel developer, just a puny Python one...
-- 
(Not so) Random fortune:
Terrorism isn't a crime against people or property. It's a crime against
our minds, using the death of innocents and destruction of property to
make us fearful.
	    -- Bruce Schneier

From tjreedy at udel.edu  Wed Aug  3 16:47:50 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 3 Aug 2016 16:47:50 -0400
Subject: [Python-Dev] stuck issue 26826
In-Reply-To: <20160803172312.cmnao5y6r35gr4se@diablo.grulicueva.local>
References: <20160803172312.cmnao5y6r35gr4se@diablo.grulicueva.local>
Message-ID: <nntl9v$omo$1@blaine.gmane.org>

On 8/3/2016 1:23 PM, Marcos Dione wrote:
>
>     Hi pythonistas. A couple of moths ago I opened an issue in the bug
> tracker for adding a new syscall to the os module. It's based on new
> developments in the Linux kernel. Here's the link:
>
> https://bugs.python.org/issue26826

I suggest that at some point you summarized Guido's questions and any 
other discussion here on the issue for the benefit of people not reading 
this thread.  Adding a link to the thread (in the archive) would be even 
better.

-- 
Terry Jan Reedy


From random832 at fastmail.com  Wed Aug  3 17:02:17 2016
From: random832 at fastmail.com (Random832)
Date: Wed, 03 Aug 2016 17:02:17 -0400
Subject: [Python-Dev] stuck issue 26826
In-Reply-To: <20160803203255.wcf6h42v364pjgqw@diablo.grulicueva.local>
References: <20160803203255.wcf6h42v364pjgqw@diablo.grulicueva.local>
Message-ID: <1470258137.2767455.685283505.70998478@webmail.messagingengine.com>

On Wed, Aug 3, 2016, at 16:32, Marcos Dione wrote:
> (it needs to check the availability of the function and the suitability
> for the parameters given; copy_file_range() only works on files on the
> same filesystem[1]). Hmm...

What is the benefit to using copy_file_range over sendfile in this
scenario? Or does sendfile not work with regular files on Linux?

Maybe os.sendfile should use copy_file_range if available/applicable,
and the shutils functions can use it?

From vadmium at gmail.com  Wed Aug  3 19:56:10 2016
From: vadmium at gmail.com (Martin Panter)
Date: Wed, 3 Aug 2016 23:56:10 +0000
Subject: [Python-Dev] stuck issue 26826
In-Reply-To: <20160803172312.cmnao5y6r35gr4se@diablo.grulicueva.local>
References: <20160803172312.cmnao5y6r35gr4se@diablo.grulicueva.local>
Message-ID: <CA+eR4cFtjTTiDJj4iV5nTH=LFvskeywRx20ftCY1+ROkO1RuSg@mail.gmail.com>

On 03/08/2016, Marcos Dione <mdione at grulic.org.ar> wrote:
>     Hi pythonistas. A couple of moths ago I opened an issue in the bug
> tracker for adding a new syscall to the os module. It's based on new
> developments in the Linux kernel. Here's the link:
>
> https://bugs.python.org/issue26826

To give more context, this is about adding support for Linux?s
copy_file_range() system call.

>     After two months and a half I managed to create a nice patch with
> unit tests and documentation (yay!), but then the issue went cold. I
> would like to know how to proceed to get the issue going again.

I thought the main problem remaining was getting concensus about
adding it, since a couple people mentioned waiting for glibc support.
I don?t have much personal opinion on this, but FWIW I don?t see much
disadvantage in adding it now.

I did mean to look over your latest patch, but that has drifted
towards the bottom of my list of things to do :)

From alexander.belopolsky at gmail.com  Wed Aug  3 20:24:13 2016
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 3 Aug 2016 20:24:13 -0400
Subject: [Python-Dev] Method signatures in the datetime module
 documentation
In-Reply-To: <CAP7h-xagLQ2nQerPasc+uOaXzJ=NiTU+dWmK=fGG+BSc82amUw@mail.gmail.com>
References: <CAP7h-xagLQ2nQerPasc+uOaXzJ=NiTU+dWmK=fGG+BSc82amUw@mail.gmail.com>
Message-ID: <CAP7h-xaNNF85wnT4aQryVzH9+Nzjs2R1YShPAxAs4-EF5irPvQ@mail.gmail.com>

On Fri, Jul 29, 2016 at 12:55 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
>
> How should I present the signature of the new replace method in the
> documentation?

Having not seeing any replies, let me make a proposal:

datetime.replace(hour=self.hour, minute=self.minute, second=self.second,
    microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=self.fold)

while not a valid method signature, this seems to correctly reflect
what replace() does.

From vadmium+py at gmail.com  Wed Aug  3 22:41:56 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Thu, 4 Aug 2016 02:41:56 +0000
Subject: [Python-Dev] Method signatures in the datetime module
 documentation
In-Reply-To: <CAP7h-xaNNF85wnT4aQryVzH9+Nzjs2R1YShPAxAs4-EF5irPvQ@mail.gmail.com>
References: <CAP7h-xagLQ2nQerPasc+uOaXzJ=NiTU+dWmK=fGG+BSc82amUw@mail.gmail.com>
 <CAP7h-xaNNF85wnT4aQryVzH9+Nzjs2R1YShPAxAs4-EF5irPvQ@mail.gmail.com>
Message-ID: <CA+eR4cGtHuQC1XJ0dJk+K1wRmfuj5N0t9igaKDkWRJ6X8XWUrQ@mail.gmail.com>

On 4 August 2016 at 00:24, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Fri, Jul 29, 2016 at 12:55 PM, Alexander Belopolsky
> <alexander.belopolsky at gmail.com> wrote:
>>
>> How should I present the signature of the new replace method in the
>> documentation?
>
> Having not seeing any replies, let me make a proposal:
>
> datetime.replace(hour=self.hour, minute=self.minute, second=self.second,
>     microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=self.fold)
>
> while not a valid method signature, this seems to correctly reflect
> what replace() does.

Does that mean replace() accepts most of its arguments as either
positional or keyword arguments? IMO when the documentation uses the
?name=default? notation, it implies keyword argument support, while
the [name] notation implies only supporting a positional argument. If
replace() actually supported these keywords all along, then I am happy
with this proposal.

There are a few bug reports that may be relevant:

https://bugs.python.org/issue13386: notation where there is no simple
default value
https://bugs.python.org/issue23738: notation for defaults for
positional-only parameters

From srkunze at mail.de  Thu Aug  4 05:07:32 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Thu, 4 Aug 2016 11:07:32 +0200
Subject: [Python-Dev] The devguide is now hosted on GitHub
In-Reply-To: <CAP1=2W4tOSd9LiwkkGMiVZAc2yoYQjNNtaKTiTvPdRscKCKWpA@mail.gmail.com>
References: <CAP1=2W4tOSd9LiwkkGMiVZAc2yoYQjNNtaKTiTvPdRscKCKWpA@mail.gmail.com>
Message-ID: <eba466a3-1e41-75e4-e0c9-b28233c94fd5@mail.de>

Thanks a lot. :)


On 22.07.2016 22:04, Brett Cannon wrote:
> https://github.com/python/devguide
>
> I have also moved all issues over as well and hooked up Read The Docs 
> so that there's a mirror which is always up-to-date (vs. 
> docs.python.org/devguide <http://docs.python.org/devguide> which is on 
> a cronjob).
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160804/7e33aa0c/attachment.html>

From ncoghlan at gmail.com  Thu Aug  4 08:14:49 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 4 Aug 2016 22:14:49 +1000
Subject: [Python-Dev] stuck issue 26826
In-Reply-To: <20160803203255.wcf6h42v364pjgqw@diablo.grulicueva.local>
References: <CAP7+vJ+Xy5XaVw_4eH62qOd2yVnJSqyBpQzK-pJoW9yLKTJ3zQ@mail.gmail.com>
 <20160803203255.wcf6h42v364pjgqw@diablo.grulicueva.local>
Message-ID: <CADiSq7egfjWJ7MWnAWBbZcL-WoibFytWxdSsd-gM7_hTadM2nA@mail.gmail.com>

On 4 August 2016 at 06:32, Marcos Dione <mdione at grulic.org.ar> wrote:
>     Maybe you're right. Maybe, to keep Python's own code simple, we could
> skip these optimizations, and leave them in a 3rd party module.

Having the scandir package on PyPI made it possible for folks to
quantify the benefits of the new os.scandir() call for different
workloads before we committed to adding it to the standard library. It
also had the benefit of allowing folks to achieve the speedups by
installing the library and changing their code if that was easier for
them than waiting for a new release of CPython and upgrading to it
(e.g. that's common for library authors that focus on Linux can often
require an extra C dependency without too much hassle, but also
frequently target the older versions of Python shipped by long term
support distros).

It seems to me that a dedicated "os_linux" addon module on PyPI could
serve a dual purpose in making updated os module APIs available on
older versions of Python, as well as in providing a venue for folks to
experiment with the performance of new syscalls before proposing them
for stdlib inclusion.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From alexander.belopolsky at gmail.com  Thu Aug  4 12:11:47 2016
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 4 Aug 2016 12:11:47 -0400
Subject: [Python-Dev] Status of the Argument Clinic DSL
Message-ID: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>

What is the current status of the Argument Clinic DSL?  The clinic
preprocessor was released [1] with Python 3.4 with a promise [2] "that
signature metadata for programmatic introspection will be added to
additional callables implemented in C as part of Python 3.4
maintenance releases."  We are now close to 3.6 beta and clinic
coverage in stdlib is sparse at best.  For example, the math module
makes no use of the clinic (but interestingly a few methods in cmath
do).  The documentation is very sparse as well.  While one would
expect PEP 436 to serve as *the* documentation while clinic is being
developed, but the PEP has the "Draft" status and has not been updated
since 2013.  Furthermore, the 3.4 release notes contain a warning
saying "The Argument Clinic PEP is not fully up to date with the state
of the implementation."  It does not look like this situation has
improved since.

The practical question is: does it make sense to invest time in
learning the clinic DSL and converting more modules (e.g. datetime) to
it or it is likely that the whole approach will be revised in the
future?

[1] http://bugs.python.org/issue16612
[2] https://docs.python.org/3.6/whatsnew/3.4.html?highlight=clinic#pep-436-argument-clinic

From alexander.belopolsky at gmail.com  Thu Aug  4 13:12:27 2016
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 4 Aug 2016 13:12:27 -0400
Subject: [Python-Dev] Method signatures in the datetime module
 documentation
In-Reply-To: <CA+eR4cGtHuQC1XJ0dJk+K1wRmfuj5N0t9igaKDkWRJ6X8XWUrQ@mail.gmail.com>
References: <CAP7h-xagLQ2nQerPasc+uOaXzJ=NiTU+dWmK=fGG+BSc82amUw@mail.gmail.com>
 <CAP7h-xaNNF85wnT4aQryVzH9+Nzjs2R1YShPAxAs4-EF5irPvQ@mail.gmail.com>
 <CA+eR4cGtHuQC1XJ0dJk+K1wRmfuj5N0t9igaKDkWRJ6X8XWUrQ@mail.gmail.com>
Message-ID: <CAP7h-xaHnZ=cU2witQoQ23BMjcLGtToL_u4-T-cnvdMSKvU0Vw@mail.gmail.com>

On Wed, Aug 3, 2016 at 10:41 PM, Martin Panter <vadmium+py at gmail.com> wrote:
> If replace() actually supported these keywords all along, then I am happy
> with this proposal.
>

With the obvious exception of "fold", it did at least since Python 3.0
(and likely since 2.x where x is 5 ? 1.)

> There are a few bug reports that may be relevant:
>
> https://bugs.python.org/issue13386: notation where there is no simple
> default value
> https://bugs.python.org/issue23738: notation for defaults for
> positional-only parameters

Thanks.  These are helpful, but they don't address the case in hand
where the member function defaults are taken from the other
attributes.

From brett at python.org  Thu Aug  4 14:09:27 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 04 Aug 2016 18:09:27 +0000
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
Message-ID: <CAP1=2W6agthi8UNWg14gm9ok_1jr64xD2BJWpQee7H8xf4O0+g@mail.gmail.com>

On Thu, 4 Aug 2016 at 09:14 Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

> What is the current status of the Argument Clinic DSL?  The clinic
> preprocessor was released [1] with Python 3.4 with a promise [2] "that
> signature metadata for programmatic introspection will be added to
> additional callables implemented in C as part of Python 3.4
> maintenance releases."  We are now close to 3.6 beta and clinic
> coverage in stdlib is sparse at best.  For example, the math module
> makes no use of the clinic (but interestingly a few methods in cmath
> do).  The documentation is very sparse as well.  While one would
> expect PEP 436 to serve as *the* documentation while clinic is being
> developed, but the PEP has the "Draft" status and has not been updated
> since 2013.  Furthermore, the 3.4 release notes contain a warning
> saying "The Argument Clinic PEP is not fully up to date with the state
> of the implementation."  It does not look like this situation has
> improved since.
>

I always read https://docs.python.org/3/howto/clinic.html as the
documentation and not the PEP.


>
> The practical question is: does it make sense to invest time in
> learning the clinic DSL and converting more modules (e.g. datetime) to
> it


Yes. Part of the issue is there was a big push to convert modules for the
3.4 release and some of us helped convert however many we can, but then
people stopped putting the time in to convert code a module at a time once
3.4 was released. Serhiy has actually been working on a patch to speed up
keyword parsing which will automatically be applied to code using Argument
Clinic: http://bugs.python.org/issue27574 .


> or it is likely that the whole approach will be revised in the
> future?
>
>
I don't see it being revised, but maybe Larry has something to say on the
matter.


> [1] http://bugs.python.org/issue16612
> [2]
> https://docs.python.org/3.6/whatsnew/3.4.html?highlight=clinic#pep-436-argument-clinic
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160804/cb77a5b9/attachment.html>

From larry at hastings.org  Thu Aug  4 14:19:50 2016
From: larry at hastings.org (Larry Hastings)
Date: Thu, 4 Aug 2016 11:19:50 -0700
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
Message-ID: <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>



On 08/04/2016 09:11 AM, Alexander Belopolsky wrote:
> Furthermore, the 3.4 release notes contain a warning
> saying "The Argument Clinic PEP is not fully up to date with the state
> of the implementation."  It does not look like this situation has
> improved since.

It hasn't.  The PEP lags behind the implementation.  The HOWTO is 
up-to-date though.


> The practical question is: does it make sense to invest time in
> learning the clinic DSL and converting more modules (e.g. datetime) to
> it or it is likely that the whole approach will be revised in the
> future?

AFAIK the Clinic DSL can handle all of Python's C extensions.  I have no 
plans to "revise the whole approach"; if someone else does I haven't 
heard about it.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160804/84c75a5c/attachment.html>

From alexander.belopolsky at gmail.com  Thu Aug  4 18:45:39 2016
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 4 Aug 2016 18:45:39 -0400
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
 <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>
Message-ID: <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>

On Thu, Aug 4, 2016 at 2:19 PM, Larry Hastings <larry at hastings.org> wrote:

> AFAIK the Clinic DSL can handle all of Python's C extensions.  I have no
> plans to "revise the whole approach"; if someone else does I haven't heard
> about it.


I was just wondering that with so much effort to bring typing to the
mainstream, a more "pythonic" DSL may emerge for describing the signatures
of functions in C modules.

BTW, is there any document describing the syntax of "text signatures" such
as:

>>> os.rename.__text_signature__
'($module, /, src, dst, *, src_dir_fd=None, dst_dir_fd=None)'

?

What does the "$module, /," part mean?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160804/db1d8e96/attachment.html>

From alexander.belopolsky at gmail.com  Thu Aug  4 19:04:53 2016
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 4 Aug 2016 19:04:53 -0400
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <CAP1=2W7uJ4xaE3O+454Y2ZJTEP77dACC0QwEW+302BcWrMsHFg@mail.gmail.com>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
 <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>
 <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>
 <CAP1=2W7uJ4xaE3O+454Y2ZJTEP77dACC0QwEW+302BcWrMsHFg@mail.gmail.com>
Message-ID: <CAP7h-xaSXCcWKUBTSf_4ZhYR_24+SMgxm+ao4OFbfVrsEHi-4Q@mail.gmail.com>

On Thu, Aug 4, 2016 at 7:00 PM, Brett Cannon <brett at python.org> wrote:
>> >>> os.rename.__text_signature__
>> '($module, /, src, dst, *, src_dir_fd=None, dst_dir_fd=None)'
>>
>> ?
>>
>> What does the "$module, /," part mean?
>
>
> Don't remember what the $module means but the / means positional-only
> arguments.

Not in this case.  Apparently the / applies to the $module thingy
because src and dst are not positional only (and the / would be after
them if they were, I think.)

From brett at python.org  Thu Aug  4 19:00:09 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 04 Aug 2016 23:00:09 +0000
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
 <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>
 <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>
Message-ID: <CAP1=2W7uJ4xaE3O+454Y2ZJTEP77dACC0QwEW+302BcWrMsHFg@mail.gmail.com>

On Thu, 4 Aug 2016 at 15:47 Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Thu, Aug 4, 2016 at 2:19 PM, Larry Hastings <larry at hastings.org> wrote:
>
>> AFAIK the Clinic DSL can handle all of Python's C extensions.  I have no
>> plans to "revise the whole approach"; if someone else does I haven't heard
>> about it.
>
>
> I was just wondering that with so much effort to bring typing to the
> mainstream, a more "pythonic" DSL may emerge for describing the signatures
> of functions in C modules.
>
> BTW, is there any document describing the syntax of "text signatures" such
> as:
>
> >>> os.rename.__text_signature__
> '($module, /, src, dst, *, src_dir_fd=None, dst_dir_fd=None)'
>
> ?
>
> What does the "$module, /," part mean?
>

Don't remember what the $module means but the / means positional-only
arguments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160804/fc904617/attachment.html>

From larry at hastings.org  Thu Aug  4 19:12:02 2016
From: larry at hastings.org (Larry Hastings)
Date: Thu, 4 Aug 2016 16:12:02 -0700
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
 <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>
 <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>
Message-ID: <1ded45e5-c77b-fe23-8f67-4f4e29f8ba08@hastings.org>

On 08/04/2016 03:45 PM, Alexander Belopolsky wrote:
>
> On Thu, Aug 4, 2016 at 2:19 PM, Larry Hastings <larry at hastings.org 
> <mailto:larry at hastings.org>> wrote:
>
>     AFAIK the Clinic DSL can handle all of Python's C extensions.  I
>     have no plans to "revise the whole approach"; if someone else does
>     I haven't heard about it.
>
>
> I was just wondering that with so much effort to bring typing to the 
> mainstream, a more "pythonic" DSL may emerge for describing the 
> signatures of functions in C modules.
>
> BTW, is there any document describing the syntax of "text signatures" 
> such as:
>
> >>> os.rename.__text_signature__
> '($module, /, src, dst, *, src_dir_fd=None, dst_dir_fd=None)'
>
> ?
>
> What does the "$module, /," part mean?


/ is the delimiter between positional-only parameters and 
positional-or-keyword arguments.  It's not actual Python syntax, but 
Guido said (somewhere) that *if* Python ever sprouted a syntax for 
positional-only parameters, that was as good a delimiter as any.  I 
think he picked it because / is the inverse of *.

The "$" in "$module" means it's what we called a "bound parameter", a 
parameter which gets bound to a value before Python ever sees it.  C 
extension functions get the module passed in automatically, but this is 
done internally and from the Python level you can't see it.  So it's 
accurate to present it there, but we suppress it before we compute the 
inspect.signature.  For example, os_chdir_impl in Modules/posixmodule.c 
takes two arguments, the first being the module, the second being the 
path; inspect.signature(os.chdir) only shows one parameter, the path.

__text_signature__, although user-visible, is not considered information 
for users.  It's an internal implementation detail like co_code.  
However, with the exception of things like "$module" and "/" it's just a 
Python function declaration. Literally we remove the funny bits like 
"$module" and "/", prepend that string with "def foo", append that 
string with ": pass", and parse the result with ast.parse.

If you have more questions about __text_signature__, I recommend reading 
the implementation of inspect.signature, since that's the one and only 
consumer of it.


//arry/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160804/d8f7d001/attachment.html>

From alexander.belopolsky at gmail.com  Thu Aug  4 19:33:30 2016
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 4 Aug 2016 19:33:30 -0400
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <1ded45e5-c77b-fe23-8f67-4f4e29f8ba08@hastings.org>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
 <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>
 <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>
 <1ded45e5-c77b-fe23-8f67-4f4e29f8ba08@hastings.org>
Message-ID: <CAP7h-xbb84qoTuDLum0RHwZUWNrLMa6LCiu0w6z1kBYkSrCWfg@mail.gmail.com>

On Thu, Aug 4, 2016 at 7:12 PM, Larry Hastings <larry at hastings.org> wrote:

> C extension functions get the module passed in automatically, but this is
> done internally and from the Python level you can't see it.


Always something new to learn!  This was not so in Python 2.x - self was
passed as NULL to the C module functions.  When did this change?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160804/5850c8d9/attachment-0001.html>

From eryksun at gmail.com  Thu Aug  4 20:27:02 2016
From: eryksun at gmail.com (eryk sun)
Date: Fri, 5 Aug 2016 00:27:02 +0000
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <CAP7h-xbb84qoTuDLum0RHwZUWNrLMa6LCiu0w6z1kBYkSrCWfg@mail.gmail.com>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
 <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>
 <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>
 <1ded45e5-c77b-fe23-8f67-4f4e29f8ba08@hastings.org>
 <CAP7h-xbb84qoTuDLum0RHwZUWNrLMa6LCiu0w6z1kBYkSrCWfg@mail.gmail.com>
Message-ID: <CACL+1auZqs29VFdum=gJtSbPT7_-xQudWH_6=eG6OxB=4uZWqw@mail.gmail.com>

On Thu, Aug 4, 2016 at 11:33 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
>
> On Thu, Aug 4, 2016 at 7:12 PM, Larry Hastings <larry at hastings.org> wrote:
>>
>> C extension functions get the module passed in automatically, but this is
>> done internally and from the Python level you can't see it.
>
> Always something new to learn!  This was not so in Python 2.x - self was
> passed as NULL to the C module functions.  When did this change?

In 2.x this is the `self` parameter (actually named "passthrough" in
the source) of Py_InitModule4 [1, 2]. You probably use the
Py_InitModule or Py_InitModule3 macros, which pass NULL for this
parameter:

    #define Py_InitModule(name, methods) \
        Py_InitModule4(name, methods, (char *)NULL, (PyObject *)NULL, \
                   PYTHON_API_VERSION)

    #define Py_InitModule3(name, methods, doc) \
        Py_InitModule4(name, methods, doc, (PyObject *)NULL, \
                   PYTHON_API_VERSION)

Python 3's PyModule_Create2 [3-5] API makes this a reference to the
module. It's currently implemented in
PyModule_AddFunctions [6, 7].

[1]: https://docs.python.org/2/c-api/allocation.html#c.Py_InitModule4
[2]: https://hg.python.org/cpython/file/v2.7.12/Python/modsupport.c#l31
[3]: https://docs.python.org/3/c-api/module.html#c.PyModule_Create2
[4]: https://hg.python.org/cpython/file/v3.5.2/Objects/moduleobject.c#l133
[5]: https://hg.python.org/cpython/file/v3.0b1/Objects/moduleobject.c#l63
[6]: https://docs.python.org/3/c-api/module.html#c.PyModule_AddFunctions
[7]: https://hg.python.org/cpython/file/v3.5.2/Objects/moduleobject.c#l387

From ncoghlan at gmail.com  Fri Aug  5 02:58:14 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 5 Aug 2016 16:58:14 +1000
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <1ded45e5-c77b-fe23-8f67-4f4e29f8ba08@hastings.org>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
 <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>
 <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>
 <1ded45e5-c77b-fe23-8f67-4f4e29f8ba08@hastings.org>
Message-ID: <CADiSq7eCb1AAE7bBWgr1JEYOmk=5vzVpg4hdFFcd5sLChHkWNw@mail.gmail.com>

On 5 August 2016 at 09:12, Larry Hastings <larry at hastings.org> wrote:
> / is the delimiter between positional-only parameters and
> positional-or-keyword arguments.  It's not actual Python syntax, but Guido
> said (somewhere) that *if* Python ever sprouted a syntax for positional-only
> parameters, that was as good a delimiter as any.  I think he picked it
> because / is the inverse of *.

I was thinking Guido picked the "/" symbol when we were discussing the
Argument Clinic DSL at PyCon 2014 and you had also been working on
https://www.python.org/dev/peps/pep-0457/

However, re-reading the latter shows it dates back earlier than that:
https://www.python.org/dev/peps/pep-0457/#guido

> If you have more questions about __text_signature__, I recommend reading the
> implementation of inspect.signature, since that's the one and only consumer
> of it.

I occasionally wonder if we should document the "/" notation in
https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object
as it can sometimes show up in the text representation of signature
objects:

    >>> print(inspect.signature(format))
    (value, format_spec='', /)

Likewise for describing the option of using __text_signature__ to
override the default inspect.signature() output (and perhaps adding a
public "inspect.Signature.from_text" alternate constructor at the same
time).

However, either notion is a bit tricky while PEP 457 is still in draft
rather than accepted - even if it's "just" a syntax for overriding
inspect.signature, we'd still be effectively locking this in as *the*
syntax for positional-only arguments (if they're ever added)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From larry at hastings.org  Fri Aug  5 08:36:19 2016
From: larry at hastings.org (Larry Hastings)
Date: Fri, 5 Aug 2016 05:36:19 -0700
Subject: [Python-Dev] Status of the Argument Clinic DSL
In-Reply-To: <CADiSq7eCb1AAE7bBWgr1JEYOmk=5vzVpg4hdFFcd5sLChHkWNw@mail.gmail.com>
References: <CAP7h-xb0LC1EJ17xW_fVr2_ASseCTdfWno8K4OGhoYKMM_OMUg@mail.gmail.com>
 <9186afbf-bf4e-f1af-4bb9-381c881dec47@hastings.org>
 <CAP7h-xYqdo1zDw8QB_1o1nQe7ZhsX8o1TzMU45Q7kFyBBg4qdw@mail.gmail.com>
 <1ded45e5-c77b-fe23-8f67-4f4e29f8ba08@hastings.org>
 <CADiSq7eCb1AAE7bBWgr1JEYOmk=5vzVpg4hdFFcd5sLChHkWNw@mail.gmail.com>
Message-ID: <f6c8ba66-8fa6-3137-166a-79ea07e63a97@hastings.org>


On 08/04/2016 11:58 PM, Nick Coghlan wrote:
> I occasionally wonder if we should document the "/" notation in
> https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object
> as it can sometimes show up in the text representation of signature
> objects:
>
>      >>> print(inspect.signature(format))
>      (value, format_spec='', /)

I think we probably should.  After all, this same string is used for pydoc:

     >>> import os
     >>> help(os.execv)
    Help on built-in function execv in module posix:

    execv(path, argv, /)
         Execute an executable path with arguments, replacing current
    process.

         path
           Path of executable file.
         argv
           Tuple or list of strings.

so it's easily user-visible.

I've always found it  a little strange that the signatures for functions 
using Py_ArgParseTuple() had this / in them that wasn't Python syntax.  
On the other hand, it accurately reflects the fact that these functions 
have signatures that you can't write in Python.

(And, FWIW, I wasn't the person who added the code that made "/" start 
showing up in the text representations of signatures.  I was waffling on 
it, then someone else JFDI, to quote Barry.)


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160805/98a17587/attachment.html>

From status at bugs.python.org  Fri Aug  5 12:08:48 2016
From: status at bugs.python.org (Python tracker)
Date: Fri,  5 Aug 2016 18:08:48 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160805160848.A2B9956A7E@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-07-29 - 2016-08-05)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5589 ( +1)
  closed 33862 (+44)
  total  39451 (+45)

Open issues with patches: 2445 


Issues opened (26)
==================

#27650: Implement `__repr__` methods for logging.Logger and others
http://bugs.python.org/issue27650  opened by cool-RR

#27653: [Patch] Also disable the use of <netinet/tcp.h> on CloudABI
http://bugs.python.org/issue27653  opened by EdSchouten

#27654: [Patch] Use arc4random_buf() on CloudABI
http://bugs.python.org/issue27654  opened by EdSchouten

#27655: [Patch] Don't require presence of POLLPRI
http://bugs.python.org/issue27655  opened by EdSchouten

#27657: urlparse fails if the path is numeric
http://bugs.python.org/issue27657  opened by Bj??rn.Lindqvist

#27658: python 3.5.2 built from source fails to install completely on 
http://bugs.python.org/issue27658  opened by Mario Grgic

#27659: Check for the existence of crypt()
http://bugs.python.org/issue27659  opened by Chi Hsuan Yen

#27660: Replace size_t with Py_ssize_t as the type of local variable i
http://bugs.python.org/issue27660  opened by xiang.zhang

#27662: Check against PY_SSIZE_T_MAX instead of PY_SIZE_MAX in List_Ne
http://bugs.python.org/issue27662  opened by xiang.zhang

#27664: Allow specifying prefix for thread name in concurrent.futures.
http://bugs.python.org/issue27664  opened by durin42

#27665: Make create_server able to listen on several ports
http://bugs.python.org/issue27665  opened by bayandin

#27666: "stack smashing detected" in PyCursesWindow_Box
http://bugs.python.org/issue27666  opened by Steve Fink

#27671: FAQ: len() is still function for good reason.
http://bugs.python.org/issue27671  opened by methane

#27674: Quote mark breaks cookie processing
http://bugs.python.org/issue27674  opened by Artur Sm??t

#27675: IDLE file completion has 2 bugs depending on open quote used
http://bugs.python.org/issue27675  opened by egerosa

#27679: set_bitfields() unused in _ctypes_test
http://bugs.python.org/issue27679  opened by martin.panter

#27680: Reduce Github pull request rate
http://bugs.python.org/issue27680  opened by holdenweb

#27681: readline: consecutive rl_kill_word do not append
http://bugs.python.org/issue27681  opened by qsantos

#27682: Windows Error 10053, ConnectionAbortedError: [WinError 10053] 
http://bugs.python.org/issue27682  opened by SG

#27683: ipaddress subnet slicing iterator malfunction
http://bugs.python.org/issue27683  opened by tklausmann

#27687: Linux shutil.move between mountpoints as root does not retain 
http://bugs.python.org/issue27687  opened by jort.bloem

#27688: Expand documentation about Any in the typing module
http://bugs.python.org/issue27688  opened by michael0x2a

#27689: Add documentation for typing.Generator
http://bugs.python.org/issue27689  opened by michael0x2a

#27691: X509 cert with GEN_RID subject alt name causes SytemError
http://bugs.python.org/issue27691  opened by christian.heimes

#27692: Clean serveral unnecessary NULL checks in exceptions.c
http://bugs.python.org/issue27692  opened by xiang.zhang

#27693: curses.textpad.Textbox(win).edit() won't edit last character
http://bugs.python.org/issue27693  opened by Dietmar Schindler



Most recent 15 issues with no replies (15)
==========================================

#27693: curses.textpad.Textbox(win).edit() won't edit last character
http://bugs.python.org/issue27693

#27691: X509 cert with GEN_RID subject alt name causes SytemError
http://bugs.python.org/issue27691

#27689: Add documentation for typing.Generator
http://bugs.python.org/issue27689

#27688: Expand documentation about Any in the typing module
http://bugs.python.org/issue27688

#27687: Linux shutil.move between mountpoints as root does not retain 
http://bugs.python.org/issue27687

#27679: set_bitfields() unused in _ctypes_test
http://bugs.python.org/issue27679

#27671: FAQ: len() is still function for good reason.
http://bugs.python.org/issue27671

#27666: "stack smashing detected" in PyCursesWindow_Box
http://bugs.python.org/issue27666

#27664: Allow specifying prefix for thread name in concurrent.futures.
http://bugs.python.org/issue27664

#27654: [Patch] Use arc4random_buf() on CloudABI
http://bugs.python.org/issue27654

#27653: [Patch] Also disable the use of <netinet/tcp.h> on CloudABI
http://bugs.python.org/issue27653

#27646: yield from expression can be any iterable
http://bugs.python.org/issue27646

#27636: Refactor IDLE htest
http://bugs.python.org/issue27636

#27635: pickle documentation says that unpickling may not call __new__
http://bugs.python.org/issue27635

#27599: Buffer overrun in binascii
http://bugs.python.org/issue27599



Most recent 15 issues waiting for review (15)
=============================================

#27692: Clean serveral unnecessary NULL checks in exceptions.c
http://bugs.python.org/issue27692

#27691: X509 cert with GEN_RID subject alt name causes SytemError
http://bugs.python.org/issue27691

#27689: Add documentation for typing.Generator
http://bugs.python.org/issue27689

#27688: Expand documentation about Any in the typing module
http://bugs.python.org/issue27688

#27683: ipaddress subnet slicing iterator malfunction
http://bugs.python.org/issue27683

#27680: Reduce Github pull request rate
http://bugs.python.org/issue27680

#27665: Make create_server able to listen on several ports
http://bugs.python.org/issue27665

#27664: Allow specifying prefix for thread name in concurrent.futures.
http://bugs.python.org/issue27664

#27662: Check against PY_SSIZE_T_MAX instead of PY_SIZE_MAX in List_Ne
http://bugs.python.org/issue27662

#27660: Replace size_t with Py_ssize_t as the type of local variable i
http://bugs.python.org/issue27660

#27659: Check for the existence of crypt()
http://bugs.python.org/issue27659

#27655: [Patch] Don't require presence of POLLPRI
http://bugs.python.org/issue27655

#27650: Implement `__repr__` methods for logging.Logger and others
http://bugs.python.org/issue27650

#27646: yield from expression can be any iterable
http://bugs.python.org/issue27646

#27645: Supporting native backup facility of SQLite
http://bugs.python.org/issue27645



Top 10 most discussed issues (10)
=================================

#20767: Some python extensions can't be compiled with clang 3.4
http://bugs.python.org/issue20767  11 msgs

#27604: More details about `-O` flag
http://bugs.python.org/issue27604  10 msgs

#27621: Finish IDLE Query dialog appearance and behavior.
http://bugs.python.org/issue27621  10 msgs

#27660: Replace size_t with Py_ssize_t as the type of local variable i
http://bugs.python.org/issue27660  10 msgs

#1621: Do not assume signed integer overflow behavior
http://bugs.python.org/issue1621   8 msgs

#20160: broken ctypes calling convention on MSVC / 64-bit Windows (lar
http://bugs.python.org/issue20160   8 msgs

#27683: ipaddress subnet slicing iterator malfunction
http://bugs.python.org/issue27683   8 msgs

#19660: decorator syntax: allow testlist instead of just dotted_name
http://bugs.python.org/issue19660   7 msgs

#27682: Windows Error 10053, ConnectionAbortedError: [WinError 10053] 
http://bugs.python.org/issue27682   7 msgs

#27657: urlparse fails if the path is numeric
http://bugs.python.org/issue27657   6 msgs



Issues closed (41)
==================

#6926: socket module missing IPPROTO_IPV6, IPPROTO_IPV4
http://bugs.python.org/issue6926  closed by berker.peksag

#8418: Crash 0xc0000417 STATUS_INVALID_CRUNTIME_PARAMETER on program 
http://bugs.python.org/issue8418  closed by berker.peksag

#13651: Improve redirection in urllib
http://bugs.python.org/issue13651  closed by martin.panter

#17310: Ctypes callbacks shows problem on Windows Python (64bit)
http://bugs.python.org/issue17310  closed by vinay.sajip

#17599: mingw: detect REPARSE_DATA_BUFFER
http://bugs.python.org/issue17599  closed by martin.panter

#23710: C API doc for PyObject_HEAD is outdated
http://bugs.python.org/issue23710  closed by berker.peksag

#24082: Obsolete note in argument parsing (c-api/arg.rst)
http://bugs.python.org/issue24082  closed by martin.panter

#24278: Docs on Parsing arguments should say something about mem mgmt 
http://bugs.python.org/issue24278  closed by martin.panter

#24672: shutil.rmtree fails on non ascii filenames
http://bugs.python.org/issue24672  closed by r.david.murray

#26109: _Py_DumpTraceback should be PyAPI_FUNC
http://bugs.python.org/issue26109  closed by berker.peksag

#26462: Patch to enhance literal block language declaration
http://bugs.python.org/issue26462  closed by martin.panter

#26576: Tweak wording of decorator docos
http://bugs.python.org/issue26576  closed by berker.peksag

#27241: Catch exceptions raised in pstats add (repl)
http://bugs.python.org/issue27241  closed by skrah

#27366: PEP487: Simpler customization of class creation
http://bugs.python.org/issue27366  closed by ncoghlan

#27389: When a TypeError is raised due to invalid arguments to a metho
http://bugs.python.org/issue27389  closed by Steven.Barker

#27424: Failures in test.test_logging
http://bugs.python.org/issue27424  closed by vinay.sajip

#27453: $CPP invocation in configure must use $CPPFLAGS
http://bugs.python.org/issue27453  closed by xdegaye

#27568: "HTTPoxy", use of HTTP_PROXY flag supplied by attacker in CGI 
http://bugs.python.org/issue27568  closed by orsenthil

#27638: int.to_bytes() and int.from_bytes() should support 'net' and '
http://bugs.python.org/issue27638  closed by rhettinger

#27641: Do not build Programs/_freeze_importlib when cross-compiling
http://bugs.python.org/issue27641  closed by martin.panter

#27644: Expand documentation about type aliases and NewType in the typ
http://bugs.python.org/issue27644  closed by gvanrossum

#27651: bytearray changes when function mutates it
http://bugs.python.org/issue27651  closed by SilentGhost

#27652: Make use of non-standard error number ESHUTDOWN optional
http://bugs.python.org/issue27652  closed by berker.peksag

#27656: [Patch] Make presence of SCHED_* optional
http://bugs.python.org/issue27656  closed by python-dev

#27661: Add tzinfo= argument to datetime.combine
http://bugs.python.org/issue27661  closed by python-dev

#27663: Inconsistent CSV Writer Behaviour
http://bugs.python.org/issue27663  closed by SilentGhost

#27667: subprocess32 unable to be installed via pip and python install
http://bugs.python.org/issue27667  closed by berker.peksag

#27668: list illogical affectation
http://bugs.python.org/issue27668  closed by SilentGhost

#27669: Bug in re.fullmatch() specific to Windows XP
http://bugs.python.org/issue27669  closed by zach.ware

#27670: Use https instead of http in the PYTHON_VERSION constant
http://bugs.python.org/issue27670  closed by python-dev

#27672: x fromat of % operator broken in Python 3.5.2
http://bugs.python.org/issue27672  closed by SilentGhost

#27673: lambda function in loop
http://bugs.python.org/issue27673  closed by ebarry

#27676: comports() failing due to internal 4G module
http://bugs.python.org/issue27676  closed by zach.ware

#27677: Reopening issue 20930
http://bugs.python.org/issue27677  closed by SilentGhost

#27678: itertools.permutations/.combinations should have len
http://bugs.python.org/issue27678  closed by r.david.murray

#27684: Incorrect exception message of ImportError constructor
http://bugs.python.org/issue27684  closed by berker.peksag

#27685: altinstall with --enable-shared showing incorrect behaviour
http://bugs.python.org/issue27685  closed by ubehera

#27686: Spam
http://bugs.python.org/issue27686  closed by zach.ware

#27690: os.path.normcase broken.
http://bugs.python.org/issue27690  closed by serhiy.storchaka

#27694: Socket incorrectly states IP address that start with a zero af
http://bugs.python.org/issue27694  closed by martin.panter

#1521950: shlex.split() does not tokenize like the shell
http://bugs.python.org/issue1521950  closed by python-dev

From brett at python.org  Fri Aug  5 17:04:30 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 05 Aug 2016 21:04:30 +0000
Subject: [Python-Dev] C99
In-Reply-To: <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
Message-ID: <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>

Where did we finally land on this discussion? Do we want to update PEP 7 to
say that starting in 3.6 we may use C99 features common to all supported
compilers and list what those compilers are (i.e. gcc, clang, and MSVC)?

On Wed, 8 Jun 2016 at 01:28 Victor Stinner <victor.stinner at gmail.com> wrote:

> I guess that as usual, we should use the "common denominator" of all
> compilers supported by CPython. For example, if MSVC doesn't support a
> feature, we should not use it in CPython.
>
> In practice, it's easy to check if a feature is supported or not: we
> have buildbots building Python at each commit. It was very common to
> get a compilation error only on MSVC when a variable was defined in
> the middle of a function. We are now using
> -Werror=declaration-after-statement with GCC because of MSVC!
>
> Maybe GCC has an option to ask for the subset of the C99 standard
> compatible with MSVC? Something like "-std=c99 -pedantic"?
>
> Note: I tried -pedantic, GCC emits a lot of warnings on code which
> looks valid and/or is protected with #ifdef for features specific to
> GCC like computed goto.
>
> Victor
>
> 2016-06-07 21:45 GMT+02:00 Guido van Rossum <gvanrossum at gmail.com>:
> > We should definitely keep supporting MSVC.
> >
> > --Guido (mobile)
> >
> > On Jun 7, 2016 12:39 PM, "Sturla Molden" <sturla.molden at gmail.com>
> wrote:
> >>
> >> Victor Stinner <victor.stinner at gmail.com> wrote:
> >>
> >> > Is it worth to support a compiler that in 2016 doesn't support the C
> >> > standard released in 1999, 17 years ago?
> >>
> >> MSVC only supports C99 when its needed for C++11 or some MS extension to
> >> C.
> >>
> >> Is it worth supporting MSVC? If not, we have Intel C, Clang and Cygwin
> GCC
> >> are the viable options we have on Windows (and perhaps Embarcadero, but
> I
> >> haven't used C++ builder for a very long time). Even MinGW does not
> fully
> >> support C99, because it depends on Microsoft's CRT. If we think MSVC and
> >> MinGW are worth supporting, we cannot just use C99 indiscriminantly.
> >>
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> >
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
> >
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160805/9c8df32c/attachment.html>

From gvanrossum at gmail.com  Fri Aug  5 18:06:55 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Fri, 5 Aug 2016 15:06:55 -0700
Subject: [Python-Dev] C99
In-Reply-To: <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
Message-ID: <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>

That sounds fine to me, but we need to list specific compiler versions.

On Fri, Aug 5, 2016 at 2:04 PM, Brett Cannon <brett at python.org> wrote:
> Where did we finally land on this discussion? Do we want to update PEP 7 to
> say that starting in 3.6 we may use C99 features common to all supported
> compilers and list what those compilers are (i.e. gcc, clang, and MSVC)?
>
> On Wed, 8 Jun 2016 at 01:28 Victor Stinner <victor.stinner at gmail.com> wrote:
>>
>> I guess that as usual, we should use the "common denominator" of all
>> compilers supported by CPython. For example, if MSVC doesn't support a
>> feature, we should not use it in CPython.
>>
>> In practice, it's easy to check if a feature is supported or not: we
>> have buildbots building Python at each commit. It was very common to
>> get a compilation error only on MSVC when a variable was defined in
>> the middle of a function. We are now using
>> -Werror=declaration-after-statement with GCC because of MSVC!
>>
>> Maybe GCC has an option to ask for the subset of the C99 standard
>> compatible with MSVC? Something like "-std=c99 -pedantic"?
>>
>> Note: I tried -pedantic, GCC emits a lot of warnings on code which
>> looks valid and/or is protected with #ifdef for features specific to
>> GCC like computed goto.
>>
>> Victor
>>
>> 2016-06-07 21:45 GMT+02:00 Guido van Rossum <gvanrossum at gmail.com>:
>> > We should definitely keep supporting MSVC.
>> >
>> > --Guido (mobile)
>> >
>> > On Jun 7, 2016 12:39 PM, "Sturla Molden" <sturla.molden at gmail.com>
>> > wrote:
>> >>
>> >> Victor Stinner <victor.stinner at gmail.com> wrote:
>> >>
>> >> > Is it worth to support a compiler that in 2016 doesn't support the C
>> >> > standard released in 1999, 17 years ago?
>> >>
>> >> MSVC only supports C99 when its needed for C++11 or some MS extension
>> >> to
>> >> C.
>> >>
>> >> Is it worth supporting MSVC? If not, we have Intel C, Clang and Cygwin
>> >> GCC
>> >> are the viable options we have on Windows (and perhaps Embarcadero, but
>> >> I
>> >> haven't used C++ builder for a very long time). Even MinGW does not
>> >> fully
>> >> support C99, because it depends on Microsoft's CRT. If we think MSVC
>> >> and
>> >> MinGW are worth supporting, we cannot just use C99 indiscriminantly.
>> >>
>> >> _______________________________________________
>> >> Python-Dev mailing list
>> >> Python-Dev at python.org
>> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> Unsubscribe:
>> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>> >
>> >
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> >
>> > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>> >
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From gvanrossum at gmail.com  Fri Aug  5 18:17:12 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Fri, 5 Aug 2016 15:17:12 -0700
Subject: [Python-Dev] C99
In-Reply-To: <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
Message-ID: <CAP7+vJLtqwxkbm-DA_PQx7gQN1sz4QybO_YZmGW7XhQ6fTFAjg@mail.gmail.com>

I want it to list specific versions that are known to be good right
now, so we can point fingers appropriately when a regression happens.
If you have to ask Steve what version he used, ask!

On Fri, Aug 5, 2016 at 3:15 PM, Brett Cannon <brett at python.org> wrote:
>
>
> On Fri, 5 Aug 2016 at 15:07 Guido van Rossum <gvanrossum at gmail.com> wrote:
>>
>> That sounds fine to me, but we need to list specific compiler versions.
>
>
> Would you want this to be static (e.g. MSVC 2016 until we choose to update
> to support C11), or would you want it to vary based on what's available when
> the current/last Python is/was released (e.g. whatever version of MSVC Steve
> uses to build the binaries for 3.5 or 3.6 in our current case)?
>
>>
>>
>> On Fri, Aug 5, 2016 at 2:04 PM, Brett Cannon <brett at python.org> wrote:
>> > Where did we finally land on this discussion? Do we want to update PEP 7
>> > to
>> > say that starting in 3.6 we may use C99 features common to all supported
>> > compilers and list what those compilers are (i.e. gcc, clang, and MSVC)?
>> >
>> > On Wed, 8 Jun 2016 at 01:28 Victor Stinner <victor.stinner at gmail.com>
>> > wrote:
>> >>
>> >> I guess that as usual, we should use the "common denominator" of all
>> >> compilers supported by CPython. For example, if MSVC doesn't support a
>> >> feature, we should not use it in CPython.
>> >>
>> >> In practice, it's easy to check if a feature is supported or not: we
>> >> have buildbots building Python at each commit. It was very common to
>> >> get a compilation error only on MSVC when a variable was defined in
>> >> the middle of a function. We are now using
>> >> -Werror=declaration-after-statement with GCC because of MSVC!
>> >>
>> >> Maybe GCC has an option to ask for the subset of the C99 standard
>> >> compatible with MSVC? Something like "-std=c99 -pedantic"?
>> >>
>> >> Note: I tried -pedantic, GCC emits a lot of warnings on code which
>> >> looks valid and/or is protected with #ifdef for features specific to
>> >> GCC like computed goto.
>> >>
>> >> Victor
>> >>
>> >> 2016-06-07 21:45 GMT+02:00 Guido van Rossum <gvanrossum at gmail.com>:
>> >> > We should definitely keep supporting MSVC.
>> >> >
>> >> > --Guido (mobile)
>> >> >
>> >> > On Jun 7, 2016 12:39 PM, "Sturla Molden" <sturla.molden at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Victor Stinner <victor.stinner at gmail.com> wrote:
>> >> >>
>> >> >> > Is it worth to support a compiler that in 2016 doesn't support the
>> >> >> > C
>> >> >> > standard released in 1999, 17 years ago?
>> >> >>
>> >> >> MSVC only supports C99 when its needed for C++11 or some MS
>> >> >> extension
>> >> >> to
>> >> >> C.
>> >> >>
>> >> >> Is it worth supporting MSVC? If not, we have Intel C, Clang and
>> >> >> Cygwin
>> >> >> GCC
>> >> >> are the viable options we have on Windows (and perhaps Embarcadero,
>> >> >> but
>> >> >> I
>> >> >> haven't used C++ builder for a very long time). Even MinGW does not
>> >> >> fully
>> >> >> support C99, because it depends on Microsoft's CRT. If we think MSVC
>> >> >> and
>> >> >> MinGW are worth supporting, we cannot just use C99 indiscriminantly.
>> >> >>
>> >> >> _______________________________________________
>> >> >> Python-Dev mailing list
>> >> >> Python-Dev at python.org
>> >> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> >> Unsubscribe:
>> >> >>
>> >> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Python-Dev mailing list
>> >> > Python-Dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/python-dev
>> >> > Unsubscribe:
>> >> >
>> >> >
>> >> > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>> >> >
>> >> _______________________________________________
>> >> Python-Dev mailing list
>> >> Python-Dev at python.org
>> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> Unsubscribe:
>> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)



-- 
--Guido van Rossum (python.org/~guido)

From brett at python.org  Fri Aug  5 18:27:33 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 05 Aug 2016 22:27:33 +0000
Subject: [Python-Dev] C99
In-Reply-To: <CAP7+vJLtqwxkbm-DA_PQx7gQN1sz4QybO_YZmGW7XhQ6fTFAjg@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <CAP7+vJLtqwxkbm-DA_PQx7gQN1sz4QybO_YZmGW7XhQ6fTFAjg@mail.gmail.com>
Message-ID: <CAP1=2W7RQ5eXtOhkL=+ZV_C7Ri_n9Y2J-6b6htR3tuExPtfqaQ@mail.gmail.com>

On Fri, 5 Aug 2016 at 15:17 Guido van Rossum <gvanrossum at gmail.com> wrote:

> I want it to list specific versions that are known to be good right
> now, so we can point fingers appropriately when a regression happens.
>

OK, then we could pin it to MSVC 2016, gcc 6.1, and clang 3.8.1 which are
the latest releases (unless I'm missing a compiler we all feel like we
should officially support).


> If you have to ask Steve what version he used, ask!
>

I know of what compiler has been used by Steve, I was just trying to give a
very loose example of how it could be phrased if we wanted a shifting
window of support based on what was available at the time of release for
Python instead of the "what was available Aug 2016" static specification
that you said you wanted.


>
> On Fri, Aug 5, 2016 at 3:15 PM, Brett Cannon <brett at python.org> wrote:
> >
> >
> > On Fri, 5 Aug 2016 at 15:07 Guido van Rossum <gvanrossum at gmail.com>
> wrote:
> >>
> >> That sounds fine to me, but we need to list specific compiler versions.
> >
> >
> > Would you want this to be static (e.g. MSVC 2016 until we choose to
> update
> > to support C11), or would you want it to vary based on what's available
> when
> > the current/last Python is/was released (e.g. whatever version of MSVC
> Steve
> > uses to build the binaries for 3.5 or 3.6 in our current case)?
> >
> >>
> >>
> >> On Fri, Aug 5, 2016 at 2:04 PM, Brett Cannon <brett at python.org> wrote:
> >> > Where did we finally land on this discussion? Do we want to update
> PEP 7
> >> > to
> >> > say that starting in 3.6 we may use C99 features common to all
> supported
> >> > compilers and list what those compilers are (i.e. gcc, clang, and
> MSVC)?
> >> >
> >> > On Wed, 8 Jun 2016 at 01:28 Victor Stinner <victor.stinner at gmail.com>
> >> > wrote:
> >> >>
> >> >> I guess that as usual, we should use the "common denominator" of all
> >> >> compilers supported by CPython. For example, if MSVC doesn't support
> a
> >> >> feature, we should not use it in CPython.
> >> >>
> >> >> In practice, it's easy to check if a feature is supported or not: we
> >> >> have buildbots building Python at each commit. It was very common to
> >> >> get a compilation error only on MSVC when a variable was defined in
> >> >> the middle of a function. We are now using
> >> >> -Werror=declaration-after-statement with GCC because of MSVC!
> >> >>
> >> >> Maybe GCC has an option to ask for the subset of the C99 standard
> >> >> compatible with MSVC? Something like "-std=c99 -pedantic"?
> >> >>
> >> >> Note: I tried -pedantic, GCC emits a lot of warnings on code which
> >> >> looks valid and/or is protected with #ifdef for features specific to
> >> >> GCC like computed goto.
> >> >>
> >> >> Victor
> >> >>
> >> >> 2016-06-07 21:45 GMT+02:00 Guido van Rossum <gvanrossum at gmail.com>:
> >> >> > We should definitely keep supporting MSVC.
> >> >> >
> >> >> > --Guido (mobile)
> >> >> >
> >> >> > On Jun 7, 2016 12:39 PM, "Sturla Molden" <sturla.molden at gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Victor Stinner <victor.stinner at gmail.com> wrote:
> >> >> >>
> >> >> >> > Is it worth to support a compiler that in 2016 doesn't support
> the
> >> >> >> > C
> >> >> >> > standard released in 1999, 17 years ago?
> >> >> >>
> >> >> >> MSVC only supports C99 when its needed for C++11 or some MS
> >> >> >> extension
> >> >> >> to
> >> >> >> C.
> >> >> >>
> >> >> >> Is it worth supporting MSVC? If not, we have Intel C, Clang and
> >> >> >> Cygwin
> >> >> >> GCC
> >> >> >> are the viable options we have on Windows (and perhaps
> Embarcadero,
> >> >> >> but
> >> >> >> I
> >> >> >> haven't used C++ builder for a very long time). Even MinGW does
> not
> >> >> >> fully
> >> >> >> support C99, because it depends on Microsoft's CRT. If we think
> MSVC
> >> >> >> and
> >> >> >> MinGW are worth supporting, we cannot just use C99
> indiscriminantly.
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> Python-Dev mailing list
> >> >> >> Python-Dev at python.org
> >> >> >> https://mail.python.org/mailman/listinfo/python-dev
> >> >> >> Unsubscribe:
> >> >> >>
> >> >> >>
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > Python-Dev mailing list
> >> >> > Python-Dev at python.org
> >> >> > https://mail.python.org/mailman/listinfo/python-dev
> >> >> > Unsubscribe:
> >> >> >
> >> >> >
> >> >> >
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
> >> >> >
> >> >> _______________________________________________
> >> >> Python-Dev mailing list
> >> >> Python-Dev at python.org
> >> >> https://mail.python.org/mailman/listinfo/python-dev
> >> >> Unsubscribe:
> >> >>
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
> >>
> >>
> >>
> >> --
> >> --Guido van Rossum (python.org/~guido)
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160805/5e473637/attachment-0001.html>

From gvanrossum at gmail.com  Fri Aug  5 18:37:46 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Fri, 5 Aug 2016 15:37:46 -0700
Subject: [Python-Dev] C99
In-Reply-To: <CAP1=2W7RQ5eXtOhkL=+ZV_C7Ri_n9Y2J-6b6htR3tuExPtfqaQ@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <CAP7+vJLtqwxkbm-DA_PQx7gQN1sz4QybO_YZmGW7XhQ6fTFAjg@mail.gmail.com>
 <CAP1=2W7RQ5eXtOhkL=+ZV_C7Ri_n9Y2J-6b6htR3tuExPtfqaQ@mail.gmail.com>
Message-ID: <CAP7+vJLbf_Q=4G4V_SnTzZgK40nhEGVkqhtn1uz_N-bW-gvjww@mail.gmail.com>

I think if we want to revisit this in the future it should be an
explicit change.

On Fri, Aug 5, 2016 at 3:27 PM, Brett Cannon <brett at python.org> wrote:
>
>
> On Fri, 5 Aug 2016 at 15:17 Guido van Rossum <gvanrossum at gmail.com> wrote:
>>
>> I want it to list specific versions that are known to be good right
>> now, so we can point fingers appropriately when a regression happens.
>
>
> OK, then we could pin it to MSVC 2016, gcc 6.1, and clang 3.8.1 which are
> the latest releases (unless I'm missing a compiler we all feel like we
> should officially support).
>
>>
>> If you have to ask Steve what version he used, ask!
>
>
> I know of what compiler has been used by Steve, I was just trying to give a
> very loose example of how it could be phrased if we wanted a shifting window
> of support based on what was available at the time of release for Python
> instead of the "what was available Aug 2016" static specification that you
> said you wanted.
>
>>
>>
>> On Fri, Aug 5, 2016 at 3:15 PM, Brett Cannon <brett at python.org> wrote:
>> >
>> >
>> > On Fri, 5 Aug 2016 at 15:07 Guido van Rossum <gvanrossum at gmail.com>
>> > wrote:
>> >>
>> >> That sounds fine to me, but we need to list specific compiler versions.
>> >
>> >
>> > Would you want this to be static (e.g. MSVC 2016 until we choose to
>> > update
>> > to support C11), or would you want it to vary based on what's available
>> > when
>> > the current/last Python is/was released (e.g. whatever version of MSVC
>> > Steve
>> > uses to build the binaries for 3.5 or 3.6 in our current case)?
>> >
>> >>
>> >>
>> >> On Fri, Aug 5, 2016 at 2:04 PM, Brett Cannon <brett at python.org> wrote:
>> >> > Where did we finally land on this discussion? Do we want to update
>> >> > PEP 7
>> >> > to
>> >> > say that starting in 3.6 we may use C99 features common to all
>> >> > supported
>> >> > compilers and list what those compilers are (i.e. gcc, clang, and
>> >> > MSVC)?
>> >> >
>> >> > On Wed, 8 Jun 2016 at 01:28 Victor Stinner <victor.stinner at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> I guess that as usual, we should use the "common denominator" of all
>> >> >> compilers supported by CPython. For example, if MSVC doesn't support
>> >> >> a
>> >> >> feature, we should not use it in CPython.
>> >> >>
>> >> >> In practice, it's easy to check if a feature is supported or not: we
>> >> >> have buildbots building Python at each commit. It was very common to
>> >> >> get a compilation error only on MSVC when a variable was defined in
>> >> >> the middle of a function. We are now using
>> >> >> -Werror=declaration-after-statement with GCC because of MSVC!
>> >> >>
>> >> >> Maybe GCC has an option to ask for the subset of the C99 standard
>> >> >> compatible with MSVC? Something like "-std=c99 -pedantic"?
>> >> >>
>> >> >> Note: I tried -pedantic, GCC emits a lot of warnings on code which
>> >> >> looks valid and/or is protected with #ifdef for features specific to
>> >> >> GCC like computed goto.
>> >> >>
>> >> >> Victor
>> >> >>
>> >> >> 2016-06-07 21:45 GMT+02:00 Guido van Rossum <gvanrossum at gmail.com>:
>> >> >> > We should definitely keep supporting MSVC.
>> >> >> >
>> >> >> > --Guido (mobile)
>> >> >> >
>> >> >> > On Jun 7, 2016 12:39 PM, "Sturla Molden" <sturla.molden at gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Victor Stinner <victor.stinner at gmail.com> wrote:
>> >> >> >>
>> >> >> >> > Is it worth to support a compiler that in 2016 doesn't support
>> >> >> >> > the
>> >> >> >> > C
>> >> >> >> > standard released in 1999, 17 years ago?
>> >> >> >>
>> >> >> >> MSVC only supports C99 when its needed for C++11 or some MS
>> >> >> >> extension
>> >> >> >> to
>> >> >> >> C.
>> >> >> >>
>> >> >> >> Is it worth supporting MSVC? If not, we have Intel C, Clang and
>> >> >> >> Cygwin
>> >> >> >> GCC
>> >> >> >> are the viable options we have on Windows (and perhaps
>> >> >> >> Embarcadero,
>> >> >> >> but
>> >> >> >> I
>> >> >> >> haven't used C++ builder for a very long time). Even MinGW does
>> >> >> >> not
>> >> >> >> fully
>> >> >> >> support C99, because it depends on Microsoft's CRT. If we think
>> >> >> >> MSVC
>> >> >> >> and
>> >> >> >> MinGW are worth supporting, we cannot just use C99
>> >> >> >> indiscriminantly.
>> >> >> >>
>> >> >> >> _______________________________________________
>> >> >> >> Python-Dev mailing list
>> >> >> >> Python-Dev at python.org
>> >> >> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> >> >> Unsubscribe:
>> >> >> >>
>> >> >> >>
>> >> >> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>> >> >> >
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > Python-Dev mailing list
>> >> >> > Python-Dev at python.org
>> >> >> > https://mail.python.org/mailman/listinfo/python-dev
>> >> >> > Unsubscribe:
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>> >> >> >
>> >> >> _______________________________________________
>> >> >> Python-Dev mailing list
>> >> >> Python-Dev at python.org
>> >> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> >> Unsubscribe:
>> >> >>
>> >> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>> >>
>> >>
>> >>
>> >> --
>> >> --Guido van Rossum (python.org/~guido)
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)



-- 
--Guido van Rossum (python.org/~guido)

From brett at python.org  Fri Aug  5 18:15:22 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 05 Aug 2016 22:15:22 +0000
Subject: [Python-Dev] C99
In-Reply-To: <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
Message-ID: <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>

On Fri, 5 Aug 2016 at 15:07 Guido van Rossum <gvanrossum at gmail.com> wrote:

> That sounds fine to me, but we need to list specific compiler versions.
>

Would you want this to be static (e.g. MSVC 2016 until we choose to update
to support C11), or would you want it to vary based on what's available
when the current/last Python is/was released (e.g. whatever version of MSVC
Steve uses to build the binaries for 3.5 or 3.6 in our current case)?


>
> On Fri, Aug 5, 2016 at 2:04 PM, Brett Cannon <brett at python.org> wrote:
> > Where did we finally land on this discussion? Do we want to update PEP 7
> to
> > say that starting in 3.6 we may use C99 features common to all supported
> > compilers and list what those compilers are (i.e. gcc, clang, and MSVC)?
> >
> > On Wed, 8 Jun 2016 at 01:28 Victor Stinner <victor.stinner at gmail.com>
> wrote:
> >>
> >> I guess that as usual, we should use the "common denominator" of all
> >> compilers supported by CPython. For example, if MSVC doesn't support a
> >> feature, we should not use it in CPython.
> >>
> >> In practice, it's easy to check if a feature is supported or not: we
> >> have buildbots building Python at each commit. It was very common to
> >> get a compilation error only on MSVC when a variable was defined in
> >> the middle of a function. We are now using
> >> -Werror=declaration-after-statement with GCC because of MSVC!
> >>
> >> Maybe GCC has an option to ask for the subset of the C99 standard
> >> compatible with MSVC? Something like "-std=c99 -pedantic"?
> >>
> >> Note: I tried -pedantic, GCC emits a lot of warnings on code which
> >> looks valid and/or is protected with #ifdef for features specific to
> >> GCC like computed goto.
> >>
> >> Victor
> >>
> >> 2016-06-07 21:45 GMT+02:00 Guido van Rossum <gvanrossum at gmail.com>:
> >> > We should definitely keep supporting MSVC.
> >> >
> >> > --Guido (mobile)
> >> >
> >> > On Jun 7, 2016 12:39 PM, "Sturla Molden" <sturla.molden at gmail.com>
> >> > wrote:
> >> >>
> >> >> Victor Stinner <victor.stinner at gmail.com> wrote:
> >> >>
> >> >> > Is it worth to support a compiler that in 2016 doesn't support the
> C
> >> >> > standard released in 1999, 17 years ago?
> >> >>
> >> >> MSVC only supports C99 when its needed for C++11 or some MS extension
> >> >> to
> >> >> C.
> >> >>
> >> >> Is it worth supporting MSVC? If not, we have Intel C, Clang and
> Cygwin
> >> >> GCC
> >> >> are the viable options we have on Windows (and perhaps Embarcadero,
> but
> >> >> I
> >> >> haven't used C++ builder for a very long time). Even MinGW does not
> >> >> fully
> >> >> support C99, because it depends on Microsoft's CRT. If we think MSVC
> >> >> and
> >> >> MinGW are worth supporting, we cannot just use C99 indiscriminantly.
> >> >>
> >> >> _______________________________________________
> >> >> Python-Dev mailing list
> >> >> Python-Dev at python.org
> >> >> https://mail.python.org/mailman/listinfo/python-dev
> >> >> Unsubscribe:
> >> >>
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >> >
> >> >
> >> > _______________________________________________
> >> > Python-Dev mailing list
> >> > Python-Dev at python.org
> >> > https://mail.python.org/mailman/listinfo/python-dev
> >> > Unsubscribe:
> >> >
> >> >
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
> >> >
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160805/d0df895e/attachment.html>

From steve.dower at python.org  Fri Aug  5 22:15:30 2016
From: steve.dower at python.org (Steve Dower)
Date: Fri, 5 Aug 2016 19:15:30 -0700
Subject: [Python-Dev] C99
In-Reply-To: <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
Message-ID: <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>

FYI, it's MSVC 14.0 (which is included in VS 2015).

Personally I'd like to see it restricted to the common subset of C99 and some version of C++ (which is obviously mostly C and includes no C++), because I know there are a few things in C99 only that are very obscure because they aren't also in C++. 

I'd settle for saying that all header files must be C++ compatible so that embedders and extenders have the option of C or C++, even if we allow the obscure parts of C99 inside our code.

Cheers,
Steve

Top-posted from my Windows Phone

-----Original Message-----
From: "Brett Cannon" <brett at python.org>
Sent: ?8/?5/?2016 15:44
To: "Guido van Rossum" <gvanrossum at gmail.com>
Cc: "Python-Dev" <python-dev at python.org>; "Sturla Molden" <sturla.molden at gmail.com>
Subject: Re: [Python-Dev] C99




On Fri, 5 Aug 2016 at 15:07 Guido van Rossum <gvanrossum at gmail.com> wrote:

That sounds fine to me, but we need to list specific compiler versions.



Would you want this to be static (e.g. MSVC 2016 until we choose to update to support C11), or would you want it to vary based on what's available when the current/last Python is/was released (e.g. whatever version of MSVC Steve uses to build the binaries for 3.5 or 3.6 in our current case)?
 

On Fri, Aug 5, 2016 at 2:04 PM, Brett Cannon <brett at python.org> wrote:
> Where did we finally land on this discussion? Do we want to update PEP 7 to
> say that starting in 3.6 we may use C99 features common to all supported
> compilers and list what those compilers are (i.e. gcc, clang, and MSVC)?
>
> On Wed, 8 Jun 2016 at 01:28 Victor Stinner <victor.stinner at gmail.com> wrote:
>>
>> I guess that as usual, we should use the "common denominator" of all
>> compilers supported by CPython. For example, if MSVC doesn't support a
>> feature, we should not use it in CPython.
>>
>> In practice, it's easy to check if a feature is supported or not: we
>> have buildbots building Python at each commit. It was very common to
>> get a compilation error only on MSVC when a variable was defined in
>> the middle of a function. We are now using
>> -Werror=declaration-after-statement with GCC because of MSVC!
>>
>> Maybe GCC has an option to ask for the subset of the C99 standard
>> compatible with MSVC? Something like "-std=c99 -pedantic"?
>>
>> Note: I tried -pedantic, GCC emits a lot of warnings on code which
>> looks valid and/or is protected with #ifdef for features specific to
>> GCC like computed goto.
>>
>> Victor
>>
>> 2016-06-07 21:45 GMT+02:00 Guido van Rossum <gvanrossum at gmail.com>:
>> > We should definitely keep supporting MSVC.
>> >
>> > --Guido (mobile)
>> >
>> > On Jun 7, 2016 12:39 PM, "Sturla Molden" <sturla.molden at gmail.com>
>> > wrote:
>> >>
>> >> Victor Stinner <victor.stinner at gmail.com> wrote:
>> >>
>> >> > Is it worth to support a compiler that in 2016 doesn't support the C
>> >> > standard released in 1999, 17 years ago?
>> >>
>> >> MSVC only supports C99 when its needed for C++11 or some MS extension
>> >> to
>> >> C.
>> >>
>> >> Is it worth supporting MSVC? If not, we have Intel C, Clang and Cygwin
>> >> GCC
>> >> are the viable options we have on Windows (and perhaps Embarcadero, but
>> >> I
>> >> haven't used C++ builder for a very long time). Even MinGW does not
>> >> fully
>> >> support C99, because it depends on Microsoft's CRT. If we think MSVC
>> >> and
>> >> MinGW are worth supporting, we cannot just use C99 indiscriminantly.
>> >>
>> >> _______________________________________________
>> >> Python-Dev mailing list
>> >> Python-Dev at python.org
>> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> Unsubscribe:
>> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>> >
>> >
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> >
>> > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>> >
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org



--
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160805/28363411/attachment.html>

From ncoghlan at gmail.com  Fri Aug  5 23:02:56 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 6 Aug 2016 13:02:56 +1000
Subject: [Python-Dev] C99
In-Reply-To: <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
Message-ID: <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>

On 6 August 2016 at 12:15, Steve Dower <steve.dower at python.org> wrote:
> FYI, it's MSVC 14.0 (which is included in VS 2015).
>
> Personally I'd like to see it restricted to the common subset of C99 and
> some version of C++ (which is obviously mostly C and includes no C++),
> because I know there are a few things in C99 only that are very obscure
> because they aren't also in C++.

As a pragmatic requirement, what if we went with:

- must compile with the Python 3.5 Windows compiler (so MSVC 14.0)
- must compile with the Python 3.5 Mac OS X compiler (some version of clang?)
- must compile with the manylinux1 gcc compiler (gcc 4.8.2, as per PEP 513)

The reason I like the pragmatic definition is that it's one we can
readily automate by ensuring these 3 configurations are always present
in the stable buildbot fleet, while the formal standards based version
is technically more precise, but not in a way we can easily test.

It also gives us an objective criterion for when we change the
definitions: when the baseline compiler version for the respective
platform builds change, and Python 3.x versions using the older
definition are no longer supported by python-dev.

Phrasing the policy that way would mean moving future iterations of
the manylinux spec out of distutils-sig only territory into one which
needs to be reviewed by both distutils-sig and python-dev (since it
would now also modify CPython's compiler compatibility requirements in
PEP 7), but I think that's a reasonable position to adopt, and
unlikely to cause any major problems (manylinux is a *trailing* edge
definition that is only likely to get updated as the oldest enterprise
Linux definitions drop out of commercial support)

The other question we need to consider is the possible impact on PEP
11: what happens to support for alternate compilers and platforms that
*have* a buildbot, but may not support particular C99 features? Should
there be a 4th clause that says "- must compile and run on all stable
buildbots"?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From nad at python.org  Fri Aug  5 23:43:20 2016
From: nad at python.org (Ned Deily)
Date: Fri, 5 Aug 2016 23:43:20 -0400
Subject: [Python-Dev] C99
In-Reply-To: <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
Message-ID: <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>

On Aug 5, 2016, at 23:02, Nick Coghlan <ncoghlan at gmail.com> wrote:
> As a pragmatic requirement, what if we went with:
> 
> - must compile with the Python 3.5 Windows compiler (so MSVC 14.0)
> - must compile with the Python 3.5 Mac OS X compiler (some version of clang?)
> - must compile with the manylinux1 gcc compiler (gcc 4.8.2, as per PEP 513)
> 
> The reason I like the pragmatic definition is that it's one we can
> readily automate by ensuring these 3 configurations are always present
> in the stable buildbot fleet, while the formal standards based version
> is technically more precise, but not in a way we can easily test.
> 
> It also gives us an objective criterion for when we change the
> definitions: when the baseline compiler version for the respective
> platform builds change, and Python 3.x versions using the older
> definition are no longer supported by python-dev.
> 
> Phrasing the policy that way would mean moving future iterations of
> the manylinux spec out of distutils-sig only territory into one which
> needs to be reviewed by both distutils-sig and python-dev (since it
> would now also modify CPython's compiler compatibility requirements in
> PEP 7), but I think that's a reasonable position to adopt, and
> unlikely to cause any major problems (manylinux is a *trailing* edge
> definition that is only likely to get updated as the oldest enterprise
> Linux definitions drop out of commercial support)
> 
> The other question we need to consider is the possible impact on PEP
> 11: what happens to support for alternate compilers and platforms that
> *have* a buildbot, but may not support particular C99 features? Should
> there be a 4th clause that says "- must compile and run on all stable
> buildbots"?

[Nick's reply above arrived just as I was getting ready to send my own response below which covers some of the same territory.]

On Aug 5, 2016, at 18:37, Guido van Rossum <gvanrossum at gmail.com> wrote:
> I think if we want to revisit this in the future it should be an
> explicit change.
> On Fri, Aug 5, 2016 at 3:27 PM, Brett Cannon <brett at python.org> wrote:
>> On Fri, 5 Aug 2016 at 15:17 Guido van Rossum <gvanrossum at gmail.com> wrote:
>>> I want it to list specific versions that are known to be good right
>>> now, so we can point fingers appropriately when a regression happens.
>> OK, then we could pin it to MSVC 2016, gcc 6.1, and clang 3.8.1 which are
>> the latest releases (unless I'm missing a compiler we all feel like we
>> should officially support).

While in principle allowing C99 features to be used in cpython is a *good* thing, I'm sure that I don't fully understand all of the implications of adopting this policy for 3.6.  I think we need to be sure we don't inadvertently drop support for some platforms and/or older releases of some platforms.  The key word is "inadvertently".  Not all of those compiler versions above are supported everywhere we currently run.  If that's what we want to do, that's fine but we should be explicit about that.  Two sets of concerns that come to mind are our buildbots and support for older OS X releases.

If I gathered the data correctly, for the default branch (upcoming 3.6 release), we currently have 31 buildbots: 19 classified as "stable" and 12 as "unstable".  Of the stable buildbots, here is the count, version, and platform architecture of compilers in use: 

2 x MSVC 14.0 2015 64 bit (AMD64)
1 x MSVC 14.0 2015 32 bit (AMD64)

1 x Clang 3.8.0 (AMD64)
1 x Clang 3.4.1 (AMD64)

1 x GCC 5.3.1 (s390x)
2 x GCC 4.9.3 (x86)
2 x GCC 4.9.2 (AMD64)
1 x GCC 4.9.2 (PPC64LE)
2 x GCC 4.8.5 (s390x)
1 x GCC 4.8.4 (x86)
1 x GCC 4.8.3 (PPC64)
1 x GCC 4.3.3 (AMD64)
2 x GCC 4.2.1 (AMD64)
1 x GCC 4.0.1 (x86)

I know very little about the details of C99 support in those compilers but I gather from this link (https://gcc.gnu.org/c99status.html) that GCC 4.5 is potentially the earliest level that properly supports C99.  Note that we have a number of buildbots running version of GCC older than 4.5 (including all of our OS X stable buildbots) and only one (an s390) running 5.x and none yet running 6.x.

I don't know that we've written this policy anywhere but I think it has been our de facto policy to require only the "standard" platform build tools (including compilers) to build or install Python on our supported platform versions.  With my OS X hat on, I know this likely presents an issue for the OS X installer builds.  Up to now, we have been providing installers that run on all Macs that are supported back to at least OS X 10.6 and, to do that, we build on 10.6 with the standard Apple build tools available there including GCC 4.2.  If GCC 4.2 proves to no longer be suitable, we'll have to make some adjustment in either what versions of OS X we support and/or how we build python.org Pythons for OS X.  There are several possibilities, and, at some point, we will need to do something anyway.  I am hoping to propose and implement some changes prior to 3.6.0b1 and this discussion will shape that proposal.

I don't think OS X support should be a gating factor in deciding to move ahead with C99 adoption but it does point out that there might be more ramifications to this decision.  What may be more difficult is to judge the impact on other platforms that don't get as much attention from most of us.  For this to move forward, we need to be able to state what the impact to current users will be.

--
  Ned Deily
  nad at python.org -- []


From guido at python.org  Sat Aug  6 00:04:49 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 5 Aug 2016 21:04:49 -0700
Subject: [Python-Dev] C99
In-Reply-To: <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
Message-ID: <CAP7+vJ+MDt0eFCKteco+5NVcymZNavGQaVv-4-P9PKidXwiNBQ@mail.gmail.com>

Different question. What features are we actually talking about? Is it
possible to enumerate them?

The only thing I'm aware of is declarations following non-declarations
in the same block, but I'm no C expert any more.

On Fri, Aug 5, 2016 at 8:43 PM, Ned Deily <nad at python.org> wrote:
> On Aug 5, 2016, at 23:02, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> As a pragmatic requirement, what if we went with:
>>
>> - must compile with the Python 3.5 Windows compiler (so MSVC 14.0)
>> - must compile with the Python 3.5 Mac OS X compiler (some version of clang?)
>> - must compile with the manylinux1 gcc compiler (gcc 4.8.2, as per PEP 513)
>>
>> The reason I like the pragmatic definition is that it's one we can
>> readily automate by ensuring these 3 configurations are always present
>> in the stable buildbot fleet, while the formal standards based version
>> is technically more precise, but not in a way we can easily test.
>>
>> It also gives us an objective criterion for when we change the
>> definitions: when the baseline compiler version for the respective
>> platform builds change, and Python 3.x versions using the older
>> definition are no longer supported by python-dev.
>>
>> Phrasing the policy that way would mean moving future iterations of
>> the manylinux spec out of distutils-sig only territory into one which
>> needs to be reviewed by both distutils-sig and python-dev (since it
>> would now also modify CPython's compiler compatibility requirements in
>> PEP 7), but I think that's a reasonable position to adopt, and
>> unlikely to cause any major problems (manylinux is a *trailing* edge
>> definition that is only likely to get updated as the oldest enterprise
>> Linux definitions drop out of commercial support)
>>
>> The other question we need to consider is the possible impact on PEP
>> 11: what happens to support for alternate compilers and platforms that
>> *have* a buildbot, but may not support particular C99 features? Should
>> there be a 4th clause that says "- must compile and run on all stable
>> buildbots"?
>
> [Nick's reply above arrived just as I was getting ready to send my own response below which covers some of the same territory.]
>
> On Aug 5, 2016, at 18:37, Guido van Rossum <gvanrossum at gmail.com> wrote:
>> I think if we want to revisit this in the future it should be an
>> explicit change.
>> On Fri, Aug 5, 2016 at 3:27 PM, Brett Cannon <brett at python.org> wrote:
>>> On Fri, 5 Aug 2016 at 15:17 Guido van Rossum <gvanrossum at gmail.com> wrote:
>>>> I want it to list specific versions that are known to be good right
>>>> now, so we can point fingers appropriately when a regression happens.
>>> OK, then we could pin it to MSVC 2016, gcc 6.1, and clang 3.8.1 which are
>>> the latest releases (unless I'm missing a compiler we all feel like we
>>> should officially support).
>
> While in principle allowing C99 features to be used in cpython is a *good* thing, I'm sure that I don't fully understand all of the implications of adopting this policy for 3.6.  I think we need to be sure we don't inadvertently drop support for some platforms and/or older releases of some platforms.  The key word is "inadvertently".  Not all of those compiler versions above are supported everywhere we currently run.  If that's what we want to do, that's fine but we should be explicit about that.  Two sets of concerns that come to mind are our buildbots and support for older OS X releases.
>
> If I gathered the data correctly, for the default branch (upcoming 3.6 release), we currently have 31 buildbots: 19 classified as "stable" and 12 as "unstable".  Of the stable buildbots, here is the count, version, and platform architecture of compilers in use:
>
> 2 x MSVC 14.0 2015 64 bit (AMD64)
> 1 x MSVC 14.0 2015 32 bit (AMD64)
>
> 1 x Clang 3.8.0 (AMD64)
> 1 x Clang 3.4.1 (AMD64)
>
> 1 x GCC 5.3.1 (s390x)
> 2 x GCC 4.9.3 (x86)
> 2 x GCC 4.9.2 (AMD64)
> 1 x GCC 4.9.2 (PPC64LE)
> 2 x GCC 4.8.5 (s390x)
> 1 x GCC 4.8.4 (x86)
> 1 x GCC 4.8.3 (PPC64)
> 1 x GCC 4.3.3 (AMD64)
> 2 x GCC 4.2.1 (AMD64)
> 1 x GCC 4.0.1 (x86)
>
> I know very little about the details of C99 support in those compilers but I gather from this link (https://gcc.gnu.org/c99status.html) that GCC 4.5 is potentially the earliest level that properly supports C99.  Note that we have a number of buildbots running version of GCC older than 4.5 (including all of our OS X stable buildbots) and only one (an s390) running 5.x and none yet running 6.x.
>
> I don't know that we've written this policy anywhere but I think it has been our de facto policy to require only the "standard" platform build tools (including compilers) to build or install Python on our supported platform versions.  With my OS X hat on, I know this likely presents an issue for the OS X installer builds.  Up to now, we have been providing installers that run on all Macs that are supported back to at least OS X 10.6 and, to do that, we build on 10.6 with the standard Apple build tools available there including GCC 4.2.  If GCC 4.2 proves to no longer be suitable, we'll have to make some adjustment in either what versions of OS X we support and/or how we build python.org Pythons for OS X.  There are several possibilities, and, at some point, we will need to do something anyway.  I am hoping to propose and implement some changes prior to 3.6.0b1 and this discussion will shape that proposal.
>
> I don't think OS X support should be a gating factor in deciding to move ahead with C99 adoption but it does point out that there might be more ramifications to this decision.  What may be more difficult is to judge the impact on other platforms that don't get as much attention from most of us.  For this to move forward, we need to be able to state what the impact to current users will be.
>
> --
>   Ned Deily
>   nad at python.org -- []
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From songofacandy at gmail.com  Sat Aug  6 01:27:15 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Sat, 6 Aug 2016 14:27:15 +0900
Subject: [Python-Dev] C99
In-Reply-To: <CAP7+vJ+MDt0eFCKteco+5NVcymZNavGQaVv-4-P9PKidXwiNBQ@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <CAP7+vJ+MDt0eFCKteco+5NVcymZNavGQaVv-4-P9PKidXwiNBQ@mail.gmail.com>
Message-ID: <CAEfz+Twc9t2pO4zzO3yyfvAPN0aWWV99YnvSjuPSq1=v3Lkt1Q@mail.gmail.com>

I think these features may improve C code readability.
(Easy feature first).

* // one line comment

* inline function
static inline function can be used instead of may macros.
It is more readable, and type safe.

* Designated Initializers; { .key = value };
https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html

Defining new type may be much shorter.

* stdint.h
https://en.wikibooks.org/wiki/C_Programming/C_Reference/stdint.h

If we can relying to it, we can remove like PY_INT_32

* variadic macro

It may reduce some wrapper functions, but I'm not sure.


On Sat, Aug 6, 2016 at 1:04 PM, Guido van Rossum <guido at python.org> wrote:
> Different question. What features are we actually talking about? Is it
> possible to enumerate them?
>
> The only thing I'm aware of is declarations following non-declarations
> in the same block, but I'm no C expert any more.
>
> On Fri, Aug 5, 2016 at 8:43 PM, Ned Deily <nad at python.org> wrote:
>> On Aug 5, 2016, at 23:02, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> As a pragmatic requirement, what if we went with:
>>>
>>> - must compile with the Python 3.5 Windows compiler (so MSVC 14.0)
>>> - must compile with the Python 3.5 Mac OS X compiler (some version of clang?)
>>> - must compile with the manylinux1 gcc compiler (gcc 4.8.2, as per PEP 513)
>>>
>>> The reason I like the pragmatic definition is that it's one we can
>>> readily automate by ensuring these 3 configurations are always present
>>> in the stable buildbot fleet, while the formal standards based version
>>> is technically more precise, but not in a way we can easily test.
>>>
>>> It also gives us an objective criterion for when we change the
>>> definitions: when the baseline compiler version for the respective
>>> platform builds change, and Python 3.x versions using the older
>>> definition are no longer supported by python-dev.
>>>
>>> Phrasing the policy that way would mean moving future iterations of
>>> the manylinux spec out of distutils-sig only territory into one which
>>> needs to be reviewed by both distutils-sig and python-dev (since it
>>> would now also modify CPython's compiler compatibility requirements in
>>> PEP 7), but I think that's a reasonable position to adopt, and
>>> unlikely to cause any major problems (manylinux is a *trailing* edge
>>> definition that is only likely to get updated as the oldest enterprise
>>> Linux definitions drop out of commercial support)
>>>
>>> The other question we need to consider is the possible impact on PEP
>>> 11: what happens to support for alternate compilers and platforms that
>>> *have* a buildbot, but may not support particular C99 features? Should
>>> there be a 4th clause that says "- must compile and run on all stable
>>> buildbots"?
>>
>> [Nick's reply above arrived just as I was getting ready to send my own response below which covers some of the same territory.]
>>
>> On Aug 5, 2016, at 18:37, Guido van Rossum <gvanrossum at gmail.com> wrote:
>>> I think if we want to revisit this in the future it should be an
>>> explicit change.
>>> On Fri, Aug 5, 2016 at 3:27 PM, Brett Cannon <brett at python.org> wrote:
>>>> On Fri, 5 Aug 2016 at 15:17 Guido van Rossum <gvanrossum at gmail.com> wrote:
>>>>> I want it to list specific versions that are known to be good right
>>>>> now, so we can point fingers appropriately when a regression happens.
>>>> OK, then we could pin it to MSVC 2016, gcc 6.1, and clang 3.8.1 which are
>>>> the latest releases (unless I'm missing a compiler we all feel like we
>>>> should officially support).
>>
>> While in principle allowing C99 features to be used in cpython is a *good* thing, I'm sure that I don't fully understand all of the implications of adopting this policy for 3.6.  I think we need to be sure we don't inadvertently drop support for some platforms and/or older releases of some platforms.  The key word is "inadvertently".  Not all of those compiler versions above are supported everywhere we currently run.  If that's what we want to do, that's fine but we should be explicit about that.  Two sets of concerns that come to mind are our buildbots and support for older OS X releases.
>>
>> If I gathered the data correctly, for the default branch (upcoming 3.6 release), we currently have 31 buildbots: 19 classified as "stable" and 12 as "unstable".  Of the stable buildbots, here is the count, version, and platform architecture of compilers in use:
>>
>> 2 x MSVC 14.0 2015 64 bit (AMD64)
>> 1 x MSVC 14.0 2015 32 bit (AMD64)
>>
>> 1 x Clang 3.8.0 (AMD64)
>> 1 x Clang 3.4.1 (AMD64)
>>
>> 1 x GCC 5.3.1 (s390x)
>> 2 x GCC 4.9.3 (x86)
>> 2 x GCC 4.9.2 (AMD64)
>> 1 x GCC 4.9.2 (PPC64LE)
>> 2 x GCC 4.8.5 (s390x)
>> 1 x GCC 4.8.4 (x86)
>> 1 x GCC 4.8.3 (PPC64)
>> 1 x GCC 4.3.3 (AMD64)
>> 2 x GCC 4.2.1 (AMD64)
>> 1 x GCC 4.0.1 (x86)
>>
>> I know very little about the details of C99 support in those compilers but I gather from this link (https://gcc.gnu.org/c99status.html) that GCC 4.5 is potentially the earliest level that properly supports C99.  Note that we have a number of buildbots running version of GCC older than 4.5 (including all of our OS X stable buildbots) and only one (an s390) running 5.x and none yet running 6.x.
>>
>> I don't know that we've written this policy anywhere but I think it has been our de facto policy to require only the "standard" platform build tools (including compilers) to build or install Python on our supported platform versions.  With my OS X hat on, I know this likely presents an issue for the OS X installer builds.  Up to now, we have been providing installers that run on all Macs that are supported back to at least OS X 10.6 and, to do that, we build on 10.6 with the standard Apple build tools available there including GCC 4.2.  If GCC 4.2 proves to no longer be suitable, we'll have to make some adjustment in either what versions of OS X we support and/or how we build python.org Pythons for OS X.  There are several possibilities, and, at some point, we will need to do something anyway.  I am hoping to propose and implement some changes prior to 3.6.0b1 and this discussion will shape that proposal.
>>
>> I don't think OS X support should be a gating factor in deciding to move ahead with C99 adoption but it does point out that there might be more ramifications to this decision.  What may be more difficult is to judge the impact on other platforms that don't get as much attention from most of us.  For this to move forward, we need to be able to state what the impact to current users will be.
>>
>> --
>>   Ned Deily
>>   nad at python.org -- []
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com



-- 
INADA Naoki  <songofacandy at gmail.com>

From doko at ubuntu.com  Sat Aug  6 06:58:23 2016
From: doko at ubuntu.com (Matthias Klose)
Date: Sat, 6 Aug 2016 12:58:23 +0200
Subject: [Python-Dev] C99
In-Reply-To: <CAP1=2W7RQ5eXtOhkL=+ZV_C7Ri_n9Y2J-6b6htR3tuExPtfqaQ@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <CAP7+vJLtqwxkbm-DA_PQx7gQN1sz4QybO_YZmGW7XhQ6fTFAjg@mail.gmail.com>
 <CAP1=2W7RQ5eXtOhkL=+ZV_C7Ri_n9Y2J-6b6htR3tuExPtfqaQ@mail.gmail.com>
Message-ID: <fc49182b-894f-8dfb-6b69-0e341e87eba6@ubuntu.com>

On 06.08.2016 00:27, Brett Cannon wrote:
> On Fri, 5 Aug 2016 at 15:17 Guido van Rossum <gvanrossum at gmail.com> wrote:
> 
>> I want it to list specific versions that are known to be good right
>> now, so we can point fingers appropriately when a regression happens.
>>
> 
> OK, then we could pin it to MSVC 2016, gcc 6.1, and clang 3.8.1 which are
> the latest releases (unless I'm missing a compiler we all feel like we
> should officially support).

please make that GCC 6.2 once released later this month (6.1 is the first 6.x
release).

GCC 5 and GCC 6 default to C11 (-std=gnu11), does the restriction to C99 mean
that -std=gnu99 should be passed explicitly?


From sturla.molden at gmail.com  Sat Aug  6 07:25:21 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Sat, 6 Aug 2016 11:25:21 +0000 (UTC)
Subject: [Python-Dev] C99
References: <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <CAP7+vJLtqwxkbm-DA_PQx7gQN1sz4QybO_YZmGW7XhQ6fTFAjg@mail.gmail.com>
 <CAP1=2W7RQ5eXtOhkL=+ZV_C7Ri_n9Y2J-6b6htR3tuExPtfqaQ@mail.gmail.com>
 <fc49182b-894f-8dfb-6b69-0e341e87eba6@ubuntu.com>
Message-ID: <1122345322492175262.563031sturla.molden-gmail.com@news.gmane.org>

Matthias Klose <doko at ubuntu.com> wrote:

> GCC 5 and GCC 6 default to C11 (-std=gnu11), does the restriction to C99 mean
> that -std=gnu99 should be passed explicitly?

Also note that -std=c99 is not the same as -std=gnu99. The latter allows
GNU extensions like computed goto. Does the interpreter depend on those?
(Presumably it could be a benefit.)

Sturla


From turnbull.stephen.fw at u.tsukuba.ac.jp  Sat Aug  6 01:16:13 2016
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Sat, 6 Aug 2016 14:16:13 +0900
Subject: [Python-Dev] C99
In-Reply-To: <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
Message-ID: <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>

Ned Deily writes:

 > I don't think OS X support should be a gating factor in deciding to
 > move ahead with C99 adoption but it does point out that there might
 > be more ramifications to this decision.

I may be talking through my hat here, but Apple has been using LLVM
for several major releases now.  They seem to be keeping the GCC
frontend stuck at 4.2.1, though.  So just because we've been using GCC
4.2.1 on Mac, including on buildbots, doesn't mean that there is no
C99 compiler available on Macs -- it's quite possible that the
available clang frontend does already support the C99 features we
would use.  I suppose that might mean fiddling with the installer
build process as well as the buildbots.

Also, it seems like it's been a decade since a C compiler came
standard with Mac OS X -- you need to download and install Xcode or
at least xcodebuildtools separately.  So unless the Macs are quite
ancient (fvo ancient = 5 years old), at least at one time they were
upgradeable by installing Xcode.

Of course Mac users may not be willing or able to upgrade Xcode, but
at least the possibility is worth investigating.

 > What may be more difficult is to judge the impact on other
 > platforms that don't get as much attention from most of us.  For
 > this to move forward, we need to be able to state what the impact
 > to current users will be.

We could post a poll to python-list (to be hosted on surveymonkey or
similar, *not* back to this list!), explaining the proposal and asking
"what is your platform?", "what compiler and version do you use?",
"does it support C99 (perhaps with switches)?", "if not, can you
upgrade to one with C99 support?", and "if you can upgrade, how much
pain would it be to upgrade?"

I guess the last one would take values from "I'd be annoyed for less
than 10 minutes then forget it" to "enough so that I wouldn't upgrade
Python".  I suppose one intermediate answer might be "I'd have to
maintain a separate compiler just for Python."


From nad at python.org  Sat Aug  6 11:36:38 2016
From: nad at python.org (Ned Deily)
Date: Sat, 6 Aug 2016 11:36:38 -0400
Subject: [Python-Dev] C99
In-Reply-To: <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>
Message-ID: <82C8393C-CCF1-4F24-A77F-D6F379F19F3F@python.org>

On Aug 6, 2016, at 01:16, Stephen J. Turnbull <turnbull.stephen.fw at u.tsukuba.ac.jp> wrote:
> I may be talking through my hat here, but Apple has been using LLVM
> for several major releases now.  They seem to be keeping the GCC
> frontend stuck at 4.2.1, though.  So just because we've been using GCC
> 4.2.1 on Mac, including on buildbots, doesn't mean that there is no
> C99 compiler available on Macs -- it's quite possible that the
> available clang frontend does already support the C99 features we
> would use.  I suppose that might mean fiddling with the installer
> build process as well as the buildbots.

Sorry, I wasn't clear; I did not mean to imply there is no C99 compiler available from Apple for OS X.  On current OS X releases, clang is the default and only supported compiler.  I was bringing up the example of the impact on building on older releases with the supported build tools for those releases where clang is either not available or was too immature to be usable.  As I said, there are a number of solutions to that problem - building on newer systems with deployment targets, installing third-party compilers, etc.

But the point I was trying to make is that, by changing the language requirement, we will likely have an effect on what platforms (across the board) and versions we support and we should take that into account when making this decision.  It may be the right decision, in balance, to drop support for some of these but we shouldn't do it by accident.

--
  Ned Deily
  nad at python.org -- []


From nad at python.org  Sat Aug  6 13:15:07 2016
From: nad at python.org (Ned Deily)
Date: Sat, 6 Aug 2016 13:15:07 -0400
Subject: [Python-Dev] C99
In-Reply-To: <CAP7+vJ+MDt0eFCKteco+5NVcymZNavGQaVv-4-P9PKidXwiNBQ@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <CAP7+vJ+MDt0eFCKteco+5NVcymZNavGQaVv-4-P9PKidXwiNBQ@mail.gmail.com>
Message-ID: <BEDFAC1F-AAB0-412D-A6C5-481E50306A26@python.org>

On Aug 6, 2016, at 00:04, Guido van Rossum <guido at python.org> wrote:
> Different question. What features are we actually talking about? Is it
> possible to enumerate them?
> 
> The only thing I'm aware of is declarations following non-declarations
> in the same block, but I'm no C expert any more.

I'm certainly not, either.  Looking back over the original thread, there were a number of features talked about.  I think Larry's list here captures most of them:

"Clearly it'll be 3.5+ only, and clearly it'll be a specific list of 
features ("C89 but also permitting //-comments, variadic macros, 
variable declarations on any line, inline functions, and designated 
initializers").  But I'm looking forward to it!"

https://mail.python.org/pipermail/python-dev/2016-June/144831.html

So, if that set of features seems reasonable, perhaps the next step is for someone (not me) to put together a test patch that incorporates the use of them and then we can run it against the buildbots, either by using the special build buildbots or just temporarily checking the patch in.  I'm willing to do the testing and results-gathering if a patch is supplied.

--
  Ned Deily
  nad at python.org -- []


From brett at python.org  Sat Aug  6 15:03:25 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 06 Aug 2016 19:03:25 +0000
Subject: [Python-Dev] C99
In-Reply-To: <BEDFAC1F-AAB0-412D-A6C5-481E50306A26@python.org>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <CAP7+vJ+MDt0eFCKteco+5NVcymZNavGQaVv-4-P9PKidXwiNBQ@mail.gmail.com>
 <BEDFAC1F-AAB0-412D-A6C5-481E50306A26@python.org>
Message-ID: <CAP1=2W7ORZVj5cdrQatPuoft2YGzgbkhkNnWkWyKRyC+gJy-Ag@mail.gmail.com>

On Sat, 6 Aug 2016 at 10:16 Ned Deily <nad at python.org> wrote:

> On Aug 6, 2016, at 00:04, Guido van Rossum <guido at python.org> wrote:
> > Different question. What features are we actually talking about? Is it
> > possible to enumerate them?
> >
> > The only thing I'm aware of is declarations following non-declarations
> > in the same block, but I'm no C expert any more.
>
> I'm certainly not, either.  Looking back over the original thread, there
> were a number of features talked about.  I think Larry's list here captures
> most of them:
>
> "Clearly it'll be 3.5+ only, and clearly it'll be a specific list of
> features ("C89 but also permitting //-comments, variadic macros,
> variable declarations on any line, inline functions, and designated
> initializers").  But I'm looking forward to it!"
>
> https://mail.python.org/pipermail/python-dev/2016-June/144831.html
>
> So, if that set of features seems reasonable, perhaps the next step is for
> someone (not me) to put together a test patch that incorporates the use of
> them and then we can run it against the buildbots, either by using the
> special build buildbots or just temporarily checking the patch in.  I'm
> willing to do the testing and results-gathering if a patch is supplied.
>

If you want to take that route then you could try temporarily patching one
of the xx* modules as an example.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160806/bb2aa217/attachment.html>

From sturla.molden at gmail.com  Sat Aug  6 17:41:56 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Sat, 6 Aug 2016 21:41:56 +0000 (UTC)
Subject: [Python-Dev] C99
References: <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>
 <82C8393C-CCF1-4F24-A77F-D6F379F19F3F@python.org>
Message-ID: <630763818492212254.856876sturla.molden-gmail.com@news.gmane.org>

Ned Deily <nad at python.org> wrote:

> On Aug 6, 2016, at 01:16, Stephen J. Turnbull <turnbull.stephen.fw at u.tsukuba.ac.jp> wrote:
>> I may be talking through my hat here, but Apple has been using LLVM
>> for several major releases now.  They seem to be keeping the GCC
>> frontend stuck at 4.2.1, though.  So just because we've been using GCC
>> 4.2.1 on Mac, including on buildbots, doesn't mean that there is no
>> C99 compiler available on Macs -- it's quite possible that the
>> available clang frontend does already support the C99 features we
>> would use.  I suppose that might mean fiddling with the installer
>> build process as well as the buildbots.
> 
> Sorry, I wasn't clear; I did not mean to imply there is no C99 compiler
> available from Apple for OS X.  On current OS X releases, clang is the
> default and only supported compiler.  I was bringing up the example of
> the impact on building on older releases with the supported build tools
> for those releases where clang is either not available or was too
> immature to be usable.  As I said, there are a number of solutions to
> that problem - building on newer systems with deployment targets,
> installing third-party compilers, etc.



Clang is also available (and installed) on OSX 10.8 and earlier, although
gcc 4.2.1 is the default frontend to LLVM. The easiest solution to get C99
on those platforms is

$ export CC=clang

Not very difficult, and highly recommended.



Sturla Molden


From sturla.molden at gmail.com  Sat Aug  6 17:55:49 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Sat, 6 Aug 2016 21:55:49 +0000 (UTC)
Subject: [Python-Dev] C99
References: <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>
Message-ID: <207377633492212785.218750sturla.molden-gmail.com@news.gmane.org>

"Stephen J. Turnbull" <turnbull.stephen.fw at u.tsukuba.ac.jp> wrote:

> I may be talking through my hat here, but Apple has been using LLVM
> for several major releases now.  They seem to be keeping the GCC
> frontend stuck at 4.2.1, though.  So just because we've been using GCC
> 4.2.1 on Mac, including on buildbots, doesn't mean that there is no
> C99 compiler available on Macs -- it's quite possible that the
> available clang frontend does already support the C99 features we
> would use.  I suppose that might mean fiddling with the installer
> build process as well as the buildbots.

On OSX 10.8 and earlier, the default CC is llvm-gcc-4.2.1, available as the
gcc command. clang is also installed, so we can always 

    $ export CC=clang
    $ export CXX=clang++

to remedy the problem.

On OSX 10.9 and later, gcc is just a symlink to clang.

Xcode must be installed to build anything on Mac. It is not optional. Users
who need to build Python without installing Xcode need to fix their heads.
Because that is where their problem resides. There is no remedy for
stubbornness to the level of stupidity. Either they install Xcode or they
don't get to build anything. 


Sturla


From nad at python.org  Sat Aug  6 18:09:28 2016
From: nad at python.org (Ned Deily)
Date: Sat, 6 Aug 2016 18:09:28 -0400
Subject: [Python-Dev] C99
In-Reply-To: <207377633492212785.218750sturla.molden-gmail.com@news.gmane.org>
References: <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>
 <207377633492212785.218750sturla.molden-gmail.com@news.gmane.org>
Message-ID: <13787A8D-67B6-41A0-AF66-A046D5C8FF30@python.org>

On Aug 6, 2016, at 17:55, Sturla Molden <sturla.molden at gmail.com> wrote:
> On OSX 10.8 and earlier, the default CC is llvm-gcc-4.2.1, available as the
> gcc command. clang is also installed, so we can always 
[...]

Thanks, but this is getting way off-topic.  I am well aware of what compilers are available on what releases and how to use them.  That's not the issue under discussion here.

--
  Ned Deily
  nad at python.org -- []


From guido at python.org  Sat Aug  6 18:09:29 2016
From: guido at python.org (Guido van Rossum)
Date: Sat, 6 Aug 2016 15:09:29 -0700
Subject: [Python-Dev] C99
In-Reply-To: <207377633492212785.218750sturla.molden-gmail.com@news.gmane.org>
References: <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>
 <207377633492212785.218750sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAP7+vJJaMbKkoCjY+0ywZWHrgndqqqgj+09FbCNYFi11R1zfPw@mail.gmail.com>

On Sat, Aug 6, 2016 at 2:55 PM, Sturla Molden <sturla.molden at gmail.com> wrote:
> Xcode must be installed to build anything on Mac. It is not optional. Users
> who need to build Python without installing Xcode need to fix their heads.
> Because that is where their problem resides. There is no remedy for
> stubbornness to the level of stupidity. Either they install Xcode or they
> don't get to build anything.

This feels close to a code of conduct violation. This kind of language
may be okay on the Linux kernel list, but I don't see the point of it
here.

-- 
--Guido van Rossum (python.org/~guido)

From alexander.belopolsky at gmail.com  Sat Aug  6 18:34:57 2016
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Sat, 6 Aug 2016 18:34:57 -0400
Subject: [Python-Dev] C99
In-Reply-To: <207377633492212785.218750sturla.molden-gmail.com@news.gmane.org>
References: <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>
 <207377633492212785.218750sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAP7h-xbQQuUsjk6mya_i2bGZo0RqT1FzrL1zNi9KQNAuux9Gjg@mail.gmail.com>

On Sat, Aug 6, 2016 at 5:55 PM, Sturla Molden <sturla.molden at gmail.com> wrote:
> Either they install Xcode or they
> don't get to build anything.

I always thought "Command Line Tools" would be enough.

https://developer.apple.com/library/ios/technotes/tn2339/_index.html#//apple_ref/doc/uid/DTS40014588-CH1-WHAT_IS_THE_COMMAND_LINE_TOOLS_PACKAGE_

From sturla.molden at gmail.com  Sat Aug  6 18:47:12 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Sat, 6 Aug 2016 22:47:12 +0000 (UTC)
Subject: [Python-Dev] C99
References: <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>
 <207377633492212785.218750sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJaMbKkoCjY+0ywZWHrgndqqqgj+09FbCNYFi11R1zfPw@mail.gmail.com>
Message-ID: <2109566242492216221.926168sturla.molden-gmail.com@news.gmane.org>

Guido van Rossum <guido at python.org> wrote:

> This feels close to a code of conduct violation. This kind of language
> may be okay on the Linux kernel list, but I don't see the point of it
> here.

Sorry, I should have found a more diplomatic formulation. But the principle
remains, build problems araising from missing Xcode installation should not
be CPython's problem. It is ok to assume that Xcode is always installed
when CPython is built on OSX.


From hugo.fisher at gmail.com  Sat Aug  6 20:08:08 2016
From: hugo.fisher at gmail.com (Hugh Fisher)
Date: Sun, 7 Aug 2016 10:08:08 +1000
Subject: [Python-Dev] C99
Message-ID: <CAAahdHP6vJAU2zuTF8ZqZZ_XH_S6T8kffu4aRioNifvRx4FF3Q@mail.gmail.com>

> Message: 2
> Date: Sat, 6 Aug 2016 14:27:15 +0900
> From: INADA Naoki <songofacandy at gmail.com>
> To: Guido van Rossum <guido at python.org>
> Cc: Ned Deily <nad at python.org>, Python-Dev <python-dev at python.org>
> Subject: Re: [Python-Dev] C99
> Message-ID:
>         <CAEfz+Twc9t2pO4zzO3yyfvAPN0aWWV99YnvSjuPSq1=v3Lkt1Q at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> I think these features may improve C code readability.
> (Easy feature first).
>
> * // one line comment
>
> * inline function
> static inline function can be used instead of may macros.
> It is more readable, and type safe.

My experience from a few months ago with some cross-platform code is
that clang, GCC and MSVC have different ideas about how inline
functions in C work. Are they static or extern? Do you need provide an
implementation body somewhere?

There are also differences between the C++ and C99 definitions.

Probably all solvable with build settings, but not as easy as one line comments.

-- 

        cheers,
        Hugh Fisher

From songofacandy at gmail.com  Sat Aug  6 22:36:04 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Sun, 7 Aug 2016 11:36:04 +0900
Subject: [Python-Dev] C99
In-Reply-To: <CAAahdHP6vJAU2zuTF8ZqZZ_XH_S6T8kffu4aRioNifvRx4FF3Q@mail.gmail.com>
References: <CAAahdHP6vJAU2zuTF8ZqZZ_XH_S6T8kffu4aRioNifvRx4FF3Q@mail.gmail.com>
Message-ID: <CAEfz+TxyNGTd8zLhQYWX+Lbd_g4qWXQd54jbwb98yXopMv0_uQ@mail.gmail.com>

>>
>> * inline function
>> static inline function can be used instead of may macros.
>> It is more readable, and type safe.
>
> My experience from a few months ago with some cross-platform code is
> that clang, GCC and MSVC have different ideas about how inline
> functions in C work. Are they static or extern? Do you need provide an
> implementation body somewhere?

You're right. I think inline function should be used as "static inline" always.

Python uses "static inline" already, via "Py_LOCAL_INLINE" macro, when
USE_INLINE is defined. So using "static inline" is not problem.

If we can assume inline is usable, we can remove the macro, and use
"static inline"
more often to replace macros.

From stefan at bytereef.org  Sun Aug  7 06:59:58 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Sun, 7 Aug 2016 12:59:58 +0200
Subject: [Python-Dev] libmpdec already uses and compiles with -std=c99
 -pedantic everywhere
Message-ID: <20160807105958.GA3788@bytereef.org>


The following C99 features are supported by any buildbot compiler (including
obscure ones on snakebite):

  - stdint.h

  - static inline functions in header files

  - extern inline functions (not really necessary for CPython)

  - variadic macros


I'd like to add:

  - standard conforming C99 spelling of the "struct hack" (supported by gcc
    since I can remember).  Our current way leads to problems for some
    analyzers.


Some people (not me) would like:

  - variable declarations everywhere (rather than only at the beginning
    of blocks).



I think this would probably be enough for the start. Obviously things
like variable-length arrays should never be used anyway.



Stefan Krah





From brett at python.org  Sun Aug  7 13:07:37 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 07 Aug 2016 17:07:37 +0000
Subject: [Python-Dev] libmpdec already uses and compiles with -std=c99
 -pedantic everywhere
In-Reply-To: <20160807105958.GA3788@bytereef.org>
References: <20160807105958.GA3788@bytereef.org>
Message-ID: <CAP1=2W7XtcPq+=2KixjQucMy31S-p_vw8LLDkdhLJ3WwfYWuCg@mail.gmail.com>

On Sun, 7 Aug 2016 at 08:24 Stefan Krah <stefan at bytereef.org> wrote:

>
> The following C99 features are supported by any buildbot compiler
> (including
> obscure ones on snakebite):
>
>   - stdint.h
>
>   - static inline functions in header files
>
>   - extern inline functions (not really necessary for CPython)
>
>   - variadic macros
>
>
> I'd like to add:
>
>   - standard conforming C99 spelling of the "struct hack" (supported by gcc
>     since I can remember).  Our current way leads to problems for some
>     analyzers.
>
>
> Some people (not me) would like:
>
>   - variable declarations everywhere (rather than only at the beginning
>     of blocks).
>
>
>
> I think this would probably be enough for the start. Obviously things
> like variable-length arrays should never be used anyway.
>

Thanks for the list, Stefan! The only thing I would want to add is:

- bool type
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160807/7acab0a3/attachment.html>

From turnbull.stephen.fw at u.tsukuba.ac.jp  Mon Aug  8 04:02:45 2016
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Mon, 8 Aug 2016 17:02:45 +0900
Subject: [Python-Dev] C99
In-Reply-To: <82C8393C-CCF1-4F24-A77F-D6F379F19F3F@python.org>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <0C0E7774-301B-4FD0-9BA6-7C9D03125F37@python.org>
 <22437.29341.73802.749116@turnbull.sk.tsukuba.ac.jp>
 <82C8393C-CCF1-4F24-A77F-D6F379F19F3F@python.org>
Message-ID: <22440.15525.391255.261426@turnbull.sk.tsukuba.ac.jp>

Ned Deily writes:

 > But the point I was trying to make is that, by changing the
 > language requirement, we will likely have an effect on what
 > platforms (across the board) and versions we support and we should
 > take that into account when making this decision.  It may be the
 > right decision, in balance, to drop support for some of these but
 > we shouldn't do it by accident.

Sure, you were clear enough about that.  My point was simply that at
least for older Macs it probably is not that big a problem (but I do
have a Panther still running, at least as of March it did ;-).

Similarly, for platforms where we build with GCC, many of these
features have been available for a long time with switches.

Adding it all up, we don't want to break anybody inadvertantly and we
should take care to fix what breakage we can in advance, but I think
it's time to allow at least some of these features, and maybe move to
C99 wholesale.

Steve

From tim at brew.sh  Mon Aug  8 02:18:16 2016
From: tim at brew.sh (tdsmith)
Date: Sun, 7 Aug 2016 23:18:16 -0700
Subject: [Python-Dev] Review request: issue 10910,
 pyport.h causes trouble for C++ extensions on BSDs
Message-ID: <CAGF0r4az4R1O=7iqwRdOsdSqJSrtLRDRTjkFeRdTzv-OPgQF1g@mail.gmail.com>

Hi python-dev! I'm a maintainer for Homebrew, a third-party package
manager for OS X, where I'm the resident parseltongue.

Issue 10910 is related to problems building CPython extension modules
with C++ code on BSDs. As I understand it, pyport.h has some code
specific to BSD and friends, including OS X, which causes conflicts
with the C++ stdlib.

We've been carrying the patch Ronald Oussoren wrote in 2011 [2]
against Python 2.7 since olden times. We were recently prompted to add
the patch to our 3.5 package as well [3] because the bug was causing
build problems in the wild. [4]

We strive to apply as few patches as possible in Homebrew and we (I)
would love to see a fix for this deployed upstream. Can I do anything
to help code get checked in?

Thanks,
Tim

[1] https://bugs.python.org/issue10910
[2] https://bugs.python.org/issue10910#msg135414
[3] https://github.com/Homebrew/homebrew-core/pull/3396
[4] https://github.com/IntelPNI/brainiak/pull/82

-- 
Tim Smith
Freenode: tdsmith, #machomebrew
https://tds.xyz, https://github.com/tdsmith

From nad at python.org  Mon Aug  8 11:59:07 2016
From: nad at python.org (Ned Deily)
Date: Mon, 8 Aug 2016 11:59:07 -0400
Subject: [Python-Dev] [python-committers] Failed to build select
In-Reply-To: <CAOTb1wfj3YT5xhb5isefsp7s4Bx3R7rq_0vNVofGCHxspwTqNw@mail.gmail.com>
References: <20160808025804.GB6608@ando.pearwood.info>
 <CAPOVWORHR4m0__KDGwdtM_5Gkf7zjVcpDx4c3hNrKeQ+EfzK1A@mail.gmail.com>
 <8DBDE981-842A-4869-A573-9FF75262B4AC@python.org>
 <20160808064544.GA31842@ando.pearwood.info>
 <CAOTb1wfj3YT5xhb5isefsp7s4Bx3R7rq_0vNVofGCHxspwTqNw@mail.gmail.com>
Message-ID: <253B7A63-BE05-48B0-AD3B-F006D1B78A36@python.org>

On Aug 8, 2016, at 02:45, Steven D'Aprano <steve at pearwood.info> wrote:
> On Mon, Aug 08, 2016 at 12:17:21AM -0400, Ned Deily wrote:
>> Also, try without setting PYTHONHOME.  I'm not sure what you're trying to do by setting that but you shouldn't need to.
> I didn't think I needed to either, but when I try without it, I get:
> 
> Could not find platform dependent libraries <exec_prefix>
> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
> Could not find platform dependent libraries <exec_prefix>
> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]



On Aug 8, 2016, at 03:25, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
> FWIW, I would be interested in learning more about the above warning
> (its significance, how it can be addressed, whether it can be ignored,
> etc). I also get this message when installing 3.5.2 from source on
> Ubuntu 14.04.

Those messages are harmless and are generated by the Makefile steps that update Importlib's bootstrap files, Python/importlib_external.h and Python/importlib_external.h.  See http://bugs.python.org/issue14928 for the origins of this.  It should be possible to fix the Makefile to suppress those messages.  I suggest you open an issue about it.

--
  Ned Deily
  nad at python.org -- []


From brett at python.org  Mon Aug  8 12:55:44 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 08 Aug 2016 16:55:44 +0000
Subject: [Python-Dev] Review request: issue 10910,
 pyport.h causes trouble for C++ extensions on BSDs
In-Reply-To: <CAGF0r4az4R1O=7iqwRdOsdSqJSrtLRDRTjkFeRdTzv-OPgQF1g@mail.gmail.com>
References: <CAGF0r4az4R1O=7iqwRdOsdSqJSrtLRDRTjkFeRdTzv-OPgQF1g@mail.gmail.com>
Message-ID: <CAP1=2W72NQ2M8TtpOzQTLzsVT5UftAiJNXOFP5LePieAa23KpQ@mail.gmail.com>

On Mon, 8 Aug 2016 at 08:10 tdsmith <tim at brew.sh> wrote:

> Hi python-dev! I'm a maintainer for Homebrew, a third-party package
> manager for OS X, where I'm the resident parseltongue.
>
> Issue 10910 is related to problems building CPython extension modules
> with C++ code on BSDs. As I understand it, pyport.h has some code
> specific to BSD and friends, including OS X, which causes conflicts
> with the C++ stdlib.
>
> We've been carrying the patch Ronald Oussoren wrote in 2011 [2]
> against Python 2.7 since olden times. We were recently prompted to add
> the patch to our 3.5 package as well [3] because the bug was causing
> build problems in the wild. [4]
>
> We strive to apply as few patches as possible in Homebrew and we (I)
> would love to see a fix for this deployed upstream. Can I do anything
> to help code get checked in?
>

The trick is someone feeling up to the task of knowing enough C, C++, and
what's happening on OS X/BSD to validate the patch and apply it. Usually
that's Ronald or Ned and Ronald never applied his patch, so I guess that
leaves Ned. :)

If Ned doesn't have the time to look then just ping the issue in a week and
I will apply it since both you and FreeBSD are already carrying the patch
forward.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160808/c67f766e/attachment.html>

From chris.jerdonek at gmail.com  Mon Aug  8 14:08:52 2016
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 8 Aug 2016 11:08:52 -0700
Subject: [Python-Dev] [python-committers] Failed to build select
In-Reply-To: <253B7A63-BE05-48B0-AD3B-F006D1B78A36@python.org>
References: <20160808025804.GB6608@ando.pearwood.info>
 <CAPOVWORHR4m0__KDGwdtM_5Gkf7zjVcpDx4c3hNrKeQ+EfzK1A@mail.gmail.com>
 <8DBDE981-842A-4869-A573-9FF75262B4AC@python.org>
 <20160808064544.GA31842@ando.pearwood.info>
 <CAOTb1wfj3YT5xhb5isefsp7s4Bx3R7rq_0vNVofGCHxspwTqNw@mail.gmail.com>
 <253B7A63-BE05-48B0-AD3B-F006D1B78A36@python.org>
Message-ID: <CAOTb1wds7QaRz9BZfEZwBPDS0OPW4KNnM1XgvHymj3AJTqCymw@mail.gmail.com>

On Mon, Aug 8, 2016 at 8:59 AM, Ned Deily <nad at python.org> wrote:
> On Aug 8, 2016, at 02:45, Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> Could not find platform dependent libraries <exec_prefix>
>> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
>> Could not find platform dependent libraries <exec_prefix>
>> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
>
> On Aug 8, 2016, at 03:25, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
>> FWIW, I would be interested in learning more about the above warning
>> (its significance, how it can be addressed, whether it can be ignored,
>> etc). I also get this message when installing 3.5.2 from source on
>> Ubuntu 14.04.
>
> Those messages are harmless and are generated by the Makefile steps that update Importlib's bootstrap files, Python/importlib_external.h and Python/importlib_external.h.  See http://bugs.python.org/issue14928 for the origins of this.  It should be possible to fix the Makefile to suppress those messages.  I suggest you open an issue about it.

I created an issue for this here: http://bugs.python.org/issue27713

--Chris

From g.rodola at gmail.com  Mon Aug  8 15:33:16 2016
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Mon, 8 Aug 2016 21:33:16 +0200
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
Message-ID: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>

import timeit
import contextlib

@contextlib.contextmanager
def ctx1():
    yield

class ctx2:
    def __enter__(self):
        pass
    def __exit__(self, *args):
        pass

t1 = timeit.timeit("with ctx1(): pass", setup="from __main__ import ctx1")
t2 = timeit.timeit("with ctx2(): pass", setup="from __main__ import ctx2")
print("%.3f secs" % t1)
print("%.3f secs" % t2)
print("slowdown: -%.2fx" % (t1 / t2))


...with Python 3.5:

1.938 secs
0.443 secs
slowdown: -4.37x

I wanted to give it a try rewriting this in C but since @contextmanager has
a lot of magic I wanted to ask first whether this 1) is technically
possible 2) is desirable.
Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160808/d11013cd/attachment.html>

From yselivanov.ml at gmail.com  Mon Aug  8 16:07:25 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Mon, 8 Aug 2016 16:07:25 -0400
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
Message-ID: <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>



On 2016-08-08 3:33 PM, Giampaolo Rodola' wrote:
> I wanted to give it a try rewriting this in C but since 
> @contextmanager has a lot of magic I wanted to ask first whether this 
> 1) is technically possible 2) is desirable.

It is definitely technologically possible.  However, the C 
implementation will be quite complex, and will require a lot of time to 
review and later maintain.  My question would be how critical is the 
performance of @contextmanager?  I'd say that unless it's used in a 
tight loop it can't affect the performance too much.

Yury

From guido at python.org  Mon Aug  8 16:18:37 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Aug 2016 13:18:37 -0700
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
Message-ID: <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>

I think Nick would be interested in understanding why this is the case.
What does the decorator do that could be so expensive?

On Mon, Aug 8, 2016 at 1:07 PM, Yury Selivanov <yselivanov.ml at gmail.com>
wrote:

>
>
> On 2016-08-08 3:33 PM, Giampaolo Rodola' wrote:
>
>> I wanted to give it a try rewriting this in C but since @contextmanager
>> has a lot of magic I wanted to ask first whether this 1) is technically
>> possible 2) is desirable.
>>
>
> It is definitely technologically possible.  However, the C implementation
> will be quite complex, and will require a lot of time to review and later
> maintain.  My question would be how critical is the performance of
> @contextmanager?  I'd say that unless it's used in a tight loop it can't
> affect the performance too much.
>
> Yury
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160808/745383af/attachment.html>

From g.rodola at gmail.com  Mon Aug  8 16:19:03 2016
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Mon, 8 Aug 2016 22:19:03 +0200
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
Message-ID: <CAFYqXL-Ofek3OmR5JrKHLyRyGJ_L1fTJefYDt48JA2gsQmZgGg@mail.gmail.com>

On Mon, Aug 8, 2016 at 10:07 PM, Yury Selivanov <yselivanov.ml at gmail.com>
wrote:

>
>
> On 2016-08-08 3:33 PM, Giampaolo Rodola' wrote:
>
>> I wanted to give it a try rewriting this in C but since @contextmanager
>> has a lot of magic I wanted to ask first whether this 1) is technically
>> possible 2) is desirable.
>>
>
> It is definitely technologically possible.  However, the C implementation
> will be quite complex, and will require a lot of time to review and later
> maintain.  My question would be how critical is the performance of
> @contextmanager?  I'd say that unless it's used in a tight loop it can't
> affect the performance too much.
>

Yeah, the (my) use case is exactly that (tight loops).


-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160808/c0f62f50/attachment.html>

From yselivanov.ml at gmail.com  Mon Aug  8 16:38:51 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Mon, 8 Aug 2016 16:38:51 -0400
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
Message-ID: <f6cc59f0-3657-a9c0-5755-099be4c972d0@gmail.com>



On 2016-08-08 4:18 PM, Guido van Rossum wrote:
> I think Nick would be interested in understanding why this is the 
> case. What does the decorator do that could be so expensive?

 From the looks of it it doesn't do anything special.  Although with 
@contextlib.contextmanager we have to instantiate a generator (the 
decorated one) and advance it in __enter__.  So it's an extra object 
instantiation + extra code in __enter__ and __exit__.  Anyways, Nick 
knows much more about that code.

Giampaolo, before experimenting with a C implementation, I suggest you 
to try to compile contextlib.py with Cython.  I'll be surprised if you 
can make it more than 30-40% faster.  And you won't get much faster than 
Cython when you code contextmanager in C by hand.

Also, we don't have slots for __enter__ and __exit__, so there is no way 
to avoid the attribute lookup.

Yury

From wolfgang.maier at biologie.uni-freiburg.de  Mon Aug  8 17:14:59 2016
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Mon, 8 Aug 2016 23:14:59 +0200
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <f6cc59f0-3657-a9c0-5755-099be4c972d0@gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
 <f6cc59f0-3657-a9c0-5755-099be4c972d0@gmail.com>
Message-ID: <noasom$kvm$1@blaine.gmane.org>

On 8/8/2016 22:38, Yury Selivanov wrote:
>
>
> On 2016-08-08 4:18 PM, Guido van Rossum wrote:
>> I think Nick would be interested in understanding why this is the
>> case. What does the decorator do that could be so expensive?
>
> From the looks of it it doesn't do anything special.  Although with
> @contextlib.contextmanager we have to instantiate a generator (the
> decorated one) and advance it in __enter__.  So it's an extra object
> instantiation + extra code in __enter__ and __exit__.  Anyways, Nick
> knows much more about that code.
>

Right, I think a fairer comparison would be to:

class ctx2:
     def __enter__(self):
         self.it = iter(self)
         return next(self.it)

     def __exit__(self, *args):
         try:
             next(self.it)
         except StopIteration:
             pass

     def __iter__(self):
         yield

With this change alone the slowdown diminishes to ~ 1.7x for me. The 
rest is probably the extra overhead for being able to pass exceptions 
raised inside the with block back into the generator and such.



From rosuav at gmail.com  Mon Aug  8 17:59:39 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 9 Aug 2016 07:59:39 +1000
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <noasom$kvm$1@blaine.gmane.org>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
 <f6cc59f0-3657-a9c0-5755-099be4c972d0@gmail.com>
 <noasom$kvm$1@blaine.gmane.org>
Message-ID: <CAPTjJmoi5LSPSuJoH9tOp-=k+F9S5yWEMBYxgfXvNcFu_GzY=Q@mail.gmail.com>

On Tue, Aug 9, 2016 at 7:14 AM, Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de> wrote:
> Right, I think a fairer comparison would be to:
>
> class ctx2:
>     def __enter__(self):
>         self.it = iter(self)
>         return next(self.it)
>
>     def __exit__(self, *args):
>         try:
>             next(self.it)
>         except StopIteration:
>             pass
>
>     def __iter__(self):
>         yield
>
> With this change alone the slowdown diminishes to ~ 1.7x for me. The rest is
> probably the extra overhead for being able to pass exceptions raised inside
> the with block back into the generator and such.

I played around with a few other variants to see where the slowdown
is. They all work out pretty much the same as the above; my two
examples are both used the same way as contextlib.contextmanager is,
but are restrictive on what you can do.

import timeit
import contextlib
import functools

class ctx1:
    def __enter__(self):
        pass
    def __exit__(self, *args):
        pass

@contextlib.contextmanager
def ctx2():
    yield

class SimplerContextManager:
    """Like contextlib._GeneratorContextManager but way simpler.

    * Doesn't reinstantiate itself - just reinvokes the generator
    * Doesn't allow yielded objects (returns self)
    * Lacks a lot of error checking. USE ONLY AS DIRECTED.
    """
    def __init__(self, func):
        self.func = func
        functools.update_wrapper(self, func)
    def __call__(self, *a, **kw):
        self.gen = self.func(*a, **kw)
        return self
    def __enter__(self):
        next(self.gen)
        return self
    def __exit__(self, type, value, traceback):
        if type is None:
            try: next(self.gen)
            except StopIteration: return
            else: raise RuntimeError("generator didn't stop")
        try: self.gen.throw(type, value, traceback)
        except StopIteration: return True
        # Assume any instance of the same exception type is a proper reraise
        # This is way simpler than contextmanager normally does, and costs us
        # the ability to detect exception handlers that coincidentally raise
        # the same type of error (eg "except ZeroDivisionError: print(1/0)").
        except type: return False

# Check that it actually behaves correctly
@SimplerContextManager
def ctxdemo():
    print("Before yield")
    try:
        yield 123
    except ZeroDivisionError:
        print("Absorbing 1/0")
        return
    finally:
        print("Finalizing")
    print("After yield (no exception)")

with ctxdemo() as val:
    print("1/0 =", 1/0)
with ctxdemo() as val:
    print("1/1 =", 1/1)
#with ctxdemo() as val:
#    print("1/q =", 1/q)

@SimplerContextManager
def ctx3():
    yield

class TooSimpleContextManager:
    """Now this time you've gone too far."""
    def __init__(self, func):
        self.func = func
    def __call__(self):
        self.gen = self.func()
        return self
    def __enter__(self):
        next(self.gen)
    def __exit__(self, type, value, traceback):
        try: next(self.gen)
        except StopIteration: pass

@TooSimpleContextManager
def ctx4():
    yield

class ctx5:
    def __enter__(self):
        self.it = iter(self)
        return next(self.it)

    def __exit__(self, *args):
        try:
            next(self.it)
        except StopIteration:
            pass

    def __iter__(self):
        yield

t1 = timeit.timeit("with ctx1(): pass", setup="from __main__ import ctx1")
print("%.3f secs" % t1)

for i in range(2, 6):
    t2 = timeit.timeit("with ctx2(): pass", setup="from __main__
import ctx%d as ctx2"%i)
    print("%.3f secs" % t2)
    print("slowdown: -%.2fx" % (t2 / t1))


My numbers are:

0.320 secs
1.354 secs
slowdown: -4.23x
0.899 secs
slowdown: -2.81x
0.831 secs
slowdown: -2.60x
0.868 secs
slowdown: -2.71x

So compared to the tight custom-written context manager class, all the
"pass it a generator function" varieties look pretty much the same.
The existing contextmanager factory has several levels of indirection,
and that's probably where the rest of the performance difference comes
from, but there is some cost to the simplicity of the gen-func
approach.

My guess is that a C-implemented version could replace piles of
error-handling code with simple pointer comparisons (hence my
elimination of it), and may or may not be able to remove some of the
indirection. I'd say it'd land about in the same range as the other
examples here. Is that worth it?

ChrisA

From victor.stinner at gmail.com  Mon Aug  8 18:25:36 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Aug 2016 00:25:36 +0200
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
Message-ID: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>

Hi,

tl;dr I found a way to make CPython 3.6 faster and I validated that
there is no performance regression. I'm requesting approval of core
developers to start pushing changes.

In 2014 during a lunch at Pycon, Larry Hasting told me that he would
like to get rid of temporary tuples to call functions in Python. In
Python, positional arguments are passed as a tuple to C functions:
"PyObject *args". Larry wrote Argument Clinic which gives more control
on how C functions are called. But I guess that Larry didn't have time
to finish his implementation, since he didn't publish a patch.

While trying to optimize CPython 3.6, I wrote a proof-of-concept patch
and results were promising:
https://bugs.python.org/issue26814#msg264003
https://bugs.python.org/issue26814#msg266359

C functions get a C array "PyObject **args, int nargs". Getting the
nth argument become "arg = args[n];" at the C level. This format is
not new, it's already used internally in Python/ceval.c. A Python
function call made from a Python function already avoids a temporary
tuple in most cases: we pass the stack of the first function as the
list of arguments to the second function. My patch generalizes the
idea to C functions. It works in all directions (C=>Python, Python=>C,
C=>C, etc.).

Many function calls become faster than Python 3.5 with my full patch,
but even faster than Python 2.7! For multiple reasons (not interesting
here), tested functions are slower in Python 3.4 than Python 2.7.
Python 3.5 is better than Python 3.4, but still slower than Python 2.7
in a few cases. Using my "FASTCALL" patch, all tested function calls
become faster or as fast as Python 2.7!

But when I ran the CPython benchmark suite, I found some major
performance regressions. In fact, it took me 3 months to understand
that I didn't run benchmarks correctly and that most benchmarks of the
CPython benchmark suite are very unstable. I wrote articles explaining
how benchmarks should be run (to be stable) and I patched all
benchmarks to use my new perf module which runs benchmarks in multiple
processes and computes the average (to make benchmarks more stable).

At the end, my minimum FASTCALL patch (issue #27128) doesn't show any
major performance regression if you run "correctly" benchmarks :-)
https://bugs.python.org/issue27128#msg272197

Most benchmarks are not significant, 14 are faster, and only 4 are slower.

According to benchmarks on the "full" FASTCALL patch, the slowdown are
temporary and should become quickly speedup (with further changes).

My question is now: can I push fastcall-2.patch of the issue #27128?
This patch only adds the infrastructure to start working on more
useful optimizations, more patches will come, I expect more exciting
benchmark results.

Overview of the initial FASTCALL patch, see my first message on the issue:
https://bugs.python.org/issue27128#msg266422

--

Note: My full FASTCALL patch changes the C API: this is out of the
scope of my first simple FASTCALL patch. I will open a new discussion
to decide if it's worth it and if yes, how it should be done.

Victor

From guido at python.org  Mon Aug  8 18:40:23 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Aug 2016 15:40:23 -0700
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
In-Reply-To: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
References: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
Message-ID: <CAP7+vJJoVtetLV40qKmG2oCXSTxjnsehdJ7Fs95Ux4CCK5ecHg@mail.gmail.com>

On Mon, Aug 8, 2016 at 3:25 PM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> tl;dr I found a way to make CPython 3.6 faster and I validated that
> there is no performance regression.


But is there a performance improvement?


> I'm requesting approval of core
> developers to start pushing changes.
>
> In 2014 during a lunch at Pycon, Larry Hasting told me that he would
> like to get rid of temporary tuples to call functions in Python. In
> Python, positional arguments are passed as a tuple to C functions:
> "PyObject *args". Larry wrote Argument Clinic which gives more control
> on how C functions are called. But I guess that Larry didn't have time
> to finish his implementation, since he didn't publish a patch.
>

Hm, I agree that those tuples are probably expensive. I recall that
IronPython boasted faster Python calls by doing something closer to the
platform (in their case I'm guessing C# or the CLR :-).

Is this perhaps something that could wait until the Core devs sprint in a
few weeks? (I presume you're coming?!)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160808/ff25fd34/attachment.html>

From victor.stinner at gmail.com  Mon Aug  8 18:53:43 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Aug 2016 00:53:43 +0200
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
In-Reply-To: <CAP7+vJJoVtetLV40qKmG2oCXSTxjnsehdJ7Fs95Ux4CCK5ecHg@mail.gmail.com>
References: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
 <CAP7+vJJoVtetLV40qKmG2oCXSTxjnsehdJ7Fs95Ux4CCK5ecHg@mail.gmail.com>
Message-ID: <CAMpsgwbDR_-ENt=SDWFkdKNO-KTDsiu--C-29wcs_FoDURrPMA@mail.gmail.com>

2016-08-09 0:40 GMT+02:00 Guido van Rossum <guido at python.org>:
>> tl;dr I found a way to make CPython 3.6 faster and I validated that
>> there is no performance regression.
>
> But is there a performance improvement?

Sure.


On micro-benchmarks, you can see nice improvements:

* getattr(1, "real") becomes 44% faster
* list(filter(lambda x: x, list(range(1000)))) becomes 31% faster
* namedtuple.attr becomes -23% faster
* etc.

See https://bugs.python.org/issue26814#msg263999 for default => patch,
or https://bugs.python.org/issue26814#msg264003 for comparison python
2.7 / 3.4 / 3.5 / 3.6 / 3.6 patched.


On the CPython benchmark suite, I also saw many faster benchmarks:

Faster (25):
- pickle_list: 1.29x faster
- etree_generate: 1.22x faster
- pickle_dict: 1.19x faster
- etree_process: 1.16x faster
- mako_v2: 1.13x faster
- telco: 1.09x faster
- raytrace: 1.08x faster
- etree_iterparse: 1.08x faster
(...)

See https://bugs.python.org/issue26814#msg266359

Victor

From victor.stinner at gmail.com  Mon Aug  8 19:31:48 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Aug 2016 01:31:48 +0200
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
In-Reply-To: <CAP7+vJJoVtetLV40qKmG2oCXSTxjnsehdJ7Fs95Ux4CCK5ecHg@mail.gmail.com>
References: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
 <CAP7+vJJoVtetLV40qKmG2oCXSTxjnsehdJ7Fs95Ux4CCK5ecHg@mail.gmail.com>
Message-ID: <CAMpsgwa3c99pTFXv5njT4KALPsqZO_AOQ3GxGz7=gnk8jBadAA@mail.gmail.com>

2016-08-09 0:40 GMT+02:00 Guido van Rossum <guido at python.org>:
> Hm, I agree that those tuples are probably expensive. I recall that
> IronPython boasted faster Python calls by doing something closer to the
> platform (in their case I'm guessing C# or the CLR :-).

To be honest, I didn't expect *any* speedup just by avoiding the
temporary tuples. The C structore of tuples is simple and the
allocation of tuples is already optimized by a free list. I still
don't understand how the cost of tuple creation/destruction can have
such "large" impact on performances.

The discussion with Larry was not really my first motivation to work
on FASTCALL.

I worked on this topic because CPython already uses some "hidden"
tuples to avoid the cost of the tuple creation/destruction in various
places, but using really ugly code and this ugly code caused crashes
and surprising behaviours...

https://bugs.python.org/issue26811 is a recent crash related to
property_descr_get() optimization, whereas the optimization was
already "fixed" once:
https://hg.python.org/cpython/rev/5dbf3d932a59/

The fix is just another hack on top of the existing hack. The issue
#26811 rewrote the optimization to avoid the crash using
_PyObject_GC_UNTRACK():
https://hg.python.org/cpython/rev/a98ef122d73d

I tried to make this "optimization" the standard way to call
functions, rather than a corner case, and avoid hacks like
PyTuple_SET_ITEM(args, 0, NULL) or _PyObject_GC_UNTRACK().

Victor

From brett at python.org  Mon Aug  8 19:36:34 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 08 Aug 2016 23:36:34 +0000
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
In-Reply-To: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
References: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
Message-ID: <CAP1=2W5HTz-7CRT_x=jCY8yh=3v+LG3uSP_qkhsHj=dNBERCBg@mail.gmail.com>

I just wanted to say I'm excited about this and I'm glad someone is taking
advantage of what Argument Clinic allows for and what I know Larry had
initially hoped AC would make happen!

I should also point out that Serhiy has a patch for faster keyword argument
parsing thanks to AC: http://bugs.python.org/issue27574 . Not sure how your
two patches would intertwine (if at all).

On Mon, 8 Aug 2016 at 15:26 Victor Stinner <victor.stinner at gmail.com> wrote:

> Hi,
>
> tl;dr I found a way to make CPython 3.6 faster and I validated that
> there is no performance regression. I'm requesting approval of core
> developers to start pushing changes.
>
> In 2014 during a lunch at Pycon, Larry Hasting told me that he would
> like to get rid of temporary tuples to call functions in Python. In
> Python, positional arguments are passed as a tuple to C functions:
> "PyObject *args". Larry wrote Argument Clinic which gives more control
> on how C functions are called. But I guess that Larry didn't have time
> to finish his implementation, since he didn't publish a patch.
>
> While trying to optimize CPython 3.6, I wrote a proof-of-concept patch
> and results were promising:
> https://bugs.python.org/issue26814#msg264003
> https://bugs.python.org/issue26814#msg266359
>
> C functions get a C array "PyObject **args, int nargs". Getting the
> nth argument become "arg = args[n];" at the C level. This format is
> not new, it's already used internally in Python/ceval.c. A Python
> function call made from a Python function already avoids a temporary
> tuple in most cases: we pass the stack of the first function as the
> list of arguments to the second function. My patch generalizes the
> idea to C functions. It works in all directions (C=>Python, Python=>C,
> C=>C, etc.).
>
> Many function calls become faster than Python 3.5 with my full patch,
> but even faster than Python 2.7! For multiple reasons (not interesting
> here), tested functions are slower in Python 3.4 than Python 2.7.
> Python 3.5 is better than Python 3.4, but still slower than Python 2.7
> in a few cases. Using my "FASTCALL" patch, all tested function calls
> become faster or as fast as Python 2.7!
>
> But when I ran the CPython benchmark suite, I found some major
> performance regressions. In fact, it took me 3 months to understand
> that I didn't run benchmarks correctly and that most benchmarks of the
> CPython benchmark suite are very unstable. I wrote articles explaining
> how benchmarks should be run (to be stable) and I patched all
> benchmarks to use my new perf module which runs benchmarks in multiple
> processes and computes the average (to make benchmarks more stable).
>
> At the end, my minimum FASTCALL patch (issue #27128) doesn't show any
> major performance regression if you run "correctly" benchmarks :-)
> https://bugs.python.org/issue27128#msg272197
>
> Most benchmarks are not significant, 14 are faster, and only 4 are slower.
>
> According to benchmarks on the "full" FASTCALL patch, the slowdown are
> temporary and should become quickly speedup (with further changes).
>
> My question is now: can I push fastcall-2.patch of the issue #27128?
> This patch only adds the infrastructure to start working on more
> useful optimizations, more patches will come, I expect more exciting
> benchmark results.
>
> Overview of the initial FASTCALL patch, see my first message on the issue:
> https://bugs.python.org/issue27128#msg266422
>
> --
>
> Note: My full FASTCALL patch changes the C API: this is out of the
> scope of my first simple FASTCALL patch. I will open a new discussion
> to decide if it's worth it and if yes, how it should be done.
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160808/8d5c2a5c/attachment.html>

From victor.stinner at gmail.com  Mon Aug  8 20:01:34 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 Aug 2016 02:01:34 +0200
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
In-Reply-To: <CAP1=2W5HTz-7CRT_x=jCY8yh=3v+LG3uSP_qkhsHj=dNBERCBg@mail.gmail.com>
References: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
 <CAP1=2W5HTz-7CRT_x=jCY8yh=3v+LG3uSP_qkhsHj=dNBERCBg@mail.gmail.com>
Message-ID: <CAMpsgwZiOY5Ja+d88p7eCLZW905xT4ws-wED_5XPaz_s+nQaqg@mail.gmail.com>

2016-08-09 1:36 GMT+02:00 Brett Cannon <brett at python.org>:
> I just wanted to say I'm excited about this and I'm glad someone is taking
> advantage of what Argument Clinic allows for and what I know Larry had
> initially hoped AC would make happen!

To make "Python" faster, not only a few specific functions, "all" C
code should be updated to use the new "FASTCALL" calling convention.
But it's a pain to have to rewrite the code parsing arguments, we all
hate having to put #ifdef in the code... (for backward compatibility.)

This is where the magic happens: if your code is written using
Argument Clinic, you will get the optimization (FASTCALL) for free:
just run again Argument Clinic to get the "updated" "calling
convention".

It can be a very good motivation to rewrite your code using Argument
Clinic: get better inline documentation (docstring, help(func) in
REPL) *and* performance ;-)


> I should also point out that Serhiy has a patch for faster keyword argument
> parsing thanks to AC: http://bugs.python.org/issue27574 . Not sure how your
> two patches would intertwine (if at all).

In a first implementation, I packed *all* arguments in the same C
array: positional and keyword arguments. The problem is that all
functions expect a dict to parse keyword arguments. A dict has an
important property: O(1) for lookup. It becomes O(n) if you pass
keyword arguments as a list of (key, value) tuples in a C array.

So I chose to don't touch keyword arguments at all: continue to pass
them as a dict.

By the way, it's very rare to call a function using keyword arguments from C.

--

About http://bugs.python.org/issue27574 : it's really nice to see work
done on this part!

I recall a discussion of the performance of operator versus function
call. In some cases, the overhead of "parsing" arguments is higher
than the cost of the same feature implemented as an operator! Hum, it
was probably this issue:
https://bugs.python.org/issue17170

Extract of the issue:
"""
Some crude C benchmarking on this computer:
- calling PyUnicode_Replace is 35 ns (per call)
- calling "hundred".replace is 125 ns
- calling PyArg_ParseTuple with the same signature as "hundred".replace is 80 ns
"""

Victor

From yselivanov.ml at gmail.com  Mon Aug  8 21:16:16 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Mon, 8 Aug 2016 21:16:16 -0400
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
In-Reply-To: <CAMpsgwbDR_-ENt=SDWFkdKNO-KTDsiu--C-29wcs_FoDURrPMA@mail.gmail.com>
References: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
 <CAP7+vJJoVtetLV40qKmG2oCXSTxjnsehdJ7Fs95Ux4CCK5ecHg@mail.gmail.com>
 <CAMpsgwbDR_-ENt=SDWFkdKNO-KTDsiu--C-29wcs_FoDURrPMA@mail.gmail.com>
Message-ID: <80bc23cb-010a-0032-0c73-82f91e754a3d@gmail.com>

On 2016-08-08 6:53 PM, Victor Stinner wrote:

> 2016-08-09 0:40 GMT+02:00 Guido van Rossum <guido at python.org>:
>>> tl;dr I found a way to make CPython 3.6 faster and I validated that
>>> there is no performance regression.
>> But is there a performance improvement?
> Sure.
>
>
> On micro-benchmarks, you can see nice improvements:
>
> * getattr(1, "real") becomes 44% faster
> * list(filter(lambda x: x, list(range(1000)))) becomes 31% faster
> * namedtuple.attr becomes -23% faster
> * etc.
>
> See https://bugs.python.org/issue26814#msg263999 for default => patch,
> or https://bugs.python.org/issue26814#msg264003 for comparison python
> 2.7 / 3.4 / 3.5 / 3.6 / 3.6 patched.
>
>
> On the CPython benchmark suite, I also saw many faster benchmarks:
>
> Faster (25):
> - pickle_list: 1.29x faster
> - etree_generate: 1.22x faster
> - pickle_dict: 1.19x faster
> - etree_process: 1.16x faster
> - mako_v2: 1.13x faster
> - telco: 1.09x faster
> - raytrace: 1.08x faster
> - etree_iterparse: 1.08x faster
> (...)
>

Exceptional results, congrats Victor. Will be happy to help with code 
review.

Yury

From songofacandy at gmail.com  Tue Aug  9 09:12:00 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 9 Aug 2016 22:12:00 +0900
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
Message-ID: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>

Hi, devs.

I've implemented compact and ordered dictionary [1], which PyPy
implemented in 2015 [2].

Since it is my first large patch, I would like to have enough time for
review cycle by Python 3.6 beta1.

Could someone review it?

[1] http://bugs.python.org/issue27350
[2] https://morepypy.blogspot.jp/2015/01/faster-more-memory-efficient-and-more.html

-- 
INADA Naoki  <songofacandy at gmail.com>

From ncoghlan at gmail.com  Tue Aug  9 09:26:31 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 9 Aug 2016 23:26:31 +1000
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
Message-ID: <CADiSq7d9XPoUUdq6sJQ9Q4CvYGeN3y-vEaDT395Bb_SBRgJjpw@mail.gmail.com>

On 9 August 2016 at 06:18, Guido van Rossum <guido at python.org> wrote:

> I think Nick would be interested in understanding why this is the case.
> What does the decorator do that could be so expensive?
>

Reviewing https://hg.python.org/cpython/file/default/Lib/contextlib.py#l57,
Chris's analysis seems plausible to me: most of the cost is going to be in
the fact that instead of a single function call for __exit__ and __enter__,
we have a function call *and* a generator frame resumption, and in the case
of __exit__, the expected flow includes catching StopIteration. The object
creation is also marginally more expensive, since it's indirect through a
factory function rather than calling the type directly.

There's also currently some introspection of "func" being done in __init__
as a result of issue #19330 that should probably be moved out to the
contextmanager decorator and passed in from the closure as a separate
argument rather than being figured out on each call to
_GeneratorContextManager.__init__.

So I don't see any obvious reason we shouldn't be able to get the standard
library version at least to a similar level of performance to Chris's
simpler alternatives. There are also some potential object allocation
efficiencies we could consider, like using __slots__ to eliminate the
__dict__ allocation (that does have backwards compatibility implications,
but may be worth it if it buys a non-trivial speed improvement).

Beyond that, while I think a C accelerator may be worth trying, I'm not
sure how much it will actually gain us, as it seems plausible that a
function call + generator frame resumption will inherently take twice as
long as just doing a function call.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160809/d1037fdc/attachment-0001.html>

From ncoghlan at gmail.com  Tue Aug  9 09:30:25 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 9 Aug 2016 23:30:25 +1000
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CADiSq7d9XPoUUdq6sJQ9Q4CvYGeN3y-vEaDT395Bb_SBRgJjpw@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
 <CADiSq7d9XPoUUdq6sJQ9Q4CvYGeN3y-vEaDT395Bb_SBRgJjpw@mail.gmail.com>
Message-ID: <CADiSq7dzbb2paiXQ=UPU=YVQH4vbmOdeu_zN795EKtfTnJOufA@mail.gmail.com>

On 9 August 2016 at 23:26, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 9 August 2016 at 06:18, Guido van Rossum <guido at python.org> wrote:
>
>> I think Nick would be interested in understanding why this is the case.
>> What does the decorator do that could be so expensive?
>>
>
> Reviewing https://hg.python.org/cpython/file/default/Lib/contextlib.py#l57,
> Chris's analysis seems plausible to me
>

Sorry Wolfgang - I missed that Chris was expanding on a comparison you
initially made!

Either way, I agree that aspect does make up the bulk of the difference in
speed, so moving to C likely wouldn't help much. However, the speed
difference relative to the simpler warppers is far less defensible - I
think there are some opportunities for improvement there, especially around
moving introspection work out of _GeneratorContextManager.__init__ and into
the contextmanager decorator itself.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160809/82173b1a/attachment.html>

From Artyom.Skrobov at arm.com  Tue Aug  9 04:59:09 2016
From: Artyom.Skrobov at arm.com (Artyom Skrobov)
Date: Tue, 9 Aug 2016 08:59:09 +0000
Subject: [Python-Dev] Python parser performance optimizations
In-Reply-To: <DB6PR0801MB18770B0D765015EF3BF072B0873B0@DB6PR0801MB1877.eurprd08.prod.outlook.com>
References: <DB6PR0801MB1400C8D31C17EFC547E8544287410@DB6PR0801MB1400.eurprd08.prod.outlook.com>
 <DB6PR0801MB18770B0D765015EF3BF072B0873B0@DB6PR0801MB1877.eurprd08.prod.outlook.com>
Message-ID: <HE1PR0802MB24740187E1C95BCCE8E0B4A3871C0@HE1PR0802MB2474.eurprd08.prod.outlook.com>

Hello,

This is a monthly ping to get a review on http://bugs.python.org/issue26415 -- "Excessive peak memory consumption by the Python parser".

Following the comments from July, the patches now include updating Misc/NEWS and compiler.rst to describe the change.

The code change itself is still the same as a month ago.


From: Artyom Skrobov
Sent: 07 July 2016 15:44
To: python-dev at python.org; steve at pearwood.info; mafagafogigante at gmail.com; greg.ewing at canterbury.ac.nz
Cc: nd
Subject: RE: Python parser performance optimizations

Hello,

This is a monthly ping to get a review on http://bugs.python.org/issue26415 -- "Excessive peak memory consumption by the Python parser".
The first patch of the series (an NFC refactoring) was successfully committed earlier in June, so the next step is to get the second patch, "the payload", reviewed and committed.

To address the concerns raised by the commenters back in May: the patch doesn't lead to negative memory consumption, of course. The base for calculating percentages is the smaller number of the two; this is the same style of reporting that perf.py uses. In other words, "200% less memory usage" is a threefold shrink.

The absolute values, and the way they were produced, are all reported under the ticket.


From: Artyom Skrobov
Sent: 26 May 2016 11:19
To: 'python-dev at python.org'
Subject: Python parser performance optimizations

Hello,

Back in March, I've posted a patch at http://bugs.python.org/issue26526 -- "In parsermodule.c, replace over 2KLOC of hand-crafted validation code, with a DFA".

The motivation for this patch was to enable a memory footprint optimization, discussed at http://bugs.python.org/issue26415
My proposed optimization reduces the memory footprint by up to 30% on the standard benchmarks, and by 200% on a degenerate case which sparked the discussion.
The run time stays unaffected by this optimization.

Python Developer's Guide says: "If you don't get a response within a few days after pinging the issue, then you can try emailing python-dev at python.org<mailto:python-dev at python.org> asking for someone to review your patch."

So, here I am.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160809/4d385f27/attachment.html>

From g.rodola at gmail.com  Tue Aug  9 14:29:45 2016
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 9 Aug 2016 20:29:45 +0200
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CAPTjJmoi5LSPSuJoH9tOp-=k+F9S5yWEMBYxgfXvNcFu_GzY=Q@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
 <f6cc59f0-3657-a9c0-5755-099be4c972d0@gmail.com>
 <noasom$kvm$1@blaine.gmane.org>
 <CAPTjJmoi5LSPSuJoH9tOp-=k+F9S5yWEMBYxgfXvNcFu_GzY=Q@mail.gmail.com>
Message-ID: <CAFYqXL9udrY_+nk=F8uDfua_-n_12Ag6rOO0eAAOVBNeE7JLHw@mail.gmail.com>

On Mon, Aug 8, 2016 at 11:59 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Tue, Aug 9, 2016 at 7:14 AM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de> wrote:
> > Right, I think a fairer comparison would be to:
> >
> > class ctx2:
> >     def __enter__(self):
> >         self.it = iter(self)
> >         return next(self.it)
> >
> >     def __exit__(self, *args):
> >         try:
> >             next(self.it)
> >         except StopIteration:
> >             pass
> >
> >     def __iter__(self):
> >         yield
> >
> > With this change alone the slowdown diminishes to ~ 1.7x for me. The
> rest is
> > probably the extra overhead for being able to pass exceptions raised
> inside
> > the with block back into the generator and such.
>
> I played around with a few other variants to see where the slowdown
> is. They all work out pretty much the same as the above; my two
> examples are both used the same way as contextlib.contextmanager is,
> but are restrictive on what you can do.
>
> import timeit
> import contextlib
> import functools
>
> class ctx1:
>     def __enter__(self):
>         pass
>     def __exit__(self, *args):
>         pass
>
> @contextlib.contextmanager
> def ctx2():
>     yield
>
> class SimplerContextManager:
>     """Like contextlib._GeneratorContextManager but way simpler.
>
>     * Doesn't reinstantiate itself - just reinvokes the generator
>     * Doesn't allow yielded objects (returns self)
>     * Lacks a lot of error checking. USE ONLY AS DIRECTED.
>     """
>     def __init__(self, func):
>         self.func = func
>         functools.update_wrapper(self, func)
>     def __call__(self, *a, **kw):
>         self.gen = self.func(*a, **kw)
>         return self
>     def __enter__(self):
>         next(self.gen)
>         return self
>     def __exit__(self, type, value, traceback):
>         if type is None:
>             try: next(self.gen)
>             except StopIteration: return
>             else: raise RuntimeError("generator didn't stop")
>         try: self.gen.throw(type, value, traceback)
>         except StopIteration: return True
>         # Assume any instance of the same exception type is a proper
> reraise
>         # This is way simpler than contextmanager normally does, and costs
> us
>         # the ability to detect exception handlers that coincidentally
> raise
>         # the same type of error (eg "except ZeroDivisionError:
> print(1/0)").
>         except type: return False
>
> # Check that it actually behaves correctly
> @SimplerContextManager
> def ctxdemo():
>     print("Before yield")
>     try:
>         yield 123
>     except ZeroDivisionError:
>         print("Absorbing 1/0")
>         return
>     finally:
>         print("Finalizing")
>     print("After yield (no exception)")
>
> with ctxdemo() as val:
>     print("1/0 =", 1/0)
> with ctxdemo() as val:
>     print("1/1 =", 1/1)
> #with ctxdemo() as val:
> #    print("1/q =", 1/q)
>
> @SimplerContextManager
> def ctx3():
>     yield
>
> class TooSimpleContextManager:
>     """Now this time you've gone too far."""
>     def __init__(self, func):
>         self.func = func
>     def __call__(self):
>         self.gen = self.func()
>         return self
>     def __enter__(self):
>         next(self.gen)
>     def __exit__(self, type, value, traceback):
>         try: next(self.gen)
>         except StopIteration: pass
>
> @TooSimpleContextManager
> def ctx4():
>     yield
>
> class ctx5:
>     def __enter__(self):
>         self.it = iter(self)
>         return next(self.it)
>
>     def __exit__(self, *args):
>         try:
>             next(self.it)
>         except StopIteration:
>             pass
>
>     def __iter__(self):
>         yield
>
> t1 = timeit.timeit("with ctx1(): pass", setup="from __main__ import ctx1")
> print("%.3f secs" % t1)
>
> for i in range(2, 6):
>     t2 = timeit.timeit("with ctx2(): pass", setup="from __main__
> import ctx%d as ctx2"%i)
>     print("%.3f secs" % t2)
>     print("slowdown: -%.2fx" % (t2 / t1))
>
>
> My numbers are:
>
> 0.320 secs
> 1.354 secs
> slowdown: -4.23x
> 0.899 secs
> slowdown: -2.81x
> 0.831 secs
> slowdown: -2.60x
> 0.868 secs
> slowdown: -2.71x
>
> So compared to the tight custom-written context manager class, all the
> "pass it a generator function" varieties look pretty much the same.
> The existing contextmanager factory has several levels of indirection,
> and that's probably where the rest of the performance difference comes
> from, but there is some cost to the simplicity of the gen-func
> approach.
>
> My guess is that a C-implemented version could replace piles of
> error-handling code with simple pointer comparisons (hence my
> elimination of it), and may or may not be able to remove some of the
> indirection. I'd say it'd land about in the same range as the other
> examples here. Is that worth it?
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/g.rodola%
> 40gmail.com
>


Thanks for all this useful info. And I agree, some sort of slowdown should
be expected because it's the price you pay for the additional flexibility
that @contextmanager offers over a "raw" ctx manager class, which will
always be faster as "it does less".


-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160809/fa29dbbc/attachment.html>

From g.rodola at gmail.com  Tue Aug  9 14:43:34 2016
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 9 Aug 2016 20:43:34 +0200
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CADiSq7dzbb2paiXQ=UPU=YVQH4vbmOdeu_zN795EKtfTnJOufA@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
 <CADiSq7d9XPoUUdq6sJQ9Q4CvYGeN3y-vEaDT395Bb_SBRgJjpw@mail.gmail.com>
 <CADiSq7dzbb2paiXQ=UPU=YVQH4vbmOdeu_zN795EKtfTnJOufA@mail.gmail.com>
Message-ID: <CAFYqXL85Duvo-J10DpyTe+mJgHtTJUHfK6J62iPPHa923pCOwQ@mail.gmail.com>

On Tue, Aug 9, 2016 at 3:30 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 9 August 2016 at 23:26, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 9 August 2016 at 06:18, Guido van Rossum <guido at python.org> wrote:
>>
>>> I think Nick would be interested in understanding why this is the case.
>>> What does the decorator do that could be so expensive?
>>>
>>
>> Reviewing https://hg.python.org/cpython/file/default/Lib/contextlib.py
>> #l57, Chris's analysis seems plausible to me
>>
>
> Sorry Wolfgang - I missed that Chris was expanding on a comparison you
> initially made!
>
> Either way, I agree that aspect does make up the bulk of the difference in
> speed, so moving to C likely wouldn't help much. However, the speed
> difference relative to the simpler warppers is far less defensible - I
> think there are some opportunities for improvement there, especially around
> moving introspection work out of _GeneratorContextManager.__init__ and
> into the contextmanager decorator itself.
>

By moving the __doc__ introspection out of __init__ and by introducing
__slots__ I got from -4.37x to -3.16x (__slot__ speedup was about +0.3x).
Chris' SimplerContextManager solution is faster because it avoids the
factory function but that is necessary for supporting the decoration of
methods. Here's a PoC:


diff --git a/Lib/contextlib.py b/Lib/contextlib.py
index 7d94a57..45270dd 100644
--- a/Lib/contextlib.py
+++ b/Lib/contextlib.py
@@ -2,7 +2,7 @@
 import abc
 import sys
 from collections import deque
-from functools import wraps
+from functools import wraps, update_wrapper

 __all__ = ["contextmanager", "closing", "AbstractContextManager",
            "ContextDecorator", "ExitStack", "redirect_stdout",
@@ -57,25 +57,18 @@ class ContextDecorator(object):
 class _GeneratorContextManager(ContextDecorator, AbstractContextManager):
     """Helper for @contextmanager decorator."""

+    __slots__ = ['gen', 'funcak']
+
     def __init__(self, func, args, kwds):
         self.gen = func(*args, **kwds)
-        self.func, self.args, self.kwds = func, args, kwds
-        # Issue 19330: ensure context manager instances have good
docstrings
-        doc = getattr(func, "__doc__", None)
-        if doc is None:
-            doc = type(self).__doc__
-        self.__doc__ = doc
-        # Unfortunately, this still doesn't provide good help output when
-        # inspecting the created context manager instances, since pydoc
-        # currently bypasses the instance docstring and shows the docstring
-        # for the class instead.
-        # See http://bugs.python.org/issue19404 for more details.
+        self.funcak = func, args, kwds

     def _recreate_cm(self):
         # _GCM instances are one-shot context managers, so the
         # CM must be recreated each time a decorated function is
         # called
-        return self.__class__(self.func, self.args, self.kwds)
+        func, args, kwds = self.funcak
+        return self.__class__(func, args, kwds)

     def __enter__(self):
         try:
@@ -157,6 +150,8 @@ def contextmanager(func):
     @wraps(func)
     def helper(*args, **kwds):
         return _GeneratorContextManager(func, args, kwds)
+
+    update_wrapper(helper, func)
     return helper


-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160809/02114056/attachment.html>

From rosuav at gmail.com  Tue Aug  9 14:49:41 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 10 Aug 2016 04:49:41 +1000
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CAFYqXL85Duvo-J10DpyTe+mJgHtTJUHfK6J62iPPHa923pCOwQ@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
 <CADiSq7d9XPoUUdq6sJQ9Q4CvYGeN3y-vEaDT395Bb_SBRgJjpw@mail.gmail.com>
 <CADiSq7dzbb2paiXQ=UPU=YVQH4vbmOdeu_zN795EKtfTnJOufA@mail.gmail.com>
 <CAFYqXL85Duvo-J10DpyTe+mJgHtTJUHfK6J62iPPHa923pCOwQ@mail.gmail.com>
Message-ID: <CAPTjJmpz4mJiZB6E9DO0hFzN2+-4FRzqah2WkiuXJ_2ioiMXwg@mail.gmail.com>

On Wed, Aug 10, 2016 at 4:43 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> -        return self.__class__(self.func, self.args, self.kwds)
> +        func, args, kwds = self.funcak
> +        return self.__class__(func, args, kwds)

return self.__class__(*self.funcak)

>      @wraps(func)
>      def helper(*args, **kwds):
>          return _GeneratorContextManager(func, args, kwds)
> +
> +    update_wrapper(helper, func)

Isn't update_wrapper exactly what @wraps is doing for you?

ChrisA

From storchaka at gmail.com  Tue Aug  9 15:58:14 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 9 Aug 2016 22:58:14 +0300
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CAPTjJmoi5LSPSuJoH9tOp-=k+F9S5yWEMBYxgfXvNcFu_GzY=Q@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
 <f6cc59f0-3657-a9c0-5755-099be4c972d0@gmail.com>
 <noasom$kvm$1@blaine.gmane.org>
 <CAPTjJmoi5LSPSuJoH9tOp-=k+F9S5yWEMBYxgfXvNcFu_GzY=Q@mail.gmail.com>
Message-ID: <nodckm$pq8$1@blaine.gmane.org>

On 09.08.16 00:59, Chris Angelico wrote:
> class TooSimpleContextManager:
>     """Now this time you've gone too far."""
>     def __init__(self, func):
>         self.func = func
>     def __call__(self):
>         self.gen = self.func()
>         return self
>     def __enter__(self):
>         next(self.gen)
>     def __exit__(self, type, value, traceback):
>         try: next(self.gen)
>         except StopIteration: pass
[...]
 > My numbers are:
 >
 > 0.320 secs
 > 1.354 secs
 > slowdown: -4.23x
 > 0.899 secs
 > slowdown: -2.81x
 > 0.831 secs
 > slowdown: -2.60x
 > 0.868 secs
 > slowdown: -2.71x

I have got a slowdown 1.74x with TooSimpleContextManager with following 
implementation of __enter__ and __exit__:

     def __enter__(self):
         for res in self.gen:
             return res
         raise RuntimeError("generator didn't yield")
     def __exit__(self, type, value, traceback):
         for _ in self.gen:
             raise RuntimeError("generator didn't stop")



From fijall at gmail.com  Wed Aug 10 11:06:30 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 10 Aug 2016 17:06:30 +0200
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
Message-ID: <CAK5idxRwDvpVBtHfB-qXH55pHhNcS6AFQ9iveStpgNimXh8ebw@mail.gmail.com>

Hello everyone.

I did do only a cursory look on that one, but I would like to
reiterate that this gives huge benefits in general and we measured
nice speedups on pypy (where all the dicts are ordered forever):

* you can essentially kill OrderedDict or make it almost OrderedDict =
dict (in pypy it's a simple dict subclass that has one or two extra
things that OrderedDict has in the API)
* there are nice speedups
* the C version of OrderedDict can be killed
* it saves memory, on 64bit by quite a bit (not everyone stores more
than 4bln items in a dictionary)
* it solves the problem of tests relying on order in dictionaries

In short, it has no downsides

On Tue, Aug 9, 2016 at 3:12 PM, INADA Naoki <songofacandy at gmail.com> wrote:
> Hi, devs.
>
> I've implemented compact and ordered dictionary [1], which PyPy
> implemented in 2015 [2].
>
> Since it is my first large patch, I would like to have enough time for
> review cycle by Python 3.6 beta1.
>
> Could someone review it?
>
> [1] http://bugs.python.org/issue27350
> [2] https://morepypy.blogspot.jp/2015/01/faster-more-memory-efficient-and-more.html
>
> --
> INADA Naoki  <songofacandy at gmail.com>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

From ncoghlan at gmail.com  Wed Aug 10 12:09:07 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 11 Aug 2016 02:09:07 +1000
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CAFYqXL85Duvo-J10DpyTe+mJgHtTJUHfK6J62iPPHa923pCOwQ@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
 <CADiSq7d9XPoUUdq6sJQ9Q4CvYGeN3y-vEaDT395Bb_SBRgJjpw@mail.gmail.com>
 <CADiSq7dzbb2paiXQ=UPU=YVQH4vbmOdeu_zN795EKtfTnJOufA@mail.gmail.com>
 <CAFYqXL85Duvo-J10DpyTe+mJgHtTJUHfK6J62iPPHa923pCOwQ@mail.gmail.com>
Message-ID: <CADiSq7c1-VkesEDc01dDgF0R8bcCe-UVMr=qfEb5Aq+Aepqd7A@mail.gmail.com>

On 10 August 2016 at 04:43, Giampaolo Rodola' <g.rodola at gmail.com> wrote:

>
>
> On Tue, Aug 9, 2016 at 3:30 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 9 August 2016 at 23:26, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>> On 9 August 2016 at 06:18, Guido van Rossum <guido at python.org> wrote:
>>>
>>>> I think Nick would be interested in understanding why this is the case.
>>>> What does the decorator do that could be so expensive?
>>>>
>>>
>>> Reviewing https://hg.python.org/cpython/file/default/Lib/contextlib.py
>>> #l57, Chris's analysis seems plausible to me
>>>
>>
>> Sorry Wolfgang - I missed that Chris was expanding on a comparison you
>> initially made!
>>
>> Either way, I agree that aspect does make up the bulk of the difference
>> in speed, so moving to C likely wouldn't help much. However, the speed
>> difference relative to the simpler warppers is far less defensible - I
>> think there are some opportunities for improvement there, especially around
>> moving introspection work out of _GeneratorContextManager.__init__ and
>> into the contextmanager decorator itself.
>>
>
> By moving the __doc__ introspection out of __init__ and by introducing
> __slots__ I got from -4.37x to -3.16x (__slot__ speedup was about +0.3x).
>

The draft changes aren't preserving the semantics (which may suggest we
have some missing test cases), so we can't take timing numbers just yet.


> Chris' SimplerContextManager solution is faster because it avoids the
> factory function but that is necessary for supporting the decoration of
> methods. Here's a PoC:
>
>
> diff --git a/Lib/contextlib.py b/Lib/contextlib.py
> index 7d94a57..45270dd 100644
> --- a/Lib/contextlib.py
> +++ b/Lib/contextlib.py
> @@ -2,7 +2,7 @@
>  import abc
>  import sys
>  from collections import deque
> -from functools import wraps
> +from functools import wraps, update_wrapper
>
>  __all__ = ["contextmanager", "closing", "AbstractContextManager",
>             "ContextDecorator", "ExitStack", "redirect_stdout",
> @@ -57,25 +57,18 @@ class ContextDecorator(object):
>  class _GeneratorContextManager(ContextDecorator, AbstractContextManager):
>      """Helper for @contextmanager decorator."""
>
> +    __slots__ = ['gen', 'funcak']
> +
>

Note that this is is still keeping the __dict__ allocation (you'd have to
make ContexDecorator and AbstractContextManager use __slots__ as well to
eliminate it).

While there's likely some speedup just from the dict->descriptor shift,
let's see how far we can improve *without* taking the __slots__ step, and
then consider the question of adopting __slots__ (and the backwards
compatibility implications if we take it as far as eliminating __dict__)
separately.


>      def __init__(self, func, args, kwds):
>          self.gen = func(*args, **kwds)
> -        self.func, self.args, self.kwds = func, args, kwds
> -        # Issue 19330: ensure context manager instances have good
> docstrings
> -        doc = getattr(func, "__doc__", None)
> -        if doc is None:
> -            doc = type(self).__doc__
> -        self.__doc__ = doc
> -        # Unfortunately, this still doesn't provide good help output when
> -        # inspecting the created context manager instances, since pydoc
> -        # currently bypasses the instance docstring and shows the
> docstring
> -        # for the class instead.
> -        # See http://bugs.python.org/issue19404 for more details.
> +        self.funcak = func, args, kwds
>

This actually gives me an idea - we should compare the performance of the
current code with that of using a functools.partial object defined in the
decorator function, so only the partial object needs to be passed in to
_GeneratorContextManager, and we can drop the args and kwds parameters
entirely.

That should move some more of the work out of the per-use code in
_GeneratorContextManager.__init__ and into the per-definition code in the
contextmanager decorator, and "self.gen = func()" should then be faster
than "self.gen = func(*args, **kwds)".


>      def _recreate_cm(self):
>          # _GCM instances are one-shot context managers, so the
>          # CM must be recreated each time a decorated function is
>          # called
> -        return self.__class__(self.func, self.args, self.kwds)
> +        func, args, kwds = self.funcak
> +        return self.__class__(func, args, kwds)
>
>      def __enter__(self):
>          try:
> @@ -157,6 +150,8 @@ def contextmanager(func):
>      @wraps(func)
>      def helper(*args, **kwds):
>          return _GeneratorContextManager(func, args, kwds)
> +
> +    update_wrapper(helper, func)
>      return helper
>

As Chris noted, this probably isn't having the effect you want - to move
the introspection code out to the decorator, you still need to pull __doc__
from the function (at decoration time) and set it on the
_GeneratorContextManager instance (at instance creation time).

However, we're getting down to discussing proposed patch details now, so
it's probably time to file an enhancement issue on the tracker and move
further discussion there :)

Cheers,
Nick.

P.S. I head down to Melbourne for PyCon Australia tomorrow, so I probably
won't be too responsive until the conference sprints start on Monday.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160811/fc242a85/attachment.html>

From songofacandy at gmail.com  Wed Aug 10 12:52:58 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 10 Aug 2016 16:52:58 +0000
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <CAK5idxRwDvpVBtHfB-qXH55pHhNcS6AFQ9iveStpgNimXh8ebw@mail.gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
 <CAK5idxRwDvpVBtHfB-qXH55pHhNcS6AFQ9iveStpgNimXh8ebw@mail.gmail.com>
Message-ID: <CAEfz+TzS_ZZh4sPSR0PShyVdGxr3UQsRL1+yUvOjeFy23GWR+A@mail.gmail.com>

Thanks you for supporting.

BTW, my patch doesn't include OrderedDict speedup. (I'll try it when
compact dict is merged.)

In case of Python 3, OrderedDict.move_to_end(key, last=False) can append
item at front.
So implementing OrderedDict based on compact dict is not so easy as
OrderedDict = dict.

My idea is using compact_array (described in PyPy blog) as ring, not fixed
capacity array.
It means one of these two:

a. dict has one additional word and support ring internally.
b. OrderedDict reimplements many APIs (iterating, resizing, etc...) to
support ring.

I like (a) because it's easier to implement and maintain.
But I don't know it can be in time for Python 3.6 beta.

On Thu, Aug 11, 2016 at 12:06 AM Maciej Fijalkowski <fijall at gmail.com>
wrote:

> Hello everyone.
>
> I did do only a cursory look on that one, but I would like to
> reiterate that this gives huge benefits in general and we measured
> nice speedups on pypy (where all the dicts are ordered forever):
>
> * you can essentially kill OrderedDict or make it almost OrderedDict =
> dict (in pypy it's a simple dict subclass that has one or two extra
> things that OrderedDict has in the API)
> * there are nice speedups
> * the C version of OrderedDict can be killed
> * it saves memory, on 64bit by quite a bit (not everyone stores more
> than 4bln items in a dictionary)
> * it solves the problem of tests relying on order in dictionaries
>
> In short, it has no downsides
>
> On Tue, Aug 9, 2016 at 3:12 PM, INADA Naoki <songofacandy at gmail.com>
> wrote:
> > Hi, devs.
> >
> > I've implemented compact and ordered dictionary [1], which PyPy
> > implemented in 2015 [2].
> >
> > Since it is my first large patch, I would like to have enough time for
> > review cycle by Python 3.6 beta1.
> >
> > Could someone review it?
> >
> > [1] http://bugs.python.org/issue27350
> > [2]
> https://morepypy.blogspot.jp/2015/01/faster-more-memory-efficient-and-more.html
> >
> > --
> > INADA Naoki  <songofacandy at gmail.com>
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160810/5abdd094/attachment.html>

From rosuav at gmail.com  Wed Aug 10 17:10:26 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 11 Aug 2016 07:10:26 +1000
Subject: [Python-Dev] Rewrite @contextlib.contextmanager in C
In-Reply-To: <CAFYqXL85Duvo-J10DpyTe+mJgHtTJUHfK6J62iPPHa923pCOwQ@mail.gmail.com>
References: <CAFYqXL_kwt+AaOwngpc20F6MW0dZ3njfSp8e-jwi424FmE=1kg@mail.gmail.com>
 <a9eb42fd-34ff-ff2a-7b24-8f5aec8a7c81@gmail.com>
 <CAP7+vJLEtThqq-f62hYvVQc+uDzQoLHis-+SgYHw8cwDN-D7Xw@mail.gmail.com>
 <CADiSq7d9XPoUUdq6sJQ9Q4CvYGeN3y-vEaDT395Bb_SBRgJjpw@mail.gmail.com>
 <CADiSq7dzbb2paiXQ=UPU=YVQH4vbmOdeu_zN795EKtfTnJOufA@mail.gmail.com>
 <CAFYqXL85Duvo-J10DpyTe+mJgHtTJUHfK6J62iPPHa923pCOwQ@mail.gmail.com>
Message-ID: <CAPTjJmpibzE82yi+VCcg-e0CZchaYVpyUjjVuMdW6yx1PdKZSQ@mail.gmail.com>

On Wed, Aug 10, 2016 at 4:43 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> Chris' SimplerContextManager solution is faster because it avoids the
> factory function but that is necessary for supporting the decoration of
> methods.

Hooking a tangent onto this: I have no idea, based on the current
source code, which layers of complexity are necessary for which
reasons. Possibly as a side effect of this transformation, you might
consider adding some comments to the code? There are a few (citing
tracker issues such as #11647 and #27112), but they don't cover
everything.

ChrisA

From sylvain.corlay at gmail.com  Thu Aug 11 05:51:28 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Thu, 11 Aug 2016 11:51:28 +0200
Subject: [Python-Dev] PEP487: Simpler customization of class creation
In-Reply-To: <CAK9R32QEZcUVpwWK0-pV1QMp-ff9Y7XBhOCGE4hK+Sjpxosupg@mail.gmail.com>
References: <CAK=Phk5os9otxMMjH8jKr1ChrFdBetZwR6WUXcOKAzJ5zqPK=Q@mail.gmail.com>
 <CADiSq7fdhvULHf9MFH_S=0qhUBhKZ=LwxhETLSUXjcmNCt4_xg@mail.gmail.com>
 <CAK=Phk7aih7KzE9Wckqb3TfCY2P_CzGyNjicYSTgwKujkgquiw@mail.gmail.com>
 <CADiSq7dURxyviYwtgVFP99i2mEph7kD8aYuHNHFZ6sGm1F2NCQ@mail.gmail.com>
 <CAK9R32REWb5Vs82PxBJNq2euTDGKZoywwbcVLrWm4o4K5m2o6g@mail.gmail.com>
 <CAP7+vJJ_gML0XUaVfWz8kipRXAk=s2CUdHQFYO41PJK4de3gzQ@mail.gmail.com>
 <5794ED64.20408@stoneleaf.us>
 <CAP7+vJ+mhDqTWZz+JsAz5r93ge4TbXGTQxYF2mDV8Mff4VM11g@mail.gmail.com>
 <CADiSq7e3NRGFmdshHHyvUCvC8K+wxF=TJC-PjJiW=BT1Ho4BcA@mail.gmail.com>
 <CAH0mxTRYbfpQL=A-6KSB3tcysiOuy452bkZ8s-sMALm5Rq31aw@mail.gmail.com>
 <CADiSq7fQqNUz4QjYUb1r3jf6L+fYEuzDQuA9rYcT0R8uyC3RDw@mail.gmail.com>
 <CAH0mxTTebOcedeWDtGqB82Lu5EuPi28ZKOa9zjnhrauWsqEmog@mail.gmail.com>
 <CADiSq7cRSu9=BLzEBXJd3Foc9PO2+kF+GDmAxrhYxLXyCAeK=Q@mail.gmail.com>
 <CAH0mxTR6smXHiQyq0UjJBY0sHdpZdOgn36OXhqXiCf74RGr6hg@mail.gmail.com>
 <CADiSq7c0PhF8opbKZNpXmatNQNAc5v+0AzzZT8vhx_a14pag6g@mail.gmail.com>
 <CAH0mxTT_+EVoyC1rRte1B6T8dL4am8RO3HL92OPW8wEDkOxs-w@mail.gmail.com>
 <CADiSq7fHUSVmgi4V+m1wg7XDOxW5Pbkqp+MVkNxEgPoOjZaxnA@mail.gmail.com>
 <CAK9R32RniF4qvhcpT0KSqFEmwi13zfkr3to6-g=v-0wDHjXkRw@mail.gmail.com>
 <CAK=Phk5RHGswcV69YM8gc+pSpwPMJxK0pjdvhHKh9FVTvcoGFA@mail.gmail.com>
 <CAK9R32QEZcUVpwWK0-pV1QMp-ff9Y7XBhOCGE4hK+Sjpxosupg@mail.gmail.com>
Message-ID: <CAK=Phk6HP5606jZasFQ23QyZzzuxELEcHKh26yH3Q2HP7iYZQA@mail.gmail.com>

Hi Martin,

I think that we are on the same page. The differences in the approaches
here makes me think that I should attempt a PEP487-based migration of
traitlets and see if there is a major roadblock.

After all, that would probably be a good test for the PEP if the aim is to
enable usecases like traitlets without a metaclass.

Sylvain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160811/2a442f5f/attachment.html>

From status at bugs.python.org  Fri Aug 12 12:08:47 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 12 Aug 2016 18:08:47 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160812160847.2CCA2568DF@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-08-05 - 2016-08-12)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5595 ( +6)
  closed 33909 (+47)
  total  39504 (+53)

Open issues with patches: 2444 


Issues opened (40)
==================

#12345: Add math.tau
http://bugs.python.org/issue12345  reopened by gvanrossum

#27695: Long constant calculations stall compilation
http://bugs.python.org/issue27695  opened by fcostantini

#27696: win_add2path.py does not add user site directory
http://bugs.python.org/issue27696  opened by altendky

#27698: socketpair not in socket.__all__ on Windows
http://bugs.python.org/issue27698  opened by veky

#27701: [posixmodule] [Refactoring patch] Simply call into *at() funct
http://bugs.python.org/issue27701  opened by EdSchouten

#27703: Replace two Py_XDECREFs with Py_DECREFs in do_raise
http://bugs.python.org/issue27703  opened by xiang.zhang

#27704: bytes(x) is slow when x is bytearray
http://bugs.python.org/issue27704  opened by methane

#27705: Updating old C:/Windows/System32/ucrtbased.dll
http://bugs.python.org/issue27705  opened by terry.reedy

#27706: Random.seed, whose purpose is purportedly determinism, behaves
http://bugs.python.org/issue27706  opened by glyph

#27709: difflib.HtmlDiff produces different output from difflib.ndiff
http://bugs.python.org/issue27709  opened by SilentGhost

#27712: Tiny typos in import.rst
http://bugs.python.org/issue27712  opened by xiang.zhang

#27713: Spurious "platform dependent libraries" warnings when running 
http://bugs.python.org/issue27713  opened by chris.jerdonek

#27714: some test_idle tests are not re-runnable, producing false fail
http://bugs.python.org/issue27714  opened by ned.deily

#27715: call-matcher breaks if a method is mocked with spec=True
http://bugs.python.org/issue27715  opened by carljm

#27716: http.client truncates UTF-8 encoded headers
http://bugs.python.org/issue27716  opened by Lukasa

#27717: sqlite documentation bug
http://bugs.python.org/issue27717  opened by Eyal Mor

#27718: help('signal') incomplete (e.g: signal.signal not visible)
http://bugs.python.org/issue27718  opened by Gabriel Pettier (Tshirtman)

#27719: Misleading note about "args" attribute in "User-defined Except
http://bugs.python.org/issue27719  opened by benhoyt

#27720: decimal.Context.to_eng_string wrong docstring
http://bugs.python.org/issue27720  opened by ztane

#27723: Document typing.Text and typing.AnyStr
http://bugs.python.org/issue27723  opened by michael0x2a

#27724: PEP3119 inconsintent with actual CPython impl
http://bugs.python.org/issue27724  opened by Daisuke Miyakawa

#27725: Use Py_SIZE(x) instead of x->ob_size
http://bugs.python.org/issue27725  opened by trex58

#27726: ctags -t does not exists in Makefile.pre.in
http://bugs.python.org/issue27726  opened by matrixise

#27727: Update Tools/freeze to use .vcxproj files
http://bugs.python.org/issue27727  opened by Kaeptm Blaubaer

#27728: HTTPConnection.requests has a mutable as default value for hea
http://bugs.python.org/issue27728  opened by matrixise

#27729: Provide a better error message when the file path is too long 
http://bugs.python.org/issue27729  opened by brett.cannon

#27730: Update shutil to work with max file path length on Windows
http://bugs.python.org/issue27730  opened by brett.cannon

#27731: Opt-out of MAX_PATH on Windows 10
http://bugs.python.org/issue27731  opened by steve.dower

#27732: IDLE tests should be quiet
http://bugs.python.org/issue27732  opened by terry.reedy

#27733: Test that self.bell is called in IDLE classes when intended.
http://bugs.python.org/issue27733  opened by terry.reedy

#27734: Memory leaks at Python35-32
http://bugs.python.org/issue27734  opened by ???????????? ??????????????????

#27736: repeated Py_Initialize/PyRun_SimpleString/Py_Finalize segfault
http://bugs.python.org/issue27736  opened by smcv

#27737: email.header.Header.encode() crashes with IndexError on spaces
http://bugs.python.org/issue27737  opened by Tim.Graham

#27740: Fix doc of Py_CompileStringExFlags
http://bugs.python.org/issue27740  opened by xiang.zhang

#27741: datetime.datetime.strptime functionality description incorrect
http://bugs.python.org/issue27741  opened by Valectar

#27742: Random.randint generates different values in Python2 and Pytho
http://bugs.python.org/issue27742  opened by wrohdewald

#27743: Python 2 has a wrong artificial limit on the amount of memory 
http://bugs.python.org/issue27743  opened by tzickel

#27744: Add AF_ALG (Linux Kernel crypto) to socket module
http://bugs.python.org/issue27744  opened by christian.heimes

#27746: ResourceWarnings in test_asyncio
http://bugs.python.org/issue27746  opened by martin.panter

#27747: Broken example in the queue module documentation
http://bugs.python.org/issue27747  opened by Paulo Gabriel Poiati



Most recent 15 issues with no replies (15)
==========================================

#27746: ResourceWarnings in test_asyncio
http://bugs.python.org/issue27746

#27744: Add AF_ALG (Linux Kernel crypto) to socket module
http://bugs.python.org/issue27744

#27741: datetime.datetime.strptime functionality description incorrect
http://bugs.python.org/issue27741

#27740: Fix doc of Py_CompileStringExFlags
http://bugs.python.org/issue27740

#27737: email.header.Header.encode() crashes with IndexError on spaces
http://bugs.python.org/issue27737

#27733: Test that self.bell is called in IDLE classes when intended.
http://bugs.python.org/issue27733

#27731: Opt-out of MAX_PATH on Windows 10
http://bugs.python.org/issue27731

#27723: Document typing.Text and typing.AnyStr
http://bugs.python.org/issue27723

#27713: Spurious "platform dependent libraries" warnings when running 
http://bugs.python.org/issue27713

#27712: Tiny typos in import.rst
http://bugs.python.org/issue27712

#27705: Updating old C:/Windows/System32/ucrtbased.dll
http://bugs.python.org/issue27705

#27701: [posixmodule] [Refactoring patch] Simply call into *at() funct
http://bugs.python.org/issue27701

#27696: win_add2path.py does not add user site directory
http://bugs.python.org/issue27696

#27691: X509 cert with GEN_RID subject alt name causes SytemError
http://bugs.python.org/issue27691

#27679: set_bitfields() unused in _ctypes_test
http://bugs.python.org/issue27679



Most recent 15 issues waiting for review (15)
=============================================

#27740: Fix doc of Py_CompileStringExFlags
http://bugs.python.org/issue27740

#27737: email.header.Header.encode() crashes with IndexError on spaces
http://bugs.python.org/issue27737

#27728: HTTPConnection.requests has a mutable as default value for hea
http://bugs.python.org/issue27728

#27726: ctags -t does not exists in Makefile.pre.in
http://bugs.python.org/issue27726

#27723: Document typing.Text and typing.AnyStr
http://bugs.python.org/issue27723

#27720: decimal.Context.to_eng_string wrong docstring
http://bugs.python.org/issue27720

#27719: Misleading note about "args" attribute in "User-defined Except
http://bugs.python.org/issue27719

#27712: Tiny typos in import.rst
http://bugs.python.org/issue27712

#27709: difflib.HtmlDiff produces different output from difflib.ndiff
http://bugs.python.org/issue27709

#27704: bytes(x) is slow when x is bytearray
http://bugs.python.org/issue27704

#27703: Replace two Py_XDECREFs with Py_DECREFs in do_raise
http://bugs.python.org/issue27703

#27701: [posixmodule] [Refactoring patch] Simply call into *at() funct
http://bugs.python.org/issue27701

#27696: win_add2path.py does not add user site directory
http://bugs.python.org/issue27696

#27692: Clean serveral unnecessary NULL checks in exceptions.c
http://bugs.python.org/issue27692

#27691: X509 cert with GEN_RID subject alt name causes SytemError
http://bugs.python.org/issue27691



Top 10 most discussed issues (10)
=================================

#12345: Add math.tau
http://bugs.python.org/issue12345  30 msgs

#27392: Add a server_side keyword parameter to create_connection
http://bugs.python.org/issue27392  12 msgs

#12319: [http.client] HTTPConnection.request not support "chunked" Tra
http://bugs.python.org/issue12319  11 msgs

#27714: some test_idle tests are not re-runnable, producing false fail
http://bugs.python.org/issue27714  10 msgs

#27181: Add geometric mean to `statistics` module
http://bugs.python.org/issue27181   9 msgs

#26081: Implement asyncio Future in C to improve performance
http://bugs.python.org/issue26081   8 msgs

#27742: Random.randint generates different values in Python2 and Pytho
http://bugs.python.org/issue27742   8 msgs

#23105: os.O_SHLOCK and os.O_EXLOCK are not available on Linux
http://bugs.python.org/issue23105   7 msgs

#27128: Add _PyObject_FastCall()
http://bugs.python.org/issue27128   7 msgs

#27720: decimal.Context.to_eng_string wrong docstring
http://bugs.python.org/issue27720   7 msgs



Issues closed (46)
==================

#3216: Scarce msilib documentation
http://bugs.python.org/issue3216  closed by berker.peksag

#14594: document imp.load_dynamic()
http://bugs.python.org/issue14594  closed by berker.peksag

#15639: csv.Error description is incorrectly broad
http://bugs.python.org/issue15639  closed by berker.peksag

#18548: In unittest doc change WidgetTestCase to SimpleWidgetTestCase 
http://bugs.python.org/issue18548  closed by berker.peksag

#21699: Windows Python 3.4.1 pyvenv doesn't work in directories with s
http://bugs.python.org/issue21699  closed by vinay.sajip

#21999: shlex: bug in posix mode handling of empty strings
http://bugs.python.org/issue21999  closed by python-dev

#22391: MSILIB truncates last character in summary information stream
http://bugs.python.org/issue22391  closed by berker.peksag

#22557: Local import is too slow
http://bugs.python.org/issue22557  closed by serhiy.storchaka

#22829: Add --prompt option to venv
http://bugs.python.org/issue22829  closed by python-dev

#23306: Within zipfile, use of zlib.crc32 raises OverflowError at argu
http://bugs.python.org/issue23306  closed by gregory.p.smith

#23322: parser module docs missing second example
http://bugs.python.org/issue23322  closed by berker.peksag

#23542: Update PEP 476 for using urllib2.build_opener()
http://bugs.python.org/issue23542  closed by berker.peksag

#24626: please sync cgi.parse document
http://bugs.python.org/issue24626  closed by berker.peksag

#26750: Mock autospec does not work with subclasses of property()
http://bugs.python.org/issue26750  closed by gregory.p.smith

#26945: difflib.HtmlDiff().make_file() treat few change as whole line 
http://bugs.python.org/issue26945  closed by SilentGhost

#26981: add compatibility shim for enum34 backport
http://bugs.python.org/issue26981  closed by ethan.furman

#26988: Add AutoNumberedEnum to stdlib
http://bugs.python.org/issue26988  closed by ethan.furman

#27184: Support path objects in the ntpath module
http://bugs.python.org/issue27184  closed by brett.cannon

#27204: Failing doctests in Doc/howto/
http://bugs.python.org/issue27204  closed by python-dev

#27205: Failing doctests in Library/collections.rst
http://bugs.python.org/issue27205  closed by python-dev

#27206: Failing doctests in Doc/tutorial/
http://bugs.python.org/issue27206  closed by python-dev

#27207: Failing doctests in Doc/whatsnew/3.2.rst
http://bugs.python.org/issue27207  closed by python-dev

#27208: Failing doctests in Library/traceback.rst
http://bugs.python.org/issue27208  closed by python-dev

#27209: Failing doctests in Library/email.*.rst
http://bugs.python.org/issue27209  closed by python-dev

#27210: Failing doctests due to environmental dependencies in Lib/{nnt
http://bugs.python.org/issue27210  closed by python-dev

#27242: Make the docs for NotImplemented & NotImplementedError unambig
http://bugs.python.org/issue27242  closed by python-dev

#27373: logging.handlers.SysLogHandler with TCP not working on rsyslog
http://bugs.python.org/issue27373  closed by vinay.sajip

#27502: Python -m Module Vulnerable to Buffer Over Flow.
http://bugs.python.org/issue27502  closed by skrah

#27650: Implement `__repr__` methods for logging.Logger and others
http://bugs.python.org/issue27650  closed by python-dev

#27664: Allow specifying prefix for thread name in concurrent.futures.
http://bugs.python.org/issue27664  closed by gregory.p.smith

#27688: Expand documentation about Any in the typing module
http://bugs.python.org/issue27688  closed by gvanrossum

#27689: Add documentation for typing.Generator
http://bugs.python.org/issue27689  closed by gvanrossum

#27697: Obscure bug in the int() function
http://bugs.python.org/issue27697  closed by berker.peksag

#27699: spam
http://bugs.python.org/issue27699  closed by SilentGhost

#27700: Document asyncio.AbstractEventLoop, not asyncio.BaseEventLoop
http://bugs.python.org/issue27700  closed by gvanrossum

#27702: Only use SOCK_RAW when defined
http://bugs.python.org/issue27702  closed by berker.peksag

#27707: List reports incorrect length if modifed after yield
http://bugs.python.org/issue27707  closed by matrixise

#27708: Rounding/Large Int representation error with multiplication an
http://bugs.python.org/issue27708  closed by eric.smith

#27710: Disallow fold not in [0, 1] in time and datetime constructors
http://bugs.python.org/issue27710  closed by python-dev

#27711: datetime.utctimetuple() method should respect fold disambiguat
http://bugs.python.org/issue27711  closed by belopolsky

#27721: distutils strtobool returns 0 and 1 rather than False and True
http://bugs.python.org/issue27721  closed by rhettinger

#27722: Path.touch's keyword argument mode documented incorrectly
http://bugs.python.org/issue27722  closed by python-dev

#27735: 'module' object has no attribute 'SIGALRM' - singal module
http://bugs.python.org/issue27735  closed by r.david.murray

#27738: odd behavior in creating list of lambda expressions
http://bugs.python.org/issue27738  closed by ebarry

#27739: add math.sign/signum
http://bugs.python.org/issue27739  closed by r.david.murray

#27745: Fix typos in Argument Clinic howto
http://bugs.python.org/issue27745  closed by SilentGhost

From nad at python.org  Sat Aug 13 15:24:13 2016
From: nad at python.org (Ned Deily)
Date: Sat, 13 Aug 2016 15:24:13 -0400
Subject: [Python-Dev] Reminder: 3.6.0a4 snapshot 2016-08-15 12:00 UTC
Message-ID: <58AB682A-6ADE-4BF8-8B8E-D69D9B48C0BE@python.org>

The next and final alpha snapshot planned for the 3.6 release cycle is coming up soon.  Keep in mind that the feature development phase of the 3.6 release cycle is nearing an end.  The next major milestone is a big one: the first beta shapshot, 3.6.0b1, scheduled for Monday, 09-12, a little over four weeks from now.  That will mark the end of the unrestricted feature development phase and the start of the bugfix/stabilization/testing phase.  All feature code for 3.6.0 should be checked in by b1.  During the beta phase, the emphasis will be on fixes for new features, fixes for all categories of bugs and regressions, and documentation fixes/updates.  After b1, new features should be targeted for 3.7; more details will be forthcoming by b1.

As a reminder, alpha releases are intended to make it easier for the wider community to test the current state of new features and bug fixes for an upcoming Python release as a whole and for us to test the release process.  During the alpha phase, features may be added, modified, or deleted up until the start of the beta phase.  Alpha users beware!

To recap:

2016-08-15 ~12:00 UTC: code snapshot for 3.6.0 alpha 4.

now to 2016-09-12: Alpha phase (unrestricted feature development)

2016-09-12: 3.6.0 feature code freeze, 3.7.0 feature development begins

2016-09-12 to 2016-12-04: 3.6.0 beta phase (bug and regression fixes, no new features)

2016-12-04 3.6.0 release candidate 1 (3.6.0 code freeze)

2016-12-16 3.6.0 release (3.6.0rc1 plus, if necessary, any dire emergency fixes)


https://www.python.org/dev/peps/pep-0494/

--Ne?

--
  Ned Deily
  nad at python.org -- []


From Mauri.Miettinen at student.oulu.fi  Sat Aug 13 06:56:40 2016
From: Mauri.Miettinen at student.oulu.fi (Mauri Miettinen)
Date: Sat, 13 Aug 2016 10:56:40 +0000
Subject: [Python-Dev] Issues in Python TLS
Message-ID: <DB3PR05MB217E8A99C08C28E04133F90BE100@DB3PR05MB217.eurprd05.prod.outlook.com>

Hello,

We are experimenting with a tool for inspecting how well languages and
libraries support server certificate verification when establishing TLS
connections.

We are getting rather confusing results in our first major shootout of
bundled CPython 2 and 3 versions in major, still supported OS
distributions. We would love to get any insight into the test stubs and
results. Maybe we are doing something horribly wrong?

Python 2 and 3 with Requests

Our stub code:
https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python-requests/run.py <https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python-requests/run.py>

This is good news. All major distributions successfully check the TLS
certificates in all corner cases tested by the TryTLS. It was good news
that most distros also support SNI with this combination, the only
exceptions being CentOS 6.8, Ubuntu 12.04.5 and Ubuntu 14.04.

Python 2 with urllib2

Our stub code:
https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python-urllib2/run.py <https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python-urllib2/run.py>

Alpine Edge, Alpine 3.1, Debian 8.5, Fedora 24 and Ubuntu 16.04 pass
with flying colors.

On the other hand on CentOS 7.2 the test code accepts expired certificates,
wrong hostnames, self-signed certificates and incomplete chains of trust.
For CentOS 7.2 results see
https://github.com/ouspg/trytls/tree/shootout-0.2/shootout/centos7#python-urllib2 <https://github.com/ouspg/trytls/tree/shootout-0.2/shootout/centos7#python-urllib2>

It's worth noting that when any CA-bundle is given the situation improves.
However, since the stub works on the most distributions as expected, this
might be overlooked by the developers?

Python 3 with urllib

Our stub code:
https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python3-urllib/run.py

Alpine Edge, CentOS 6.8, CentOS 7.2 and Ubuntu 16.04 pass with flying
colors.

On Debian 8.5, Ubuntu 14.04 and Ubuntu 12.04 the test code accepts
expired certificates, wrong hostnames, self-signed certificates and
incomplete chains of trust. For Debian 8.5 results see
https://github.com/ouspg/trytls/tree/shootout-0.2/shootout/debian-latest#python3-urllib

Again it is worth noting that if any CA-bundle is given then situation
improves. Some experimentation we did with the test code suggests
that:

```
urllib.request.urlopen("https://" + host + ":" + port, cafile=None) -> DANGEROUS?
urllib.request.urlopen("https://" + host + ":" + port) -> DANGEROUS?
urllib.request.urlopen("https://" + host + ":" + port, cafile=None, cadefault=False) -> DANGEROUS?
urllib.request.urlopen("https://" + host + ":" + port, cafile="/anyfile", cadefault=False) -> SAFE
urllib.request.urlopen("https://" + host + ":" + port, cafile=None, cadefault=True) -> SAFE
urllib.request.urlopen("https://" + host + ":" + port, cadefault=True) -> SAFE
urllib.request.urlopen("https://" + host + ":" + port, cafile="/anyfile") -> SAFE
```

Summary

Our results overview is available from:
https://github.com/ouspg/trytls/tree/shootout-0.2/shootout <https://github.com/ouspg/trytls/tree/shootout-0.2/shootout>

People developing Python code that uses TLS might bump into nasty
surprises with how differently bundled Python versions behave between modern and supported distributions. Or are we just simply doing something horribly wrong?

Any feedback would be very welcome, as will try to do an updated
shootout with new TryTLS version next week. We would love to get
as fair, clean and comparable results as possible. Moreover, if you
can recommend any docs on proper "Do's and Don'ts" we'd love a link
to them.

Thank you very much,

Mauri Miettinen

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160813/b91348a5/attachment-0001.html>

From benjamin at python.org  Sat Aug 13 19:14:56 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 13 Aug 2016 16:14:56 -0700
Subject: [Python-Dev] Issues in Python TLS
In-Reply-To: <DB3PR05MB217E8A99C08C28E04133F90BE100@DB3PR05MB217.eurprd05.prod.outlook.com>
References: <DB3PR05MB217E8A99C08C28E04133F90BE100@DB3PR05MB217.eurprd05.prod.outlook.com>
Message-ID: <1471130096.3908864.694577073.41FACF43@webmail.messagingengine.com>

Correctness of TLS certificate verification is known to depend deeply on
distribution. Python began to verify certificates by default only in in
version 2.7.9. Many OS distributions (in particular, Ubuntu) did not
enable verification for their stable distributions for backwards
compatibility reasons. You might find looking at distro bugs for
CVE-2014-9365 edifying.

Thank you for your work.

On Sat, Aug 13, 2016, at 03:56, Mauri Miettinen wrote:
> Hello,
> 
> We are experimenting with a tool for inspecting how well languages and
> libraries support server certificate verification when establishing TLS
> connections.
> 
> We are getting rather confusing results in our first major shootout of
> bundled CPython 2 and 3 versions in major, still supported OS
> distributions. We would love to get any insight into the test stubs and
> results. Maybe we are doing something horribly wrong?
> 
> Python 2 and 3 with Requests
> 
> Our stub code:
> https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python-requests/run.py
> <https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python-requests/run.py>
> 
> This is good news. All major distributions successfully check the TLS
> certificates in all corner cases tested by the TryTLS. It was good news
> that most distros also support SNI with this combination, the only
> exceptions being CentOS 6.8, Ubuntu 12.04.5 and Ubuntu 14.04.
> 
> Python 2 with urllib2
> 
> Our stub code:
> https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python-urllib2/run.py
> <https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python-urllib2/run.py>
> 
> Alpine Edge, Alpine 3.1, Debian 8.5, Fedora 24 and Ubuntu 16.04 pass
> with flying colors.
> 
> On the other hand on CentOS 7.2 the test code accepts expired
> certificates,
> wrong hostnames, self-signed certificates and incomplete chains of trust.
> For CentOS 7.2 results see
> https://github.com/ouspg/trytls/tree/shootout-0.2/shootout/centos7#python-urllib2
> <https://github.com/ouspg/trytls/tree/shootout-0.2/shootout/centos7#python-urllib2>
> 
> It's worth noting that when any CA-bundle is given the situation
> improves.
> However, since the stub works on the most distributions as expected, this
> might be overlooked by the developers?
> 
> Python 3 with urllib
> 
> Our stub code:
> https://github.com/ouspg/trytls/blob/v0.2.1/stubs/python3-urllib/run.py
> 
> Alpine Edge, CentOS 6.8, CentOS 7.2 and Ubuntu 16.04 pass with flying
> colors.
> 
> On Debian 8.5, Ubuntu 14.04 and Ubuntu 12.04 the test code accepts
> expired certificates, wrong hostnames, self-signed certificates and
> incomplete chains of trust. For Debian 8.5 results see
> https://github.com/ouspg/trytls/tree/shootout-0.2/shootout/debian-latest#python3-urllib
> 
> Again it is worth noting that if any CA-bundle is given then situation
> improves. Some experimentation we did with the test code suggests
> that:
> 
> ```
> urllib.request.urlopen("https://" + host + ":" + port, cafile=None) ->
> DANGEROUS?
> urllib.request.urlopen("https://" + host + ":" + port) -> DANGEROUS?
> urllib.request.urlopen("https://" + host + ":" + port, cafile=None,
> cadefault=False) -> DANGEROUS?
> urllib.request.urlopen("https://" + host + ":" + port, cafile="/anyfile",
> cadefault=False) -> SAFE
> urllib.request.urlopen("https://" + host + ":" + port, cafile=None,
> cadefault=True) -> SAFE
> urllib.request.urlopen("https://" + host + ":" + port, cadefault=True) ->
> SAFE
> urllib.request.urlopen("https://" + host + ":" + port, cafile="/anyfile")
> -> SAFE
> ```
> 
> Summary
> 
> Our results overview is available from:
> https://github.com/ouspg/trytls/tree/shootout-0.2/shootout
> <https://github.com/ouspg/trytls/tree/shootout-0.2/shootout>
> 
> People developing Python code that uses TLS might bump into nasty
> surprises with how differently bundled Python versions behave between
> modern and supported distributions. Or are we just simply doing something
> horribly wrong?
> 
> Any feedback would be very welcome, as will try to do an updated
> shootout with new TryTLS version next week. We would love to get
> as fair, clean and comparable results as possible. Moreover, if you
> can recommend any docs on proper "Do's and Don'ts" we'd love a link
> to them.
> 
> Thank you very much,
> 
> Mauri Miettinen
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org

From steve at pearwood.info  Sun Aug 14 00:20:38 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 14 Aug 2016 14:20:38 +1000
Subject: [Python-Dev] Changing the licence of statistics.py
Message-ID: <20160814042037.GK26300@ando.pearwood.info>

I'm the author of statistics.py, and for historical reasons it was 
originally included in the standard library under the Apache licence.

I now wish to change that and have it licenced under Python's standard 
licence. Is there anything I need to do other than just remove the 
Apache licence boilerplate from the file?

(Yes, I've signed the contributor form.)


-- 
Steve

From brett at python.org  Sun Aug 14 00:27:21 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 14 Aug 2016 04:27:21 +0000
Subject: [Python-Dev] Changing the licence of statistics.py
In-Reply-To: <20160814042037.GK26300@ando.pearwood.info>
References: <20160814042037.GK26300@ando.pearwood.info>
Message-ID: <CAP1=2W5SmHi5m_n5qRj3OM+kKJgMEKFwfVcJssSwBJ3pErRDcg@mail.gmail.com>

Are you the sole author of the statistics module prior to contributing it,
Steve?

On Sat, Aug 13, 2016, 21:21 Steven D'Aprano <steve at pearwood.info> wrote:

> I'm the author of statistics.py, and for historical reasons it was
> originally included in the standard library under the Apache licence.
>
> I now wish to change that and have it licenced under Python's standard
> licence. Is there anything I need to do other than just remove the
> Apache licence boilerplate from the file?
>
> (Yes, I've signed the contributor form.)
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160814/87d7eb07/attachment.html>

From nad at python.org  Sun Aug 14 00:40:00 2016
From: nad at python.org (Ned Deily)
Date: Sun, 14 Aug 2016 00:40:00 -0400
Subject: [Python-Dev] Changing the licence of statistics.py
In-Reply-To: <20160814042037.GK26300@ando.pearwood.info>
References: <20160814042037.GK26300@ando.pearwood.info>
Message-ID: <41193DC6-A971-4E7D-84CD-DF63077A0094@python.org>

On Aug 14, 2016, at 00:20, Steven D'Aprano <steve at pearwood.info> wrote:
> I'm the author of statistics.py, and for historical reasons it was 
> originally included in the standard library under the Apache licence.
> 
> I now wish to change that and have it licenced under Python's standard 
> licence. Is there anything I need to do other than just remove the 
> Apache licence boilerplate from the file?

I am not a lawyer nor an expert on this but you should read the following pages.  It's explained there that the standard licenses for code contributed to Python itself *should* be either the Apache 2.0 License or the Academic Free License v2.1.

https://www.python.org/psf/contrib/
https://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq

--
  Ned Deily
  nad at python.org -- []


From steve at pearwood.info  Sun Aug 14 00:52:21 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 14 Aug 2016 14:52:21 +1000
Subject: [Python-Dev] Changing the licence of statistics.py
In-Reply-To: <CAP1=2W5SmHi5m_n5qRj3OM+kKJgMEKFwfVcJssSwBJ3pErRDcg@mail.gmail.com>
References: <20160814042037.GK26300@ando.pearwood.info>
 <CAP1=2W5SmHi5m_n5qRj3OM+kKJgMEKFwfVcJssSwBJ3pErRDcg@mail.gmail.com>
Message-ID: <20160814045221.GL26300@ando.pearwood.info>

On Sun, Aug 14, 2016 at 04:27:21AM +0000, Brett Cannon wrote:

> Are you the sole author of the statistics module prior to contributing it,
> Steve?

Yes.

There has been at least one patch to the module as part of the 
standard library by someone else:

http://bugs.python.org/issue26002

but prior to it hitting the stdlib it was all mine.

-- 
Steve

From steve at pearwood.info  Sun Aug 14 08:42:45 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 14 Aug 2016 22:42:45 +1000
Subject: [Python-Dev] Would somebody like to review issue27573 please?
Message-ID: <20160814124244.GQ26300@ando.pearwood.info>

I have a tiny patch for issue 27573, "code.interact() should print an 
exit message".

http://bugs.python.org/issue27573


Its a one-line change to code.py, plus a dozen or so lines changed in 
test_code.py. If its not controversial, I think it would be nice to get 
it into 3.6.


Thanks,


-- 
Steve

From barry at python.org  Sun Aug 14 13:50:36 2016
From: barry at python.org (Barry Warsaw)
Date: Sun, 14 Aug 2016 13:50:36 -0400
Subject: [Python-Dev] Issues in Python TLS
In-Reply-To: <1471130096.3908864.694577073.41FACF43@webmail.messagingengine.com>
References: <DB3PR05MB217E8A99C08C28E04133F90BE100@DB3PR05MB217.eurprd05.prod.outlook.com>
 <1471130096.3908864.694577073.41FACF43@webmail.messagingengine.com>
Message-ID: <20160814135036.4aefc5b6@subdivisions.wooz.org>

On Aug 13, 2016, at 04:14 PM, Benjamin Peterson wrote:

>Correctness of TLS certificate verification is known to depend deeply on
>distribution. Python began to verify certificates by default only in in
>version 2.7.9. Many OS distributions (in particular, Ubuntu) did not
>enable verification for their stable distributions for backwards
>compatibility reasons. You might find looking at distro bugs for
>CVE-2014-9365 edifying.

In particular, we discussed this issue with the Ubuntu security team and
decided that the backward compatibility issues required not enabling this by
default for older versions.  We did however include the mechanisms from PEP 493
so that end-users and system administrators could make different choices based
on their own assessments and needs.

Cheers,
-Barry

From xavier.combelle at gmail.com  Sun Aug 14 17:02:56 2016
From: xavier.combelle at gmail.com (Xavier Combelle)
Date: Sun, 14 Aug 2016 23:02:56 +0200
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <CAK5idxRwDvpVBtHfB-qXH55pHhNcS6AFQ9iveStpgNimXh8ebw@mail.gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
 <CAK5idxRwDvpVBtHfB-qXH55pHhNcS6AFQ9iveStpgNimXh8ebw@mail.gmail.com>
Message-ID: <3a7e0f40-0fd3-fa0c-bc68-176f16cd1bf8@gmail.com>



On 10/08/2016 17:06, Maciej Fijalkowski wrote:
> * there are nice speedups
>
in this blog post
https://morepypy.blogspot.fr/2015/01/faster-more-memory-efficient-and-more.html
it is mentioned big speedup only on microbenchmark and small speedups on
pypy benchmark. is it what you call nice speedups or does there is other
things ?


From tjreedy at udel.edu  Sun Aug 14 18:55:59 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 14 Aug 2016 18:55:59 -0400
Subject: [Python-Dev] Changing the licence of statistics.py
In-Reply-To: <41193DC6-A971-4E7D-84CD-DF63077A0094@python.org>
References: <20160814042037.GK26300@ando.pearwood.info>
 <41193DC6-A971-4E7D-84CD-DF63077A0094@python.org>
Message-ID: <noqstv$d5t$1@blaine.gmane.org>

On 8/14/2016 12:40 AM, Ned Deily wrote:
> On Aug 14, 2016, at 00:20, Steven D'Aprano <steve at pearwood.info> wrote:
>> I'm the author of statistics.py, and for historical reasons it was
>> originally included in the standard library under the Apache licence.
>>
>> I now wish to change that and have it licenced under Python's standard
>> licence. Is there anything I need to do other than just remove the
>> Apache licence boilerplate from the file?
>
> I am not a lawyer nor an expert on this but you should read the following pages.  It's explained there that the standard licenses for code contributed to Python itself *should* be either the Apache 2.0 License or the Academic Free License v2.1.
>
> https://www.python.org/psf/contrib/
> https://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq

I believe you selected one of these when you signed the CA.

-- 
Terry Jan Reedy


From turnbull.stephen.fw at u.tsukuba.ac.jp  Mon Aug 15 00:56:58 2016
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Mon, 15 Aug 2016 13:56:58 +0900
Subject: [Python-Dev] Changing the licence of statistics.py
In-Reply-To: <CAP1=2W5SmHi5m_n5qRj3OM+kKJgMEKFwfVcJssSwBJ3pErRDcg@mail.gmail.com>
References: <20160814042037.GK26300@ando.pearwood.info>
 <CAP1=2W5SmHi5m_n5qRj3OM+kKJgMEKFwfVcJssSwBJ3pErRDcg@mail.gmail.com>
Message-ID: <22449.19354.370303.467249@turnbull.sk.tsukuba.ac.jp>

Please talk to the lawyers (IANAL TINLA).

Steven d'Aprano writes:

 > I now wish to change that and have it licenced under Python's
 > standard licence. Is there anything I need to do other than just
 > remove the Apache licence boilerplate from the file?

IMHO, no; in fact I would argue that its presence is a (cosmetic)
bug.[1]

Brett Cannon writes:

 > Are you the sole author of the statistics module prior to contributing it,
 > Steve?

AFAICT, this doesn't matter.  The only legally binding condition on
contributors in the CA is choice of contribution license.[2]  The CA
itself states that this allows the PSF to distribute the contributed
code under an open source license of the PSF's choice[3].

I interpret that to mean that any committer can remove language
referring to contributors' licenses, implicitly changing the license
to the "global" Python license.[4]  (Again, IANAL, TINLA, ask VanL!)
I don't see any ethical problem with this, as signing the CA implies
the contributor approves this action in advance.

Of course this is assuming that all authors signed the CA and
therefore chose one of the two approved contributor licenses, but if
they didn't there are bigger problems here.

Footnotes: 
[1]  I don't think there can be legal ramifications for the rest of
Python, there are no copyleft licenses in sight.  There could be a bit
of trouble for the file itself if other contributors chose the AFL but
there's no notice of the AFL -- AIUI legally the AFL allows you to do
that but ethically it's fishy.

[2]  I guess there's an implied warranty of provenance, ie, the
contributor has the right to contribute the code.

[3]  IIRC, "PSF's choice" means "Board-approved".

[4]  Of course these licenses allow *anybody* to sublicense with *any*
license.  Here the restriction to the Python license is due to the CA.
I'm pretty sure that anything in the public repos has to be considered
to be "distributed by the PSF" in the legal sense, and I believe that
although the contributor license allows the PSF to use *any* license,
the CA binds the PSF to distribute under a Board-approved open source
license, or the contributor's original license.

From fijall at gmail.com  Mon Aug 15 02:32:00 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 15 Aug 2016 15:32:00 +0900
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <3a7e0f40-0fd3-fa0c-bc68-176f16cd1bf8@gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
 <CAK5idxRwDvpVBtHfB-qXH55pHhNcS6AFQ9iveStpgNimXh8ebw@mail.gmail.com>
 <3a7e0f40-0fd3-fa0c-bc68-176f16cd1bf8@gmail.com>
Message-ID: <CAK5idxRWWLB0Nqi255U5WZ+FM=RPgwrZg3zGxErRWuT+OwHbPA@mail.gmail.com>

On Mon, Aug 15, 2016 at 6:02 AM, Xavier Combelle
<xavier.combelle at gmail.com> wrote:
>
>
> On 10/08/2016 17:06, Maciej Fijalkowski wrote:
>> * there are nice speedups
>>
> in this blog post
> https://morepypy.blogspot.fr/2015/01/faster-more-memory-efficient-and-more.html
> it is mentioned big speedup only on microbenchmark and small speedups on
> pypy benchmark. is it what you call nice speedups or does there is other
> things ?

Yes, making dictionaries faster by a bit would not give you huge
speedups everywhere. It'll give you a small, measurable speedup a bit
everywhere. This is much better than a lot of things that cpython does
which is a performance win.

Note that there are two PEPs (sorted order in kwargs and sorted order
in class names) which would be superseded by just reviewing this patch
and merging it.

Best regards,
Maciej Fijalkowski

From yselivanov.ml at gmail.com  Mon Aug 15 11:23:14 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Mon, 15 Aug 2016 11:23:14 -0400
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
Message-ID: <da7b2ed0-c6b5-1090-d46a-f9404d2d5a97@gmail.com>

On 2016-08-09 9:12 AM, INADA Naoki wrote:

> Hi, devs.
>
> I've implemented compact and ordered dictionary [1], which PyPy
> implemented in 2015 [2].

Does this mean that keyword arguments will become ordered?

Yury


From songofacandy at gmail.com  Mon Aug 15 11:51:03 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 16 Aug 2016 00:51:03 +0900
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <da7b2ed0-c6b5-1090-d46a-f9404d2d5a97@gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
 <da7b2ed0-c6b5-1090-d46a-f9404d2d5a97@gmail.com>
Message-ID: <CAEfz+TziPXA=7weLCGWBRN9+t0QOGbjZ7usC9b1ZE6HtPvBEqA@mail.gmail.com>

>>
>> I've implemented compact and ordered dictionary [1], which PyPy
>> implemented in 2015 [2].
>
>
> Does this mean that keyword arguments will become ordered?
>
> Yury
>

Yes, regardless it will be language spec or just an implementation detail
like PyPy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160816/12b5fb15/attachment.html>

From nad at python.org  Mon Aug 15 22:59:33 2016
From: nad at python.org (Ned Deily)
Date: Mon, 15 Aug 2016 22:59:33 -0400
Subject: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available
Message-ID: <24931ad1-1bb1-0894-c2f5-4c8835cab319@python.org>

On behalf of the Python development community and the Python 3.6 release
team, I'm happy to announce the availability of Python 3.6.0a4. 3.6.0a4
is the last of four planned alpha releases of Python 3.6, the next major
release of Python. During the alpha phase, Python 3.6 remains under
heavy development: additional features will be added and existing
features may be modified or deleted. Please keep in mind that this is a
preview release and its use is not recommended for production environments.

You can find Python 3.6.0a4 here:

https://www.python.org/downloads/release/python-360a4/

The next planned release of Python 3.6 will be 3.6.0b1, currently
scheduled for 2016-09-12. 3.6.0b1 will mark the beginning of the beta
phase of Python 3.6; at that time, feature development for 3.6 will be
complete and the emphasis will change to fixing bugs and regressions.
More information about the release schedule can be found here:

https://www.python.org/dev/peps/pep-0494/

--Ned

--
  Ned Deily
  nad at python.org -- []

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160815/4b10a90e/attachment.sig>

From victor.stinner at gmail.com  Wed Aug 17 06:52:45 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 17 Aug 2016 12:52:45 +0200
Subject: [Python-Dev] Subprocess destructor now logs a ResourceWarning
Message-ID: <CAMpsgwZikn7PkCJ4J_NiirA3VKem97bJhB5bpWd+hirV+JUUaQ@mail.gmail.com>

Hi,

FYI I made a tiny enhancement in the subprocess module in Python 3.6:
Popen destructor now logs a ResourceWarning if the child process is
still running when the destructor is called. Keeping a child process
in background is bad because it is likely to create zombi process,
because Python will no more read its exit status.

Martin Panter asked to add a new "detach() method" to subprocess.Popen
if you would like to "transfer the ownership" of the child process
(pid) to a different object. I'm not sure that it is needed in
practice (Popen is responsible to handle different resources, not only
the pid, maybe I didn't look closely to the issue), but it may be
required for specific code to avoid the new ResourceWarning. Please
take a look at :

   https://bugs.python.org/issue27068

Another enhancement:  ResourceWarning now also logs where "leaked"
resource was created if you enable the tracemalloc module. Example
with tracemalloc logging 10 frames (truncated output for readability):
---------------------
$ ./python -X tracemalloc=10 -m test test_sys test_os
Run tests sequentially
0:00:00 [1/2] test_sys
^C

.../Lib/subprocess.py:1023: ResourceWarning: subprocess 28856 is still running
  ResourceWarning, source=self)
Object allocated at (most recent call first):
  File "/home/haypo/prog/python/default/Lib/test/test_sys.py", lineno 702
    universal_newlines=True)
  (...)

.../Lib/test/libregrtest/main.py:343: ResourceWarning: unclosed file
<_io.TextIOWrapper name=3 encoding='UTF-8'>
  break
Object allocated at (most recent call first):
  File "/home/haypo/prog/python/default/Lib/subprocess.py", lineno 955
    self.stdout = io.TextIOWrapper(self.stdout)
  File "/home/haypo/prog/python/default/Lib/test/test_sys.py", lineno 702
    universal_newlines=True)
  File "/home/haypo/prog/python/default/Lib/test/test_sys.py", lineno 714
    out = self.c_locale_get_error_handler(encoding=':ignore')
  (...)

Test suite interrupted by signal SIGINT.
2 tests omitted:
    test_os test_sys

Total duration: 551 ms
---------------------

Please run tests using -Wd (enabled by default in debug builds) and
watch for ResourceWarning, and then report these warnings on the bug
tracker. Each ResourceWarning is likely to highlight a bug.

I just fixed the subprocess ResourceWarning in the
test.support.script_helper module (the child process is now killed on
error, and I added a context manager to cleanup all Popen resources).

Victor

From ncoghlan at gmail.com  Wed Aug 17 12:15:03 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Aug 2016 02:15:03 +1000
Subject: [Python-Dev] Issues in Python TLS
In-Reply-To: <1471130096.3908864.694577073.41FACF43@webmail.messagingengine.com>
References: <DB3PR05MB217E8A99C08C28E04133F90BE100@DB3PR05MB217.eurprd05.prod.outlook.com>
 <1471130096.3908864.694577073.41FACF43@webmail.messagingengine.com>
Message-ID: <CADiSq7eEcNYAGyEhSPcwvD1zoR+adME-cbV5Urn7crs=EtdfzA@mail.gmail.com>

On 14 August 2016 at 09:14, Benjamin Peterson <benjamin at python.org> wrote:
> Correctness of TLS certificate verification is known to depend deeply on
> distribution. Python began to verify certificates by default only in in
> version 2.7.9. Many OS distributions (in particular, Ubuntu) did not
> enable verification for their stable distributions for backwards
> compatibility reasons. You might find looking at distro bugs for
> CVE-2014-9365 edifying.

For RHEL (and hence CentOS),
https://access.redhat.com/articles/2039753 summarises the current
status.

Since the system Python in RHEL 7.x is nominally Python 2.7.5, the
default behaviour reflects that. However, both PEP 466 and PEP 476
have been backported (as of RHEL 7.2 and the corresponding CentOS
release), so requests picks up the new capabilities implicitly, while
the standard library can be configured for default verification by
writing:

   [https]
   verify=enable

to /etc/python/cert-verification.cfg

Since folks are now able to force verification *off* via that same
configuration file mechanism, we've also reserved the right to
eventually change the default Python standard library behaviour on a
fresh RHEL 7 installation to be to verify certificates.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From flying-sheep at web.de  Wed Aug 17 11:02:34 2016
From: flying-sheep at web.de (Philipp A.)
Date: Wed, 17 Aug 2016 15:02:34 +0000
Subject: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available
In-Reply-To: <24931ad1-1bb1-0894-c2f5-4c8835cab319@python.org>
References: <24931ad1-1bb1-0894-c2f5-4c8835cab319@python.org>
Message-ID: <CAN8d9gkgENY5RGAFms7on6eFs03cxNnhwywJfR__V2Q9W3a4NQ@mail.gmail.com>

hi, i already posted in python-ideas, but i got no replies, so i?ll post
here:

in essence i think interpreting escape sequences in f-literals is a *very*
bad idea, mainly because i think it?s fundamental that syntax highlighters
can highlight code as code.

so they should highlight the code parts of f-literals as code to avoid bugs
and misunderstandings (parts highlighted as string aren?t expected to
execute). for many syntax highlighters, it?s *impossible* to properly
highlight python code after this change makes python?s grammar recursive
(escaped code in f-literals, doubly-escaped code in escaped f-literals in
f-literals, ?).

i want this changed because i think it?s only done this way to reuse the
string tokenization code (i.e. convenience), not for some deeper reason.
the RFC even says so. however, f-literals aren?t strings, but expressions,
so string tokenization rules shouldn?t apply to the non-string parts.

how am i going about changing f-literal grammar before the beta hits?

i consider this important enough to defer f-literals to 3.7 if it can?t get
in in time.

best, philipp

Ned Deily <nad at python.org> schrieb am Di., 16. Aug. 2016 um 05:02 Uhr:

> On behalf of the Python development community and the Python 3.6 release
> team, I'm happy to announce the availability of Python 3.6.0a4. 3.6.0a4
> is the last of four planned alpha releases of Python 3.6, the next major
> release of Python. During the alpha phase, Python 3.6 remains under
> heavy development: additional features will be added and existing
> features may be modified or deleted. Please keep in mind that this is a
> preview release and its use is not recommended for production environments.
>
> You can find Python 3.6.0a4 here:
>
> https://www.python.org/downloads/release/python-360a4/
>
> The next planned release of Python 3.6 will be 3.6.0b1, currently
> scheduled for 2016-09-12. 3.6.0b1 will mark the beginning of the beta
> phase of Python 3.6; at that time, feature development for 3.6 will be
> complete and the emphasis will change to fixing bugs and regressions.
> More information about the release schedule can be found here:
>
> https://www.python.org/dev/peps/pep-0494/
>
> --Ned
>
> --
>   Ned Deily
>   nad at python.org -- []
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/flying-sheep%40web.de
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160817/f6655a55/attachment.html>

From brett at python.org  Wed Aug 17 13:15:01 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 17 Aug 2016 17:15:01 +0000
Subject: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available
In-Reply-To: <CAN8d9gkgENY5RGAFms7on6eFs03cxNnhwywJfR__V2Q9W3a4NQ@mail.gmail.com>
References: <24931ad1-1bb1-0894-c2f5-4c8835cab319@python.org>
 <CAN8d9gkgENY5RGAFms7on6eFs03cxNnhwywJfR__V2Q9W3a4NQ@mail.gmail.com>
Message-ID: <CAP1=2W6Aak53g=Z+hsWn4A+QBkMcvC=960_jyyMZ_zCZjL6Jxg@mail.gmail.com>

Please don't cross-post as it means anyone replying to your email will now
split the conversation as not everyone will be subscribed to all of the
mailing lists you sent this to. I have stripped out all but python-dev for
my reply.

On Wed, 17 Aug 2016 at 09:47 Philipp A. <flying-sheep at web.de> wrote:

> hi, i already posted in python-ideas, but i got no replies, so i?ll post
> here:
>

I don't remember specifically seeing any email on this. Do you have a link
to your post from the python-ideas archive showing your email actually made
it to the list?


>
> in essence i think interpreting escape sequences in f-literals is a *very*
> bad idea, mainly because i think it?s fundamental that syntax highlighters
> can highlight code as code.
>

I believe you're referring to
https://www.python.org/dev/peps/pep-0498/#escape-sequences ?


>
> so they should highlight the code parts of f-literals as code to avoid
> bugs and misunderstandings (parts highlighted as string aren?t expected to
> execute). for many syntax highlighters, it?s *impossible* to properly
> highlight python code after this change makes python?s grammar recursive
> (escaped code in f-literals, doubly-escaped code in escaped f-literals in
> f-literals, ?).
>
> i want this changed because i think it?s only done this way to reuse the
> string tokenization code (i.e. convenience), not for some deeper reason.
> the RFC even says so. however, f-literals aren?t strings, but expressions,
> so string tokenization rules shouldn?t apply to the non-string parts.
>
> how am i going about changing f-literal grammar before the beta hits?
>

You can post to python-ideas and start a discussion there as the PEP has
already been accepted and implemented with the current semantics or ask for
clarification for the reasoning behind the decision here on python-dev.


>
> i consider this important enough to defer f-literals to 3.7 if it can?t
> get in in time.
>

I just wanted to let you know, Philipp, that your email comes off as
somewhat demanding, e.g. "I want this changed". Had you asked why the
decision was made then your email would not come off as "I'm right and
you're wrong" and more about you asking for clarification to understand
why, and then if you still disagreed with the thought process then bring up
that you think it may have been a mistake.

-Brett


>
> best, philipp
>
> Ned Deily <nad at python.org> schrieb am Di., 16. Aug. 2016 um 05:02 Uhr:
>
>> On behalf of the Python development community and the Python 3.6 release
>> team, I'm happy to announce the availability of Python 3.6.0a4. 3.6.0a4
>> is the last of four planned alpha releases of Python 3.6, the next major
>> release of Python. During the alpha phase, Python 3.6 remains under
>> heavy development: additional features will be added and existing
>> features may be modified or deleted. Please keep in mind that this is a
>> preview release and its use is not recommended for production
>> environments.
>>
>> You can find Python 3.6.0a4 here:
>>
>> https://www.python.org/downloads/release/python-360a4/
>>
>> The next planned release of Python 3.6 will be 3.6.0b1, currently
>> scheduled for 2016-09-12. 3.6.0b1 will mark the beginning of the beta
>> phase of Python 3.6; at that time, feature development for 3.6 will be
>> complete and the emphasis will change to fixing bugs and regressions.
>> More information about the release schedule can be found here:
>>
>> https://www.python.org/dev/peps/pep-0494/
>>
>> --Ned
>>
>> --
>>   Ned Deily
>>   nad at python.org -- []
>>
>> _______________________________________________
>>
> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/flying-sheep%40web.de
>>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160817/b51b5102/attachment.html>

From flying-sheep at web.de  Wed Aug 17 13:58:41 2016
From: flying-sheep at web.de (Philipp A.)
Date: Wed, 17 Aug 2016 17:58:41 +0000
Subject: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available
In-Reply-To: <CAP1=2W6Aak53g=Z+hsWn4A+QBkMcvC=960_jyyMZ_zCZjL6Jxg@mail.gmail.com>
References: <24931ad1-1bb1-0894-c2f5-4c8835cab319@python.org>
 <CAN8d9gkgENY5RGAFms7on6eFs03cxNnhwywJfR__V2Q9W3a4NQ@mail.gmail.com>
 <CAP1=2W6Aak53g=Z+hsWn4A+QBkMcvC=960_jyyMZ_zCZjL6Jxg@mail.gmail.com>
Message-ID: <CAN8d9gkowecLLuVDhxHeDcfLJ7HAgktHtbebc4wF0G1A_r9tDg@mail.gmail.com>

Brett Cannon <brett at python.org> schrieb am Mi., 17. Aug. 2016 um 19:15 Uhr:

> Please don't cross-post as it means anyone replying to your email will now
> split the conversation as not everyone will be subscribed to all of the
> mailing lists you sent this to. I have stripped out all but python-dev for
> my reply.
>

sorry, i?ll remember that! i just hit reply on the post and didn?t realize
it was posted to more than python-dev.

I don't remember specifically seeing any email on this. Do you have a link
> to your post from the python-ideas archive showing your email actually made
> it to the list?
>

i posted here
<https://groups.google.com/forum/#!topic/python-ideas/V1U6DGL5J1s>?

I believe you're referring to
> https://www.python.org/dev/peps/pep-0498/#escape-sequences ?
>

yes!

how am i going about changing f-literal grammar before the beta hits?
>>
>
> You can post to python-ideas and start a discussion there as the PEP has
> already been accepted and implemented with the current semantics or ask for
> clarification for the reasoning behind the decision here on python-dev.
>

thanks. i?d like to hereby do the latter. i think the PEP?s wording is
pretty clear:

Due to Python's string tokenizing rules, the f-string f'abc {a['x']} def' is
> invalid. The tokenizer parses this as 3 tokens: f'abc {a[' , x , and ']}
> def' . Just like regular strings, this cannot be fixed by using raw
> strings. There are a number of correct ways to write this f-string
>

i guess that means that python?s string tokenization rules are reused for
f-literals, even though they aren?t actually strings. could someone please
explain if this is right and if so, why it was chosen to do this instead of
writing more fitting tokenization code?

naively i?d assume f'abc {a['x']} def' to tokenize as something like:

F_BEGIN
  F_STRING_BEGIN "a" "b" "c" " " F_STRING_END
  F_EXPR_START
    NAME_START "a" NAME_END
    GETITEM_BEGIN STRING_BEGIN "x" STRING_END GETITEM_END
  F_EXPR_END
  F_STRING_BEGIN " " "d" "e" "f" F_STRING_END
F_END

where f-literals are defined as F_START + F_STRING + (F_EXPR + F_STRING)* +
F_END

all of this of course accounting for different delimiters and so on

i consider this important enough to defer f-literals to 3.7 if it can?t get
>> in in time.
>>
>
> I just wanted to let you know, Philipp, that your email comes off as
> somewhat demanding, e.g. "I want this changed". Had you asked why the
> decision was made then your email would not come off as "I'm right and
> you're wrong" and more about you asking for clarification to understand
> why, and then if you still disagreed with the thought process then bring up
> that you think it may have been a mistake.
>

sorry. i just wanted to make my feelings clear since i think this is an
overlooked issue and the time is tight, in hope that maybe someone is
inspired to listen. i thought the PEP?s wording was hint enough to explain
the rationale (convenient reuse of tokenization code)

i?ll patiently await clarification about this, and again: sorry for
sounding demanding :(

-Brett
>

Cheers, Philipp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160817/0ee100a0/attachment.html>

From guido at python.org  Wed Aug 17 14:10:56 2016
From: guido at python.org (Guido van Rossum)
Date: Wed, 17 Aug 2016 11:10:56 -0700
Subject: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available
In-Reply-To: <CAN8d9gkowecLLuVDhxHeDcfLJ7HAgktHtbebc4wF0G1A_r9tDg@mail.gmail.com>
References: <24931ad1-1bb1-0894-c2f5-4c8835cab319@python.org>
 <CAN8d9gkgENY5RGAFms7on6eFs03cxNnhwywJfR__V2Q9W3a4NQ@mail.gmail.com>
 <CAP1=2W6Aak53g=Z+hsWn4A+QBkMcvC=960_jyyMZ_zCZjL6Jxg@mail.gmail.com>
 <CAN8d9gkowecLLuVDhxHeDcfLJ7HAgktHtbebc4wF0G1A_r9tDg@mail.gmail.com>
Message-ID: <CAP7+vJJzstRfshw6hRSCvCLtpD8TWum57Of6Jf2VM5YLHKq4dA@mail.gmail.com>

On Wed, Aug 17, 2016 at 10:58 AM, Philipp A. <flying-sheep at web.de> wrote:

> i?ll patiently await clarification about this, and again: sorry for
> sounding demanding :(
>

The explanation is honestly that the current approach is the most
straightforward for the implementation (it's pretty hard to intercept the
string literal before escapes have been processed) and nobody cares enough
about the edge cases to force the implementation to jump through more hoops.

I really don't think this discussion should be reopened. If you disagree,
please start a new thread on python-ideas. Everybody: please don't follow
up to python-dev.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160817/71139a94/attachment.html>

From brett at python.org  Wed Aug 17 14:28:16 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 17 Aug 2016 18:28:16 +0000
Subject: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available
In-Reply-To: <CAN8d9gkowecLLuVDhxHeDcfLJ7HAgktHtbebc4wF0G1A_r9tDg@mail.gmail.com>
References: <24931ad1-1bb1-0894-c2f5-4c8835cab319@python.org>
 <CAN8d9gkgENY5RGAFms7on6eFs03cxNnhwywJfR__V2Q9W3a4NQ@mail.gmail.com>
 <CAP1=2W6Aak53g=Z+hsWn4A+QBkMcvC=960_jyyMZ_zCZjL6Jxg@mail.gmail.com>
 <CAN8d9gkowecLLuVDhxHeDcfLJ7HAgktHtbebc4wF0G1A_r9tDg@mail.gmail.com>
Message-ID: <CAP1=2W4i0umS5_a8NvMnZD+Mu5dHfgiQugWKW2N8TbX9iLRF2w@mail.gmail.com>

On Wed, 17 Aug 2016 at 10:58 Philipp A. <flying-sheep at web.de> wrote:

> Brett Cannon <brett at python.org> schrieb am Mi., 17. Aug. 2016 um
> 19:15 Uhr:
>
>> Please don't cross-post as it means anyone replying to your email will
>> now split the conversation as not everyone will be subscribed to all of the
>> mailing lists you sent this to. I have stripped out all but python-dev for
>> my reply.
>>
>
> sorry, i?ll remember that! i just hit reply on the post and didn?t realize
> it was posted to more than python-dev.
>
> I don't remember specifically seeing any email on this. Do you have a link
>> to your post from the python-ideas archive showing your email actually made
>> it to the list?
>>
>
> i posted here
> <https://groups.google.com/forum/#!topic/python-ideas/V1U6DGL5J1s>?
>

That never made it to the mailing list because you posted it to Google
Groups (notice how it isn't in the April archives for the mailing list:
https://mail.python.org/pipermail/python-ideas/2016-April/thread.html).


>
> I believe you're referring to
>> https://www.python.org/dev/peps/pep-0498/#escape-sequences ?
>>
>
> yes!
>
> how am i going about changing f-literal grammar before the beta hits?
>>>
>>
>> You can post to python-ideas and start a discussion there as the PEP has
>> already been accepted and implemented with the current semantics or ask for
>> clarification for the reasoning behind the decision here on python-dev.
>>
>
> thanks. i?d like to hereby do the latter. i think the PEP?s wording is
> pretty clear:
>
> Due to Python's string tokenizing rules, the f-string f'abc {a['x']} def' is
>> invalid. The tokenizer parses this as 3 tokens: f'abc {a[' , x , and ']}
>> def' . Just like regular strings, this cannot be fixed by using raw
>> strings. There are a number of correct ways to write this f-string
>>
>
> i guess that means that python?s string tokenization rules are reused for
> f-literals, even though they aren?t actually strings.
>

They are still strings, there is just post-processing on the string itself
to do the interpolation.


> could someone please explain if this is right and if so, why it was chosen
> to do this instead of writing more fitting tokenization code?
>

My suspicion is simplification of the implementation, but Eric Smith can
tell me if I'm wrong. By doing it this way the implementation can use
Python itself to do the tokenizing of the string, while if you do the
string interpolation beforehand you would then need to do it entirely at
the C level which is very messy and painful since you're explicitly
avoiding Python's automatic handling of Unicode, etc. As I think you
pointed out, doing it the way it is currently implemented allows for
re-using the str.format() code which is way easier. When it comes to an
open source project where no one is paid to work on it, easy and pragmatic
beats out purer and harder (it's in the Zen of Python after all :) .

You also make it harder to work with Unicode-based variable names (or at
least explain it). If you have Unicode in a variable name but you can't use
\N{} in the string to help express it you then have to say "normal Unicode
support in the string applies everywhere *but* in the string interpolation
part".

Or another reason is you can explain f-strings as "basically
str.format_map(**locals(), **globals()), but without having to make the
actual method call" (and worrying about clashing keys but I couldn't think
of a way of using dict.update() in a single line). But with your desired
change it kills this explanation by saying f-strings aren't like this but
some magical string that does all of this stuff before normal string
normalization occurs.


>
> naively i?d assume f'abc {a['x']} def' to tokenize as something like:
>
> F_BEGIN
>   F_STRING_BEGIN "a" "b" "c" " " F_STRING_END
>   F_EXPR_START
>     NAME_START "a" NAME_END
>     GETITEM_BEGIN STRING_BEGIN "x" STRING_END GETITEM_END
>   F_EXPR_END
>   F_STRING_BEGIN " " "d" "e" "f" F_STRING_END
> F_END
>
> where f-literals are defined as F_START + F_STRING + (F_EXPR + F_STRING)*
> + F_END
>
> all of this of course accounting for different delimiters and so on
>
> i consider this important enough to defer f-literals to 3.7 if it can?t
>>> get in in time.
>>>
>>
>> I just wanted to let you know, Philipp, that your email comes off as
>> somewhat demanding, e.g. "I want this changed". Had you asked why the
>> decision was made then your email would not come off as "I'm right and
>> you're wrong" and more about you asking for clarification to understand
>> why, and then if you still disagreed with the thought process then bring up
>> that you think it may have been a mistake.
>>
>
> sorry. i just wanted to make my feelings clear since i think this is an
> overlooked issue and the time is tight, in hope that maybe someone is
> inspired to listen. i thought the PEP?s wording was hint enough to explain
> the rationale (convenient reuse of tokenization code)
>
> i?ll patiently await clarification about this, and again: sorry for
> sounding demanding :(
>

Not a problem! I figured you didn't mean for it to, hence why I took the
time to point it out and reply calmly (and if it didn't come off as that
I'm sorry).

-Brett


>
> -Brett
>>
>
> Cheers, Philipp
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160817/31ccd4f2/attachment-0001.html>

From arigo at tunes.org  Thu Aug 18 03:53:11 2016
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 18 Aug 2016 09:53:11 +0200
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <CAEfz+TzS_ZZh4sPSR0PShyVdGxr3UQsRL1+yUvOjeFy23GWR+A@mail.gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
 <CAK5idxRwDvpVBtHfB-qXH55pHhNcS6AFQ9iveStpgNimXh8ebw@mail.gmail.com>
 <CAEfz+TzS_ZZh4sPSR0PShyVdGxr3UQsRL1+yUvOjeFy23GWR+A@mail.gmail.com>
Message-ID: <CAMSv6X0vTJNVZknYA9hCn8dcXXqJy0yXgfJxLo1mLeuw62yGPw@mail.gmail.com>

Hi Inada,

On 10 August 2016 at 18:52, INADA Naoki <songofacandy at gmail.com> wrote:
> a. dict has one additional word and support ring internally.
> b. OrderedDict reimplements many APIs (iterating, resizing, etc...) to support ring.

There is a solution "c." which might be simpler.  Let's think first
about what occurs with a normal dict (with your patch, or in PyPy) if
the user repeatedly pops the oldest item and adds a new item.  In this
case, the dict will accumulate dead entries at the start of the list,
and when it gets too many of them, it needs to make a complete copy of
the list to get rid of them.  (This is not a problem as it is
amortized over many pop-and-add operations.)

So now, when the user calls move_to_end(last=False), we can do the
opposite.  We look if there are already some deleted entries at the
start, and if so, we add the item at the place of the last deleted
entry.  If there aren't any, then we make a copy of the list that
*adds* some number of entries at the start, which are initially marked
as deleted...


A bient?t,

Armin.

From levkivskyi at gmail.com  Thu Aug 18 04:16:13 2016
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Thu, 18 Aug 2016 10:16:13 +0200
Subject: [Python-Dev] review request: anti-registration of ABCs
Message-ID: <CAOMjWkmver5NL6piQ_as_RvC93j3xiC+5xYFwduXXXWgCzdvsg@mail.gmail.com>

I have unsuccessfully tried to ping the issue http://bugs.python.org/
issue25958 on "anti-registration" of implicit ABCs by setting attributes to
None.

Guido said: "But with the indicated fixes, I think this good to go (for
Python 3.6)"

I have manually "rebased" the previous patch by Andrew Barnert taking into
account part of the work that has been done on Reversible and fixing final
minor comments.

I would like this to land in time for 3.6, could someone please review the
latest patch and check that everything is OK?

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160818/d5c92206/attachment.html>

From guido at python.org  Thu Aug 18 11:12:42 2016
From: guido at python.org (Guido van Rossum)
Date: Thu, 18 Aug 2016 08:12:42 -0700
Subject: [Python-Dev] review request: anti-registration of ABCs
In-Reply-To: <CAOMjWkmver5NL6piQ_as_RvC93j3xiC+5xYFwduXXXWgCzdvsg@mail.gmail.com>
References: <CAOMjWkmver5NL6piQ_as_RvC93j3xiC+5xYFwduXXXWgCzdvsg@mail.gmail.com>
Message-ID: <CAP7+vJLFOO13MHpMF3n+V3cEO=zmOaxv6AhdD+a=cfZ+KVkm-g@mail.gmail.com>

Looking...

On Thu, Aug 18, 2016 at 1:16 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> I have unsuccessfully tried to ping the issue
> http://bugs.python.org/issue25958 on "anti-registration" of implicit ABCs
> by setting attributes to None.
>
> Guido said: "But with the indicated fixes, I think this good to go (for
> Python 3.6)"
>
> I have manually "rebased" the previous patch by Andrew Barnert taking into
> account part of the work that has been done on Reversible and fixing final
> minor comments.
>
> I would like this to land in time for 3.6, could someone please review the
> latest patch and check that everything is OK?
>
> --
> Ivan
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160818/ecd13d51/attachment.html>

From guido at python.org  Thu Aug 18 12:50:55 2016
From: guido at python.org (Guido van Rossum)
Date: Thu, 18 Aug 2016 09:50:55 -0700
Subject: [Python-Dev] review request: anti-registration of ABCs
In-Reply-To: <CAP7+vJLFOO13MHpMF3n+V3cEO=zmOaxv6AhdD+a=cfZ+KVkm-g@mail.gmail.com>
References: <CAOMjWkmver5NL6piQ_as_RvC93j3xiC+5xYFwduXXXWgCzdvsg@mail.gmail.com>
 <CAP7+vJLFOO13MHpMF3n+V3cEO=zmOaxv6AhdD+a=cfZ+KVkm-g@mail.gmail.com>
Message-ID: <CAP7+vJ+wAjU-MiptHbZRXKcKKtVdtFcH4Rk-pcn57hTAMuR90w@mail.gmail.com>

Committed. Someone please watch the buildbots.

On Thu, Aug 18, 2016 at 8:12 AM, Guido van Rossum <guido at python.org> wrote:

> Looking...
>
> On Thu, Aug 18, 2016 at 1:16 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
> wrote:
>
>> I have unsuccessfully tried to ping the issue
>> http://bugs.python.org/issue25958 on "anti-registration" of implicit
>> ABCs by setting attributes to None.
>>
>> Guido said: "But with the indicated fixes, I think this good to go (for
>> Python 3.6)"
>>
>> I have manually "rebased" the previous patch by Andrew Barnert taking
>> into account part of the work that has been done on Reversible and fixing
>> final minor comments.
>>
>> I would like this to land in time for 3.6, could someone please review
>> the latest patch and check that everything is OK?
>>
>> --
>> Ivan
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160818/6bcb0511/attachment.html>

From sylvain.corlay at gmail.com  Fri Aug 19 04:07:50 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Fri, 19 Aug 2016 10:07:50 +0200
Subject: [Python-Dev] Patches to distutils for 3.6
Message-ID: <CAK=Phk5JF40-ME6v8n1-EyJGpAraatECgfJPh9gYuSmEGmYMGg@mail.gmail.com>

Hi All,

There are a couple of related patches that were submitted to distutils a
few months ago that I think would be great to have in before the feature
freeze in 3.6b1

A bug fix in CCompiler.has_function
    http://bugs.python.org/issue25544

Adding a has_flag method to CCompiler
    http://bugs.python.org/issue26689

    useful to check if a compiler has certain flags available (such as
-std=c++11)

Is there any chance of having this merged before the release?

Thanks,

Sylvain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160819/6a67be95/attachment.html>

From eric at trueblade.com  Fri Aug 19 04:35:33 2016
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 19 Aug 2016 04:35:33 -0400
Subject: [Python-Dev] Adding PEP 515's '_' formatting
Message-ID: <b66b8a5e-0d47-a05a-49ff-f397e51f0813@trueblade.com>

PEP 515 adds underscores to numeric literals. As part of that, it adds 
optional underscores to numeric formatters (similar to PEP 378 for ','). 
See https://www.python.org/dev/peps/pep-0515/#further-changes

I had assigned myself http://bugs.python.org/issue27080 to implement 
this. Unfortunately, I'm not going to have time to implement this before 
3.6 Beta 1. This isn't a hugely complex project, and it doesn't require 
knowledge of python internals. So someone with decent C skills should be 
able to accomplish it in less than a day. I'm hoping someone can pick it 
up. I can offer some pointers if needed.

Eric.

From berker.peksag at gmail.com  Fri Aug 19 04:40:36 2016
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Fri, 19 Aug 2016 11:40:36 +0300
Subject: [Python-Dev] Patches to distutils for 3.6
In-Reply-To: <CAK=Phk5JF40-ME6v8n1-EyJGpAraatECgfJPh9gYuSmEGmYMGg@mail.gmail.com>
References: <CAK=Phk5JF40-ME6v8n1-EyJGpAraatECgfJPh9gYuSmEGmYMGg@mail.gmail.com>
Message-ID: <CAF4280+k6h5dsWoVvzTiDv+KhaGnwomp=ZL1sq9=OuvgcweVZw@mail.gmail.com>

On Fri, Aug 19, 2016 at 11:07 AM, Sylvain Corlay
<sylvain.corlay at gmail.com> wrote:
> Hi All,
>
> There are a couple of related patches that were submitted to distutils a few
> months ago that I think would be great to have in before the feature freeze
> in 3.6b1
>
> A bug fix in CCompiler.has_function
>     http://bugs.python.org/issue25544

Hi Sylvain,

This doesn't look like a new feature to me so it can be committed
after feature freeze. However, the patch needs to be tested in
Windows.

> Adding a has_flag method to CCompiler
>     http://bugs.python.org/issue26689
>
>     useful to check if a compiler has certain flags available (such as
> -std=c++11)

This needs a test case and documentation update to
https://docs.python.org/3.5/distutils/apiref.html#distutils.ccompiler.CCompiler
NamedTemporaryFile might not work as expected in Windows (we have some
open issues about NamedTemporaryFile, but I'm not a Windows user.)

From sylvain.corlay at gmail.com  Fri Aug 19 04:47:45 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Fri, 19 Aug 2016 10:47:45 +0200
Subject: [Python-Dev] Patches to distutils for 3.6
In-Reply-To: <CAF4280+k6h5dsWoVvzTiDv+KhaGnwomp=ZL1sq9=OuvgcweVZw@mail.gmail.com>
References: <CAK=Phk5JF40-ME6v8n1-EyJGpAraatECgfJPh9gYuSmEGmYMGg@mail.gmail.com>
 <CAF4280+k6h5dsWoVvzTiDv+KhaGnwomp=ZL1sq9=OuvgcweVZw@mail.gmail.com>
Message-ID: <CAK=Phk7duPkrGUcrwpob7AKZk09HA0ajQxA2h2Bb1YHqcoeE9w@mail.gmail.com>

Hi Berker,

Thanks for your reply.

If `NamedTemporaryFile` has a bug on windows, should it impact the
implementation of this?

Sylvain


> > Adding a has_flag method to CCompiler
> >     http://bugs.python.org/issue26689
> >
> >     useful to check if a compiler has certain flags available (such as
> > -std=c++11)
>
> This needs a test case and documentation update to
> https://docs.python.org/3.5/distutils/apiref.html#distutils.
> ccompiler.CCompiler
> NamedTemporaryFile might not work as expected in Windows (we have some
> open issues about NamedTemporaryFile, but I'm not a Windows user.)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160819/42529ba0/attachment.html>

From rosuav at gmail.com  Fri Aug 19 09:16:59 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 19 Aug 2016 23:16:59 +1000
Subject: [Python-Dev] Adding PEP 515's '_' formatting
In-Reply-To: <b66b8a5e-0d47-a05a-49ff-f397e51f0813@trueblade.com>
References: <b66b8a5e-0d47-a05a-49ff-f397e51f0813@trueblade.com>
Message-ID: <CAPTjJmpMSm61fnZgTZO85PCLOXhqNg16-6jzxYjMA7CaZgMnzg@mail.gmail.com>

On Fri, Aug 19, 2016 at 6:35 PM, Eric V. Smith <eric at trueblade.com> wrote:
> PEP 515 adds underscores to numeric literals. As part of that, it adds
> optional underscores to numeric formatters (similar to PEP 378 for ','). See
> https://www.python.org/dev/peps/pep-0515/#further-changes
>
> I had assigned myself http://bugs.python.org/issue27080 to implement this.
> Unfortunately, I'm not going to have time to implement this before 3.6 Beta
> 1. This isn't a hugely complex project, and it doesn't require knowledge of
> python internals. So someone with decent C skills should be able to
> accomplish it in less than a day. I'm hoping someone can pick it up. I can
> offer some pointers if needed.

I'm looking into it. Looks like it's part of the default-locale code -
and fortunately doesn't involve messing with actual locale data.

ChrisA

From rosuav at gmail.com  Fri Aug 19 10:13:45 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Aug 2016 00:13:45 +1000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
Message-ID: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>

On my main dev system (Debian Stretch), I've had a single
long-standing test failure - test_site.py,
StartupImportTests.test_startup_imports. It's annoying (partly because
it's such a noisy failure), and doesn't appear to be happening on the
buildbots, nor presumably on core devs' computers, so it's probably
not a Python bug. How can I go about figuring out what's actually
going on here?

The test effectively runs this (but with -v, which makes it noisy):

$ ./python -I -c 'import sys; print(set(sys.modules))'
{'_frozen_importlib_external', 'keyword', '_stat', 'weakref',
'sysconfig', 'genericpath', '_sysconfigdata_m', '_io', '_operator',
'_bootlocale', 'encodings.utf_8', 'itertools', '_locale', 'codecs',
'_weakrefset', 'operator', '_frozen_importlib', 'types',
'_sitebuiltins', 'os', 'posixpath', '_functools', '__main__',
'_weakref', '_warnings', '_codecs', 'builtins', 'posix', 'functools',
'io', 'encodings.aliases', 'collections.abc', 'encodings', '_thread',
'collections', 'errno', '_collections', '_collections_abc', '_heapq',
'_imp', 'site', 'sys', 'reprlib', '_signal', 'stat', 'zipimport',
'abc', 'encodings.latin_1', 'os.path', 'marshal', 'heapq',
'mpl_toolkits'}

It then runs into a failure when it asserts that this set has no
intersection with the set of collection_mods. Apparently 'import sys'
on my system (even with -I for isolation mode) loads up a bunch of
modules that it shouldn't be loading. How do I track down what and
why, and figure out whether it's a config problem unique to my system
or not?

ChrisA

From random832 at fastmail.com  Fri Aug 19 10:31:35 2016
From: random832 at fastmail.com (Random832)
Date: Fri, 19 Aug 2016 10:31:35 -0400
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
Message-ID: <1471617095.3258831.700207393.091C310C@webmail.messagingengine.com>

On Fri, Aug 19, 2016, at 10:13, Chris Angelico wrote:
> On my main dev system (Debian Stretch), I've had a single
> long-standing test failure - test_site.py,
> StartupImportTests.test_startup_imports. It's annoying (partly because
> it's such a noisy failure), and doesn't appear to be happening on the
> buildbots, nor presumably on core devs' computers, so it's probably
> not a Python bug. How can I go about figuring out what's actually
> going on here?
> 
> The test effectively runs this (but with -v, which makes it noisy):

the -v output might be helpful in determining what is causing these
modules to be imported. It would at least show what order they're
imported in.

FWIW, the list of modules that yours has that are not present in my
3.5.2:
- _bootlocale _collections _functools _heapq _locale _operator
- collections collections.abc functools heapq itertools keyword
- mpl_toolkits operator reprlib types weakref

Modules that mine has imported but yours does not:
- _sysconfigdata sitecustomize

> It then runs into a failure when it asserts that this set has no
> intersection with the set of collection_mods. Apparently 'import sys'
> on my system (even with -I for isolation mode) loads up a bunch of
> modules that it shouldn't be loading.

I very much doubt that "import sys" is responsible here.

> How do I track down what and
> why, and figure out whether it's a config problem unique to my system
> or not?

From rosuav at gmail.com  Fri Aug 19 12:02:00 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Aug 2016 02:02:00 +1000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <1471617095.3258831.700207393.091C310C@webmail.messagingengine.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <1471617095.3258831.700207393.091C310C@webmail.messagingengine.com>
Message-ID: <CAPTjJmrfe+cmmmpAtbfftoG4njnCvsOXtUHLRtc3Lq9dnMNO7Q@mail.gmail.com>

On Sat, Aug 20, 2016 at 12:31 AM, Random832 <random832 at fastmail.com> wrote:
> the -v output might be helpful in determining what is causing these
> modules to be imported. It would at least show what order they're
> imported in.

Here it is in all its spammy glory.

rosuav at sikorsky:~/cpython$ ./python -I -v -c 'import sys;
print(set(sys.modules))'
import _frozen_importlib # frozen
import _imp # builtin
import sys # builtin
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
# /home/rosuav/cpython/Lib/encodings/__pycache__/__init__.cpython-36.pyc
matches /home/rosuav/cpython/Lib/encodings/__init__.py
# code object from
'/home/rosuav/cpython/Lib/encodings/__pycache__/__init__.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/codecs.cpython-36.pyc matches
/home/rosuav/cpython/Lib/codecs.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/codecs.cpython-36.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object
at 0x7f6d060ccda0>
# /home/rosuav/cpython/Lib/encodings/__pycache__/aliases.cpython-36.pyc
matches /home/rosuav/cpython/Lib/encodings/aliases.py
# code object from
'/home/rosuav/cpython/Lib/encodings/__pycache__/aliases.cpython-36.pyc'
import 'encodings.aliases' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d060e1898>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060cc978>
# /home/rosuav/cpython/Lib/encodings/__pycache__/utf_8.cpython-36.pyc
matches /home/rosuav/cpython/Lib/encodings/utf_8.py
# code object from
'/home/rosuav/cpython/Lib/encodings/__pycache__/utf_8.cpython-36.pyc'
import 'encodings.utf_8' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d060f0710>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# /home/rosuav/cpython/Lib/encodings/__pycache__/latin_1.cpython-36.pyc
matches /home/rosuav/cpython/Lib/encodings/latin_1.py
# code object from
'/home/rosuav/cpython/Lib/encodings/__pycache__/latin_1.cpython-36.pyc'
import 'encodings.latin_1' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d060f4278>
# /home/rosuav/cpython/Lib/__pycache__/io.cpython-36.pyc matches
/home/rosuav/cpython/Lib/io.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/io.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/abc.cpython-36.pyc matches
/home/rosuav/cpython/Lib/abc.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/abc.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/_weakrefset.cpython-36.pyc
matches /home/rosuav/cpython/Lib/_weakrefset.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/_weakrefset.cpython-36.pyc'
import '_weakrefset' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060f90f0>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at
0x7f6d060f47f0>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at
0x7f6d060f44a8>
# /home/rosuav/cpython/Lib/__pycache__/site.cpython-36.pyc matches
/home/rosuav/cpython/Lib/site.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/site.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/os.cpython-36.pyc matches
/home/rosuav/cpython/Lib/os.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/os.cpython-36.pyc'
import 'errno' # <class '_frozen_importlib.BuiltinImporter'>
# /home/rosuav/cpython/Lib/__pycache__/stat.cpython-36.pyc matches
/home/rosuav/cpython/Lib/stat.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/stat.cpython-36.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at
0x7f6d060a1940>
# /home/rosuav/cpython/Lib/__pycache__/posixpath.cpython-36.pyc
matches /home/rosuav/cpython/Lib/posixpath.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/posixpath.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/genericpath.cpython-36.pyc
matches /home/rosuav/cpython/Lib/genericpath.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/genericpath.cpython-36.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060ab2e8>
import 'posixpath' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060a1fd0>
# /home/rosuav/cpython/Lib/__pycache__/_collections_abc.cpython-36.pyc
matches /home/rosuav/cpython/Lib/_collections_abc.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/_collections_abc.cpython-36.pyc'
import '_collections_abc' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d060ab8d0>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at
0x7f6d0608cf98>
# /home/rosuav/cpython/Lib/__pycache__/_sitebuiltins.cpython-36.pyc
matches /home/rosuav/cpython/Lib/_sitebuiltins.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/_sitebuiltins.cpython-36.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060912b0>
# /home/rosuav/cpython/Lib/__pycache__/sysconfig.cpython-36.pyc
matches /home/rosuav/cpython/Lib/sysconfig.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/sysconfig.cpython-36.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d06073240>
# /home/rosuav/cpython/build/lib.linux-x86_64-3.6/__pycache__/_sysconfigdata_m.cpython-36.pyc
matches /home/rosuav/cpython/build/lib.linux-x86_64-3.6/_sysconfigdata_m.py
# code object from
'/home/rosuav/cpython/build/lib.linux-x86_64-3.6/__pycache__/_sysconfigdata_m.cpython-36.pyc'
import '_sysconfigdata_m' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d0607ac88>
# /home/rosuav/cpython/Lib/__pycache__/_bootlocale.cpython-36.pyc
matches /home/rosuav/cpython/Lib/_bootlocale.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/_bootlocale.cpython-36.pyc'
import '_locale' # <class '_frozen_importlib.BuiltinImporter'>
import '_bootlocale' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d06082fd0>
# /home/rosuav/cpython/Lib/__pycache__/types.cpython-36.pyc matches
/home/rosuav/cpython/Lib/types.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/types.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/functools.cpython-36.pyc
matches /home/rosuav/cpython/Lib/functools.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/functools.cpython-36.pyc'
import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
# /home/rosuav/cpython/Lib/collections/__pycache__/__init__.cpython-36.pyc
matches /home/rosuav/cpython/Lib/collections/__init__.py
# code object from
'/home/rosuav/cpython/Lib/collections/__pycache__/__init__.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/operator.cpython-36.pyc matches
/home/rosuav/cpython/Lib/operator.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/operator.cpython-36.pyc'
import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
import 'operator' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d06041c50>
# /home/rosuav/cpython/Lib/__pycache__/keyword.cpython-36.pyc matches
/home/rosuav/cpython/Lib/keyword.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/keyword.cpython-36.pyc'
import 'keyword' # <_frozen_importlib_external.SourceFileLoader object
at 0x7f6d0604bfd0>
# /home/rosuav/cpython/Lib/__pycache__/heapq.cpython-36.pyc matches
/home/rosuav/cpython/Lib/heapq.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/heapq.cpython-36.pyc'
# extension module '_heapq' loaded from
'/home/rosuav/cpython/build/lib.linux-x86_64-3.6/_heapq.cpython-36m-x86_64-linux-gnu.so'
# extension module '_heapq' executed from
'/home/rosuav/cpython/build/lib.linux-x86_64-3.6/_heapq.cpython-36m-x86_64-linux-gnu.so'
import '_heapq' # <_frozen_importlib_external.ExtensionFileLoader
object at 0x7f6d043d3d68>
import 'heapq' # <_frozen_importlib_external.SourceFileLoader object
at 0x7f6d043d3898>
import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
# /home/rosuav/cpython/Lib/__pycache__/reprlib.cpython-36.pyc matches
/home/rosuav/cpython/Lib/reprlib.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/reprlib.cpython-36.pyc'
import 'reprlib' # <_frozen_importlib_external.SourceFileLoader object
at 0x7f6d0604b908>
import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
import 'collections' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060242b0>
# /home/rosuav/cpython/Lib/__pycache__/weakref.cpython-36.pyc matches
/home/rosuav/cpython/Lib/weakref.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/weakref.cpython-36.pyc'
import 'weakref' # <_frozen_importlib_external.SourceFileLoader object
at 0x7f6d06024630>
import 'functools' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d06017ba8>
# /home/rosuav/cpython/Lib/collections/__pycache__/abc.cpython-36.pyc
matches /home/rosuav/cpython/Lib/collections/abc.py
# code object from
'/home/rosuav/cpython/Lib/collections/__pycache__/abc.cpython-36.pyc'
import 'collections.abc' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d06019898>
import 'types' # <_frozen_importlib_external.SourceFileLoader object
at 0x7f6d0600ffd0>
import 'site' # <_frozen_importlib_external.SourceFileLoader object at
0x7f6d06102a90>
Python 3.6.0a4+ (default:4b64a049f451+, Aug 19 2016, 23:41:43)
[GCC 6.1.1 20160802] on linux
Type "help", "copyright", "credits" or "license" for more information.
{'collections', 'io', 'encodings', '_io', 'marshal', 'weakref',
'genericpath', 'os', '_imp', 'codecs', '_weakrefset', '_locale',
'keyword', '__main__', 'stat', '_sitebuiltins', 'collections.abc',
'site', 'posix', 'encodings.aliases', 'errno', '_weakref',
'_collections_abc', 'os.path', '_collections', '_heapq', 'reprlib',
'_warnings', '_frozen_importlib', 'heapq', '_stat',
'encodings.latin_1', '_operator', '_sysconfigdata_m', 'types',
'zipimport', 'posixpath', 'builtins', '_thread', 'sysconfig',
'encodings.utf_8', 'mpl_toolkits', '_functools', 'itertools',
'_codecs', 'abc', 'operator', 'functools', 'sys', '_bootlocale',
'_signal', '_frozen_importlib_external'}
# clear builtins._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.__interactivehook__
# clear sys.flags
# clear sys.float_info
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup[2] removing collections
# cleanup[2] removing io
# destroy io
# cleanup[2] removing _collections
# cleanup[2] removing sysconfig
# destroy sysconfig
# cleanup[2] removing _heapq
# cleanup[2] removing encodings
# destroy encodings
# cleanup[2] removing _io
# cleanup[2] removing marshal
# cleanup[2] removing encodings.latin_1
# cleanup[2] removing _warnings
# cleanup[2] removing _operator
# cleanup[2] removing _frozen_importlib
# cleanup[2] removing genericpath
# cleanup[2] removing heapq
# cleanup[2] removing os
# cleanup[2] removing _imp
# cleanup[2] removing itertools
# cleanup[2] removing codecs
# cleanup[2] removing _sysconfigdata_m
# destroy _sysconfigdata_m
# cleanup[2] removing _weakrefset
# destroy _weakrefset
# cleanup[2] removing types
# destroy types
# cleanup[2] removing zipimport
# cleanup[2] removing posixpath
# cleanup[2] removing _locale
# cleanup[2] removing builtins
# cleanup[2] removing keyword
# destroy keyword
# cleanup[2] removing encodings.utf_8
# cleanup[2] removing mpl_toolkits
# destroy mpl_toolkits
# cleanup[2] removing __main__
# destroy __main__
# cleanup[2] removing _functools
# cleanup[2] removing _sitebuiltins
# cleanup[2] removing collections.abc
# cleanup[2] removing _thread
# cleanup[2] removing site
# destroy site
# cleanup[2] removing reprlib
# destroy reprlib
# cleanup[2] removing posix
# cleanup[2] removing _codecs
# cleanup[2] removing stat
# cleanup[2] removing abc
# cleanup[2] removing encodings.aliases
# cleanup[2] removing errno
# cleanup[2] removing _stat
# cleanup[2] removing _weakref
# cleanup[2] removing operator
# destroy operator
# cleanup[2] removing functools
# cleanup[2] removing _collections_abc
# cleanup[2] removing _bootlocale
# destroy _bootlocale
# cleanup[2] removing _signal
# cleanup[2] removing _frozen_importlib_external
# cleanup[2] removing sys
# cleanup[2] removing os.path
# cleanup[2] removing weakref
# destroy weakref
# destroy zipimport
# destroy _signal
# destroy _sitebuiltins
# destroy errno
# destroy posixpath
# destroy abc
# destroy _stat
# destroy genericpath
# destroy stat
# destroy os
# destroy _locale
# destroy functools
# destroy _functools
# destroy heapq
# destroy collections.abc
# destroy _collections_abc
# destroy _operator
# destroy _heapq
# destroy _collections
# destroy collections
# destroy itertools
# cleanup[3] wiping _io
# cleanup[3] wiping marshal
# cleanup[3] wiping encodings.latin_1
# cleanup[3] wiping _warnings
# cleanup[3] wiping _frozen_importlib
# destroy _frozen_importlib_external
# cleanup[3] wiping _imp
# cleanup[3] wiping codecs
# cleanup[3] wiping encodings.utf_8
# cleanup[3] wiping _thread
# cleanup[3] wiping posix
# cleanup[3] wiping _codecs
# cleanup[3] wiping encodings.aliases
# cleanup[3] wiping _weakref
# cleanup[3] wiping sys
# cleanup[3] wiping builtins
rosuav at sikorsky:~/cpython$


ChrisA

From status at bugs.python.org  Fri Aug 19 12:08:45 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 19 Aug 2016 18:08:45 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160819160845.8995F567B9@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-08-12 - 2016-08-19)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5594 ( -1)
  closed 33965 (+56)
  total  39559 (+55)

Open issues with patches: 2438 


Issues opened (34)
==================

#26988: Add AutoNumberedEnum to stdlib
http://bugs.python.org/issue26988  reopened by rhettinger

#27611: test_tix cannot import _default_root after test_idle
http://bugs.python.org/issue27611  reopened by serhiy.storchaka

#27748: Simplify test_winsound
http://bugs.python.org/issue27748  opened by zach.ware

#27749: multprocessing errors on Windows: WriteFile() argument 1 must 
http://bugs.python.org/issue27749  opened by wevsty

#27752: CSV DictReader default dialect name 'excel' is misleading, as 
http://bugs.python.org/issue27752  opened by lockywolf

#27755: Retire DynOptionMenu with a ttk Combobox
http://bugs.python.org/issue27755  opened by jfoo

#27756: Add pyd icon for 3.6
http://bugs.python.org/issue27756  opened by Decorater

#27759: selectors incorrectly retain invalid file descriptors
http://bugs.python.org/issue27759  opened by Mark.Williams

#27761: Private _nth_root function loses accuracy
http://bugs.python.org/issue27761  opened by steven.daprano

#27763: Add complex case to test_builtin abs()
http://bugs.python.org/issue27763  opened by Evelyn Mitchell

#27764: [Patch] Complete bits necessary for making fcntl's file lockin
http://bugs.python.org/issue27764  opened by EdSchouten

#27766: Add ChaCha20 Poly1305 to SSL ciphers
http://bugs.python.org/issue27766  opened by christian.heimes

#27768: ssl: add public API for IA-32 processor capabilities vector
http://bugs.python.org/issue27768  opened by christian.heimes

#27769: IDLE: Replace All up, no wrap replaces one up, all down
http://bugs.python.org/issue27769  opened by Qwert225

#27772: Refer to actual format string when creating ???zero padding???
http://bugs.python.org/issue27772  opened by bignose

#27776: PEP 524: Make os.urandom() blocking on Linux
http://bugs.python.org/issue27776  opened by haypo

#27777: cgi.FieldStorage can't parse simple body with Content-Length a
http://bugs.python.org/issue27777  opened by rr-

#27778: PEP 524: Add os.getrandom()
http://bugs.python.org/issue27778  opened by haypo

#27779: Sync-up docstrings in C version of the the decimal module
http://bugs.python.org/issue27779  opened by rhettinger

#27780: memory leaks in pgen build step abort build with address sanit
http://bugs.python.org/issue27780  opened by geeknik

#27781: Change sys.getfilesystemencoding() on Windows to UTF-8
http://bugs.python.org/issue27781  opened by steve.dower

#27782: Multi-phase extension module initialization, inconsistent exce
http://bugs.python.org/issue27782  opened by xiang.zhang

#27784: Random failure of test_TCPServer() of test.test_socketserver.S
http://bugs.python.org/issue27784  opened by haypo

#27787: Avoid test_main() in test_httplib; gc.collect() dangling threa
http://bugs.python.org/issue27787  opened by martin.panter

#27788: platform module's version number doesn't match its docstring
http://bugs.python.org/issue27788  opened by brett.cannon

#27790: test_distutils spews linker messages on Windows
http://bugs.python.org/issue27790  opened by terry.reedy

#27791: test_threading: test_threads_join_2() failed with "Fatal Pytho
http://bugs.python.org/issue27791  opened by haypo

#27792: bool % int has inconsistent return type.
http://bugs.python.org/issue27792  opened by mark.dickinson

#27794: setattr a read-only property; the AttributeError should show t
http://bugs.python.org/issue27794  opened by ztane

#27796: Expose DirEntry constructor
http://bugs.python.org/issue27796  opened by moloney

#27797: ASCII file with UNIX line conventions and enough lines throws 
http://bugs.python.org/issue27797  opened by mjpieters

#27799: Fix base64-codec and bz2-codec incremental decoders
http://bugs.python.org/issue27799  opened by martin.panter

#27800: Regular expressions with multiple repeat codes
http://bugs.python.org/issue27800  opened by martin.panter

#27802: Add __eq__ and __ne__ to collections.abc.Sequence.
http://bugs.python.org/issue27802  opened by neil.g



Most recent 15 issues with no replies (15)
==========================================

#27799: Fix base64-codec and bz2-codec incremental decoders
http://bugs.python.org/issue27799

#27796: Expose DirEntry constructor
http://bugs.python.org/issue27796

#27772: Refer to actual format string when creating ???zero padding???
http://bugs.python.org/issue27772

#27764: [Patch] Complete bits necessary for making fcntl's file lockin
http://bugs.python.org/issue27764

#27748: Simplify test_winsound
http://bugs.python.org/issue27748

#27741: datetime.datetime.strptime functionality description incorrect
http://bugs.python.org/issue27741

#27737: email.header.Header.encode() crashes with IndexError on spaces
http://bugs.python.org/issue27737

#27733: Test that self.bell is called in IDLE classes when intended.
http://bugs.python.org/issue27733

#27713: Spurious "platform dependent libraries" warnings when running 
http://bugs.python.org/issue27713

#27705: Updating old C:/Windows/System32/ucrtbased.dll
http://bugs.python.org/issue27705

#27701: [posixmodule] [Refactoring patch] Simply call into *at() funct
http://bugs.python.org/issue27701

#27696: win_add2path.py does not add user site directory
http://bugs.python.org/issue27696

#27691: X509 cert with GEN_RID subject alt name causes SytemError
http://bugs.python.org/issue27691

#27679: set_bitfields() unused in _ctypes_test
http://bugs.python.org/issue27679

#27666: "stack smashing detected" in PyCursesWindow_Box
http://bugs.python.org/issue27666



Most recent 15 issues waiting for review (15)
=============================================

#27802: Add __eq__ and __ne__ to collections.abc.Sequence.
http://bugs.python.org/issue27802

#27799: Fix base64-codec and bz2-codec incremental decoders
http://bugs.python.org/issue27799

#27787: Avoid test_main() in test_httplib; gc.collect() dangling threa
http://bugs.python.org/issue27787

#27782: Multi-phase extension module initialization, inconsistent exce
http://bugs.python.org/issue27782

#27781: Change sys.getfilesystemencoding() on Windows to UTF-8
http://bugs.python.org/issue27781

#27778: PEP 524: Add os.getrandom()
http://bugs.python.org/issue27778

#27776: PEP 524: Make os.urandom() blocking on Linux
http://bugs.python.org/issue27776

#27766: Add ChaCha20 Poly1305 to SSL ciphers
http://bugs.python.org/issue27766

#27759: selectors incorrectly retain invalid file descriptors
http://bugs.python.org/issue27759

#27756: Add pyd icon for 3.6
http://bugs.python.org/issue27756

#27755: Retire DynOptionMenu with a ttk Combobox
http://bugs.python.org/issue27755

#27748: Simplify test_winsound
http://bugs.python.org/issue27748

#27746: ResourceWarnings in test_asyncio
http://bugs.python.org/issue27746

#27744: Add AF_ALG (Linux Kernel crypto) to socket module
http://bugs.python.org/issue27744

#27740: Fix doc of Py_CompileStringExFlags
http://bugs.python.org/issue27740



Top 10 most discussed issues (10)
=================================

#26988: Add AutoNumberedEnum to stdlib
http://bugs.python.org/issue26988  27 msgs

#27761: Private _nth_root function loses accuracy
http://bugs.python.org/issue27761  22 msgs

#23591: enum: Add Flags and IntFlags
http://bugs.python.org/issue23591  19 msgs

#27749: multprocessing errors on Windows: WriteFile() argument 1 must 
http://bugs.python.org/issue27749  11 msgs

#27598: Add Collection to collections.abc and typing
http://bugs.python.org/issue27598  10 msgs

#27766: Add ChaCha20 Poly1305 to SSL ciphers
http://bugs.python.org/issue27766  10 msgs

#27781: Change sys.getfilesystemencoding() on Windows to UTF-8
http://bugs.python.org/issue27781   9 msgs

#1602: windows console doesn't print or input Unicode
http://bugs.python.org/issue1602   8 msgs

#27181: Add geometric mean to `statistics` module
http://bugs.python.org/issue27181   8 msgs

#16764: Make zlib accept keyword-arguments
http://bugs.python.org/issue16764   7 msgs



Issues closed (53)
==================

#9998: ctypes find_library should search LD_LIBRARY_PATH on Linux
http://bugs.python.org/issue9998  closed by python-dev

#10910: pyport.h FreeBSD/Mac OS X "fix" causes errors in C++ compilati
http://bugs.python.org/issue10910  closed by ned.deily

#12713: argparse: allow abbreviation of sub commands by users
http://bugs.python.org/issue12713  closed by python-dev

#12946: PyModule_GetDict() claims it can never fail, but it can
http://bugs.python.org/issue12946  closed by berker.peksag

#22395: test_pathlib error for complex symlinks on Windows
http://bugs.python.org/issue22395  closed by jfoo

#24637: locals dictionary in PyRun_String
http://bugs.python.org/issue24637  closed by berker.peksag

#24853: Py_Finalize doesn't clean up PyImport_Inittab
http://bugs.python.org/issue24853  closed by haypo

#25628: Make namedtuple "verbose" and "rename" parameters into keyword
http://bugs.python.org/issue25628  closed by rhettinger

#25805: Failure in test_pkgutil run from command-line
http://bugs.python.org/issue25805  closed by brett.cannon

#25958: Implicit ABCs have no means of "anti-registration"
http://bugs.python.org/issue25958  closed by gvanrossum

#26200: SETREF adds unnecessary work in some cases
http://bugs.python.org/issue26200  closed by serhiy.storchaka

#26754: PyUnicode_FSDecoder() accepts arbitrary iterable
http://bugs.python.org/issue26754  closed by serhiy.storchaka

#26800: Don't accept bytearray as filenames part 2
http://bugs.python.org/issue26800  closed by serhiy.storchaka

#26823: Shrink recursive tracebacks
http://bugs.python.org/issue26823  closed by ncoghlan

#27173: Modern Unix key bindings for IDLE
http://bugs.python.org/issue27173  closed by terry.reedy

#27558: SystemError with bare `raise` in threading or multiprocessing
http://bugs.python.org/issue27558  closed by haypo

#27573: code.interact() should print an exit message
http://bugs.python.org/issue27573  closed by steven.daprano

#27594: Assertion failure when running "test_ast" tests with coverage.
http://bugs.python.org/issue27594  closed by ned.deily

#27698: socketpair not in socket.__all__ on Windows
http://bugs.python.org/issue27698  closed by haypo

#27704: bytes(x) is slow when x is bytearray
http://bugs.python.org/issue27704  closed by serhiy.storchaka

#27712: Tiny typos in import.rst
http://bugs.python.org/issue27712  closed by brett.cannon

#27719: Misleading note about "args" attribute in "User-defined Except
http://bugs.python.org/issue27719  closed by rhettinger

#27720: decimal.Context.to_eng_string wrong docstring
http://bugs.python.org/issue27720  closed by rhettinger

#27723: Document typing.Text and typing.AnyStr
http://bugs.python.org/issue27723  closed by gvanrossum

#27725: Use Py_SIZE(x) instead of x->ob_size
http://bugs.python.org/issue27725  closed by skrah

#27726: ctags -t does not exists in Makefile.pre.in
http://bugs.python.org/issue27726  closed by haypo

#27734: Memory leaks at Python35-32
http://bugs.python.org/issue27734  closed by r.david.murray

#27736: repeated Py_Initialize/PyRun_SimpleString/Py_Finalize segfault
http://bugs.python.org/issue27736  closed by ned.deily

#27742: Random.randint generates different values in Python2 and Pytho
http://bugs.python.org/issue27742  closed by rhettinger

#27747: Broken example in the queue module documentation
http://bugs.python.org/issue27747  closed by Paulo Gabriel Poiati

#27750: Idle Shell freezes when printing 100000+ char lines.
http://bugs.python.org/issue27750  closed by terry.reedy

#27751: Itertools -> Recipes -> pairwise()
http://bugs.python.org/issue27751  closed by YoSTEALTH

#27753: py_compile.compile() does not specify if it can accept a file 
http://bugs.python.org/issue27753  closed by r.david.murray

#27754: Changing the meaning of bool.__invert__
http://bugs.python.org/issue27754  closed by benjamin.peterson

#27757: eval() does not allow import statements to run.
http://bugs.python.org/issue27757  closed by steven.daprano

#27758: integer overflow in the _csv module's join_append_data functio
http://bugs.python.org/issue27758  closed by python-dev

#27760: integer overflow in binascii.b2a_qp
http://bugs.python.org/issue27760  closed by python-dev

#27762: ConfigParser ignores in-line comments for lines with semi-colo
http://bugs.python.org/issue27762  closed by lukasz.langa

#27765: Accept UTF-8 encoded bytes as input to json.loads()
http://bugs.python.org/issue27765  closed by haypo

#27767: Receive "A required privilege is not held by the Client" error
http://bugs.python.org/issue27767  closed by ebarry

#27770: remove run_until_complete
http://bugs.python.org/issue27770  closed by gvanrossum

#27771: Add a clear screen button or menu choice for IDLE
http://bugs.python.org/issue27771  closed by terry.reedy

#27773: Excessive Py_XDECREF in the ssl module:
http://bugs.python.org/issue27773  closed by python-dev

#27774: Py_DECREF on a non-owned object in _sre
http://bugs.python.org/issue27774  closed by python-dev

#27775: `shlex.split(None)` causes infinite loop
http://bugs.python.org/issue27775  closed by r.david.murray

#27783: potential use of unitialized memory in operator.methodcaller
http://bugs.python.org/issue27783  closed by python-dev

#27785: Module platform: Versions of Windows
http://bugs.python.org/issue27785  closed by steve.dower

#27786: longobject.c: simplify x_sub(), inline _PyLong_Negate()
http://bugs.python.org/issue27786  closed by haypo

#27789: test_asyncio Resource Warnings
http://bugs.python.org/issue27789  closed by berker.peksag

#27793: Double underscore variables in module are mangled when used in
http://bugs.python.org/issue27793  closed by r.david.murray

#27795: Cygwin compile errors
http://bugs.python.org/issue27795  closed by r.david.murray

#27798: test_sysconfig: test_srcdir() fails on "x86 Gentoo Installed w
http://bugs.python.org/issue27798  closed by zach.ware

#27801: test_update_lines_cols from test.test_curses.MiscTests fails o
http://bugs.python.org/issue27801  closed by berker.peksag

From rosuav at gmail.com  Fri Aug 19 12:10:19 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Aug 2016 02:10:19 +1000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
Message-ID: <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>

On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower <steve.dower at python.org> wrote:
> Check any .pth files you can find. I suspect mpl_toolkits has some magic in
> it to make the namespace package work on 2.7.

>>> sys.path
['/usr/local/lib/python36.zip', '/home/rosuav/cpython/Lib',
'/home/rosuav/cpython/Lib/plat-x86_64-linux-gnu',
'/home/rosuav/cpython/build/lib.linux-x86_64-3.6',
'/usr/local/lib/python3.6/site-packages']

/usr/local/lib/python36.zip doesn't exist. In
/usr/local/lib/python3.6/site-packages are a bunch of things that I've
installed using pip, but grepping for '/usr' in the -v output comes up
blank. There is, however, one .pth file there:

$ cat /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
import sys, types, os;p =
os.path.join(sys._getframe(1).f_locals['sitedir'],
*('mpl_toolkits',));ie =
os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
sys.modules.setdefault('mpl_toolkits',
types.ModuleType('mpl_toolkits'));mp = (m or []) and
m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)

Is it possible that that's the cause? And if so, should "python -I"
remove /usr/local/lib/python3.6/site-packages from sys.path?

ChrisA

From steve.dower at python.org  Fri Aug 19 12:25:43 2016
From: steve.dower at python.org (Steve Dower)
Date: Fri, 19 Aug 2016 09:25:43 -0700
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
 <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
Message-ID: <4a611645-db9e-5da6-ede5-15e87427447a@python.org>

On 19Aug2016 0910, Chris Angelico wrote:
> On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower <steve.dower at python.org> wrote:
>> Check any .pth files you can find. I suspect mpl_toolkits has some magic in
>> it to make the namespace package work on 2.7.
>
> $ cat /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
> import sys, types, os;p =
> os.path.join(sys._getframe(1).f_locals['sitedir'],
> *('mpl_toolkits',));ie =
> os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
> sys.modules.setdefault('mpl_toolkits',
> types.ModuleType('mpl_toolkits'));mp = (m or []) and
> m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
>
> Is it possible that that's the cause? And if so, should "python -I"
> remove /usr/local/lib/python3.6/site-packages from sys.path?

That explains the extra modules - types.py imports collections and 
collections.abc, which would seem to account for the rest.

I'm not sure whether that's causing the problem or not, but temporarily 
renaming that pth file so it isn't picked up may help you track down the 
test issue.

"python -S" is the only way to avoid .pth files, but that's a pretty 
heavy hammer.

Cheers,
Steve


From tjreedy at udel.edu  Fri Aug 19 12:38:45 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 19 Aug 2016 12:38:45 -0400
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
Message-ID: <np7cml$7uc$1@blaine.gmane.org>

On 8/19/2016 10:13 AM, Chris Angelico wrote:
> On my main dev system (Debian Stretch), I've had a single
> long-standing test failure - test_site.py,
> StartupImportTests.test_startup_imports. It's annoying (partly because
> it's such a noisy failure), and doesn't appear to be happening on the
> buildbots, nor presumably on core devs' computers, so it's probably
> not a Python bug. How can I go about figuring out what's actually
> going on here?
>
> The test effectively runs this (but with -v, which makes it noisy):
>
> $ ./python -I -c 'import sys; print(set(sys.modules))'
> {'_frozen_importlib_external', 'keyword', '_stat', 'weakref',
> 'sysconfig', 'genericpath', '_sysconfigdata_m', '_io', '_operator',
> '_bootlocale', 'encodings.utf_8', 'itertools', '_locale', 'codecs',
> '_weakrefset', 'operator', '_frozen_importlib', 'types',
> '_sitebuiltins', 'os', 'posixpath', '_functools', '__main__',
> '_weakref', '_warnings', '_codecs', 'builtins', 'posix', 'functools',
> 'io', 'encodings.aliases', 'collections.abc', 'encodings', '_thread',
> 'collections', 'errno', '_collections', '_collections_abc', '_heapq',
> '_imp', 'site', 'sys', 'reprlib', '_signal', 'stat', 'zipimport',
> 'abc', 'encodings.latin_1', 'os.path', 'marshal', 'heapq',
> 'mpl_toolkits'}

Sorted() gives an easier to compare listing ;-).
On Windows as of 2 days ago (using required " insteadof ') many fewer:

F:\Python\dev>36\pcbuild\win32\python_d.exe  -I -c "import sys; 
print(sorted(sys.modules))"
['__main__', '_bootlocale', '_codecs', '_collections_abc', 
'_frozen_importlib', '_frozen_importlib_external', '_imp', '_io', 
'_locale', '_signal', '_sitebuiltins', '_stat', '_thread', '_warnings', 
'_weakref', '_weakrefset', 'abc', 'builtins', 'codecs', 'encodings', 
'encodings.aliases', 'encodings.cp1252', 'encodings.cp437', 
'encodings.latin_1', 'encodings.mbcs', 'encodings.utf_8', 'errno', 
'genericpath', 'io', 'marshal', 'nt', 'ntpath', 'os', 'os.path', 'site', 
'stat', 'sys', 'sysconfig', 'winreg', 'zipimport']

test_site passes, so _collections_abc is not a problem

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Fri Aug 19 12:40:03 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 19 Aug 2016 12:40:03 -0400
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
 <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
Message-ID: <np7cp2$7uc$2@blaine.gmane.org>

On 8/19/2016 12:10 PM, Chris Angelico wrote:
> On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower <steve.dower at python.org> wrote:
>> Check any .pth files you can find. I suspect mpl_toolkits has some magic in
>> it to make the namespace package work on 2.7.
>
>>>> sys.path
> ['/usr/local/lib/python36.zip', '/home/rosuav/cpython/Lib',
> '/home/rosuav/cpython/Lib/plat-x86_64-linux-gnu',
> '/home/rosuav/cpython/build/lib.linux-x86_64-3.6',
> '/usr/local/lib/python3.6/site-packages']
>
> /usr/local/lib/python36.zip doesn't exist. In
> /usr/local/lib/python3.6/site-packages are a bunch of things that I've
> installed using pip, but grepping for '/usr' in the -v output comes up
> blank. There is, however, one .pth file there:
>
> $ cat /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
> import sys, types, os;p =
> os.path.join(sys._getframe(1).f_locals['sitedir'],
> *('mpl_toolkits',));ie =
> os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
> sys.modules.setdefault('mpl_toolkits',
> types.ModuleType('mpl_toolkits'));mp = (m or []) and
> m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
>
> Is it possible that that's the cause?

Change 'pth' to 'pthx' and try it.

> And if so, should "python -I"
> remove /usr/local/lib/python3.6/site-packages from sys.path?


-- 
Terry Jan Reedy


From rosuav at gmail.com  Fri Aug 19 12:46:30 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Aug 2016 02:46:30 +1000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <4a611645-db9e-5da6-ede5-15e87427447a@python.org>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
 <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
 <4a611645-db9e-5da6-ede5-15e87427447a@python.org>
Message-ID: <CAPTjJmp9fDkTXqpj6ZuE-aW-QLQX-Wjhvxkvp4u=w1wtgphGfA@mail.gmail.com>

On Sat, Aug 20, 2016 at 2:25 AM, Steve Dower <steve.dower at python.org> wrote:
> On 19Aug2016 0910, Chris Angelico wrote:
>>
>> On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower <steve.dower at python.org>
>> wrote:
>>>
>>> Check any .pth files you can find. I suspect mpl_toolkits has some magic
>>> in
>>> it to make the namespace package work on 2.7.
>>
>>
>> $ cat
>> /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
>> import sys, types, os;p =
>> os.path.join(sys._getframe(1).f_locals['sitedir'],
>> *('mpl_toolkits',));ie =
>> os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
>> sys.modules.setdefault('mpl_toolkits',
>> types.ModuleType('mpl_toolkits'));mp = (m or []) and
>> m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
>>
>> Is it possible that that's the cause? And if so, should "python -I"
>> remove /usr/local/lib/python3.6/site-packages from sys.path?
>
>
> That explains the extra modules - types.py imports collections and
> collections.abc, which would seem to account for the rest.
>
> I'm not sure whether that's causing the problem or not, but temporarily
> renaming that pth file so it isn't picked up may help you track down the
> test issue.

Sure enough, that did fix the problem - that is to say, renaming it
did allow test_site to pass. Thanks!

> "python -S" is the only way to avoid .pth files, but that's a pretty heavy
> hammer.

Hmmmmmmm. So the question is, what is this test testing? For
reference, here's the code:

class StartupImportTests(unittest.TestCase):

    def test_startup_imports(self):
        # This tests checks which modules are loaded by Python when it
        # initially starts upon startup.
        popen = subprocess.Popen([sys.executable, '-I', '-v', '-c',
                                  'import sys; print(set(sys.modules))'],
                                 stdout=subprocess.PIPE,
                                 stderr=subprocess.PIPE)
        stdout, stderr = popen.communicate()
        stdout = stdout.decode('utf-8')
        stderr = stderr.decode('utf-8')
        modules = eval(stdout)

        self.assertIn('site', modules)

        # http://bugs.python.org/issue19205
        re_mods = {'re', '_sre', 'sre_compile', 'sre_constants', 'sre_parse'}
        # _osx_support uses the re module in many placs
        if sys.platform != 'darwin':
            self.assertFalse(modules.intersection(re_mods), stderr)
        # http://bugs.python.org/issue9548
        self.assertNotIn('locale', modules, stderr)
        if sys.platform != 'darwin':
            # http://bugs.python.org/issue19209
            self.assertNotIn('copyreg', modules, stderr)
        # http://bugs.python.org/issue19218>
        collection_mods = {'_collections', 'collections', 'functools',
                           'heapq', 'itertools', 'keyword', 'operator',
                           'reprlib', 'types', 'weakref'
                          }.difference(sys.builtin_module_names)
        self.assertFalse(modules.intersection(collection_mods), stderr)


It asserts that 'site' is in the list of loaded modules, so simply
adding '-S' to the subpython parameters doesn't work. But this test,
as written, depends on the behaviour of arbitrary third-party modules
installed with pip, which seems wrong for a standard library test.
AIUI this is trying to catch changes that accidentally cause large
modules to be loaded on startup, correct? Which is all well and good,
but Python shouldn't have test failures due to matplotlib being
installed.

I see two (or three) ways to deal with this.

1) Declare it a bug in matplotlib, and demand that .pth files not
import anything that wouldn't otherwise be imported.

2) Change the test to add '-S' and remove the assertion about 'site'
being present; this means that site.py itself is no longer covered by
the test.

3) Accept the status quo: this test can fail if .pth files are present.

Incidentally, where's it documented that .pth files can be executed?
It's not in the site.py docs [1] or docstring, only in the addpackage
function within that (which I found by browsing the source). Maybe the
solution is to disable that one specific feature when running this
test?

ChrisA

[1] https://docs.python.org/3.6/library/site.html

From sjoerdjob at sjec.nl  Fri Aug 19 12:58:14 2016
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Fri, 19 Aug 2016 18:58:14 +0200
Subject: [Python-Dev] Failures in test_site.py - how to debug?
Message-ID: <3sG8Xm03HRzFqWC@mail.python.org>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160819/8e963a58/attachment.html>

From steve.dower at python.org  Fri Aug 19 14:59:32 2016
From: steve.dower at python.org (Steve Dower)
Date: Fri, 19 Aug 2016 11:59:32 -0700
Subject: [Python-Dev] File system path encoding on Windows
Message-ID: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>

Hi python-dev

About a week ago I proposed on python-ideas making some changes to how 
Python deals with encodings on Windows, specifically in relation to how 
Python interacts with the operating system.

Changes to the console were uncontroversial, and I have posted patches 
at http://bugs.python.org/issue1602 and 
http://bugs.python.org/issue17620 to enable the full range of Unicode 
input to be used at interactive stdin/stdout.

However, changes to sys.getfilesystemencoding(), which determines how 
the os module (and most filesystem functions in general) interpret bytes 
parameters, were more heatedly discussed. I've summarised the discussion 
in this email

I'll declare up front that my preferred change is to treat bytes as 
utf-8 in Python 3.6, and I've posted a patch to do that at 
http://bugs.python.org/issue27781. Hopefully I haven't been too biased 
in my presentation of the alternatives, but this is so you at least know 
which way I'm biased.

I'm looking for some agreement on the answers to the questions I pose in 
the summary.

There is much more detail about them presented after that, as there are 
a number of non-obvious issues at play here. I suspect this will 
eventually become a PEP, but it's presented here as a summary of a 
discussion and not a PEP.

Cheers,
Steve

Summary
=======

Representing file system paths on Windows as bytes may result in data 
loss due to the way Windows encodes/decodes strings via its bytes API.

We can mitigate this by only using Window's Unicode API and doing our 
own encoding and decoding (i.e. within posixmodule.c's path converter). 
Invalid characters could cause encoding exceptions rather than data loss.

We can go further to fix this by declaring the encoding of bytes paths 
on Windows must be utf-8, which would also prevent encoding exceptions, 
as utf-8 can fully represent all paths on Windows (natively utf-16-le).

Even though using bytes for paths on Windows has been deprecated for 
three releases, this is not widely known and it may be too soon to 
change the behaviour.

Questions:
* should we always use Window's Unicode APIs instead of switching 
between bytes/Unicode based on parameter type?
* should we allow users to pass bytes and interpret them as utf-8 rather 
than letting Windows do the decoding?
* should we do it in 3.6, 3.7 or 3.8?

Background
==========

File system paths are almost universally represented as text in some 
encoding determined by the file system. In Python, we expose these paths 
via a number of interfaces, such as the os and io modules. Paths may be 
passed either direction across these interfaces, that is, from the 
filesystem to the application (for example, os.listdir()), or from the 
application to the filesystem (for example, os.unlink()).

When paths are passed between the filesystem and the application, they 
are either passed through as a bytes blob or converted to/from str using 
sys.getfilesystemencoding(). The result of encoding a string with 
sys.getfilesystemencoding() is a blob of bytes in the native format for 
the default file system.

On Windows, the native format for the filesystem is utf-16-le. The 
recommended platform APIs for accessing the filesystem all accept and 
return text encoded in this format. However, prior to Windows NT (and 
possibly further back), the native format was a configurable machine 
option and a separate set of APIs existed to accept this format. The 
option (the "active code page") and these APIs (the "*A functions") 
still exist in recent versions of Windows for backwards compatibility, 
though new functionality often only has a utf-16-le API (the "*W 
functions").

In Python, we recommend using str as the default format because (with 
the surrogateescape handling on POSIX), it can correctly round-trip all 
characters used in paths. On Windows this is strongly recommended 
because the legacy OS support for bytes cannot round-trip all characters 
used in paths. Our support for bytes explicitly uses the *A functions 
and hence the encoding for the bytes is "whatever the active code page 
is". Since the active code page cannot represent all Unicode characters, 
the conversion of a path into bytes can lose information without warning 
(and we can't get a warning from the OS here - more on this later).

As a demonstration of this:

 >>> open('test\uAB00.txt', 'wb').close()
 >>> import glob
 >>> glob.glob('test*')
['test\uab00.txt']
 >>> glob.glob(b'test*')
[b'test?.txt']

The Unicode character in the second call to glob has been replaced by a 
'?', which means passing the path back into the filesystem will result 
in a FileNotFoundError (though ironically, passing it back into glob() 
will find the file again, since '?' is a single-character wildcard). You 
can observe the same results in os.listdir() or any function that 
matches the return type to the parameter type.

Why is this a problem?
======================

While the obvious and correct answer is to just use str everywhere, in 
general on POSIX systems there is no possibility of confusion when using 
bytes exclusively. Even if the encoding is "incorrect" by some standard, 
the file system can still map the bytes back to the file. Making use of 
this avoids the cost of decoding and reencoding, such that 
(theoretically, and only on POSIX), code like below is faster because of 
the use of `b'.'`:

 >>> for f in os.listdir(b'.'):
...     os.stat(f)
...

On Windows, if a filename exists that cannot be encoding with the active 
code page, you will receive an error from the above code. These errors 
are why in Python 3.3 the use of bytes paths on Windows was deprecated 
(listed in the What's New, but not clearly obvious in the documentation 
- more on this later). The above code produces multiple deprecation 
warnings in 3.3, 3.4 and 3.5 on Windows.

However, we still keep seeing libraries use bytes paths, which can cause 
unexpected issues on Windows (well, all platforms, but less and less 
common on POSIX as systems move to utf-8 - Windows long ago decided to 
move to utf-16 for the same reason, but Python's bytes interface did not 
keep up). Given the current approach of not-very-aggressively 
recommending that library developers either write their code twice (once 
for bytes and once for str) or use str exclusively are not working, we 
should consider alternative mitigations.

Proposals
=========

There are two dimensions here - the fix and the timing. We can basically 
choose any fix and any timing.

The main differences between the fixes are the balance between incorrect 
behaviour and backwards-incompatible behaviour. The main issue with 
respect to timing is whether or not we believe using bytes as paths on 
Windows was correctly deprecated in 3.3 and sufficiently advertised 
since to allow us to change the behaviour in 3.6.

Fixes
-----

Fix #1: Change sys.getfilesystemencoding() to utf-8 on Windows

Currently the default filesystem encoding is 'mbcs', which is a 
meta-encoder that uses the active code page. However, when bytes are 
passed to the filesystem they go through the *A APIs and the operating 
system handles encoding. In this case, paths are always encoded using 
the equivalent of 'mbcs:replace' - we have no ability to change this 
(though there is a user/machine configuration option to change the 
encoding from CP_ACP to CP_OEM, so it won't necessarily always match 
mbcs...)

This proposal would remove all use of the *A APIs and only ever call the 
*W APIs. When Windows returns paths to Python as str, they will be 
decoded from utf-16-le and returned as text. When paths are to be 
returned as bytes, we would decode from utf-16-le to utf-8 using 
surrogatepass (as Windows does not validate surrogate pairs, so it is 
possible to have invalid surrogates in filenames). Equally, when paths 
are provided as bytes, they are decoded from utf-8 into utf-16-le and 
passed to the *W APIs.

The use of utf-8 will not be configurable, with the possible exception 
of a "legacy mode" environment variable or Xflag.

surrogateescape does not apply here, as we are not concerned about 
keeping arbitrary bytes in the path. Any bytes path returned from the 
operating system will be valid; any bytes path created by the user may 
raise a decoding error (currently it would raise a file not found or 
similar OSError).

The choice of utf-8 (as opposed to returning utf-16-le bytes) is to 
ensure the ability to round-trip, while also allowing basic manipulation 
of paths - essentially just slicing and concatenating at '\' characters. 
Applications doing this have to ensure that their encoding matches 
sys.getfilesystemencoding(), or just use str everywhere.

It is debated, but I believe this is not a backwards compatibility issue 
because:
* byte paths in Python are specified as being encoded by 
sys.getfilesystemencoding()
* byte paths on Windows have been deprecated for three versions

Unfortunately, the deprecation is not explicitly called out anywhere in 
the docs apart from the What's New page, so there is an argument that it 
shouldn't be counted despite the warnings in the interpreter. However, 
this is more directly addressed in the discussion of timing below.

Equally, sys.getfilesystemencoding() documents the specific return 
values for various platforms, as well as that it is part of the protocol 
for using bytes to represent filesystem strings.

I believe both of these arguments are invalid, that the only code that 
will break as a result of this change is relying on deprecated 
functionality and incorrect encoding, and that the (probably noisy) 
breakage that will occur is less bad than the silent breakage that 
currently exists.

As far as implementation goes, there is already a patch for this at 
http://bugs.python.org/issue27781. In short, we update the path 
converter to decode bytes (path->narrow) to Unicode (path->wide) and 
remove all the code that would call *A APIs. In my patch I've changed 
path->narrow to a flag that indicates whether to convert back to bytes 
on return, and also to prevent compilation of code that tries to use 
->narrow as a string on Windows (maybe that will get too annoying for 
contributors? good discussion for the tracker IMHO).


Fix #2: Do the mbcs decoding ourselves

This is essentially the same as fix #1, but instead of changing to utf-8 
we keep mbcs as the encoding.

This approach will allow us to utilise new functionality that is only 
available as *W APIs, and also lets us be more strict about 
encoding/decoding to bytes. For example, rather than silently replacing 
Unicode characters with '?', we could warn or fail the operation, 
potentially modifying that behaviour with an environment variable or flag.

Compared to fix #1, this will enable some new functionality but will not 
fix any of the problems immediately. New runtime errors may cause some 
problems to be more obvious and lead to fixes, provided library 
maintainers are interested in supporting Windows and adding a separate 
code path to treat filesystem paths as strings.

This is a middle-ground proposal. On the positive side, it significantly 
reduces the code we have to maintain in CPython (e.g. posixmodule.c), as 
we won't require separate code paths to call the *A APIs. However, it 
doesn't really improve things for users apart from giving more 
exceptions, which are likely unexpected (people probably handle OSError 
but not UnicodeDecodeError when accessing the file system).


Fix #3: Make bytes paths on Windows an error

By preventing the use of bytes paths on Windows completely we prevent 
users from hitting encoding issues. However, we do this at the expense 
of usability. Obviously the deprecation concerns also play a big role in 
whether this is feasible.

I don't have numbers of libraries that will simply fail on Windows if 
this "fix" is made, but given I've already had people directly email me 
and tell me about their problems we can safely assume it's non-zero.

I'm really not a fan of this fix, because it doesn't actually make 
things better in a practical way, despite being more "pure".


Timing #1: Change it in 3.6

This timing assumes that we believe the deprecation of using bytes for 
paths in Python 3.3 was sufficiently well advertised that we can freely 
make changes in 3.6. A typical deprecation cycle would be two versions 
before removal (though we also often leave things in forever when they 
aren't fundamentally broken), so we have passed that point and 
theoretically can remove or change the functionality without breaking it.

In this case, we would announce in 3.6 that using bytes as paths on 
Windows is no longer deprecated, and that the encoding used is whatever 
is returned by sys.getfilesystemencoding().


Timing #2: Change it in 3.7

This timing assumes that the deprecation in 3.3 was valid, but 
acknowledges that it was not well publicised. For 3.6, we aggressively 
make it known that only strings should be used to represent paths on 
Windows and bytes are invalid and going to change in 3.7. (It has been 
suggested that I could use a keynote at PyCon to publicise this, and 
while I'd totally accept a keynote, I'd hate to subject a crowd to just 
this issue for an hour :) ).

My concern with this approach is that there is no benefit to the change 
at all. If we aggressively publicise the fact that libraries that don't 
handle Unicode paths on Windows properly are using deprecated 
functionality and need to be fixed by 3.7 in order to avoid breaking 
(more precisely - continuing to be broken, but with a different error 
message), then we will alienate non-Windows developers further from the 
platform (net loss for the ecosystem) and convince some to switch to str 
everywhere (net gain for the ecosystem). It doesn't

For those who listen and change to str, it removes the need to make any 
change in 3.7 at all, so we would really just be making noise about 
something that some people may not have noticed without necessarily 
going in and fixing anything. For those who don't listen, the change in 
3.7 is going to break them just as much as if we made the change in 3.6.


Timing #3: Change it in 3.8

This timing assumes that the deprecation in 3.3 was not sufficient and 
we need to start a new deprecation cycle. This is strengthened by the 
fact that the deprecation announcement does not explicitly include the 
io module or the builtin open() function, and so some developers may 
believe that using bytes for paths with these is okay despite the os 
module being deprecated.

The one upside to this approach is that it would also allow us to change 
locale.getpreferredencoding() to utf-8 on Windows (to affect the default 
behaviour of open(..., 'r') ), which I don't believe is going to be 
possible without a new deprecation cycle. There is a strong argument 
that the following code should also round-trip regardless of platform:

 >>> with open('list.txt', 'w') as f:
...     for i in os.listdir('.'):
...         print(i, file=f)
...
 >>> with open('list.txt', 'r') as f:
...     files = list(f)
...

Currently, the default encoding for open() cannot represent all 
filenames that may be returned from listdir(). This may affect makefiles 
and configuration files that contain paths. Currently they will work 
correctly for paths that can be represented in the machine's active code 
page (though it should be noted that the *A APIs may be changed in a 
process by user/machine configuration to use the OEM code page rather 
than the active code page, which would potentially lead to encoding 
issues even for CP_ACP compatible names).

Possibly resolving both issues simultaneously is worth waiting for two 
more releases? I'm not convinced the change to getfilesystemencoding() 
needs to wait for getpreferredencoding() to also change, or that they 
necessarily need to match, but it would not be hugely surprising to see 
the changes bundled together.

I'll also note that there has been limited discussion about changing 
getpreferredencoding() so far, though there have been a number of "+1" 
votes alongside some "+1 with significant concerns" votes. Changing the 
default encoding of the contents of data files is pretty scary, so I'm 
not in any rush to force it in. On the other hand, changing the encoding 
for paths without changing the default encoding for text files may break 
"bytes in, bytes through, bytes out" for some files (especially 
makefiles and .ini files). Arguably this idea was already deprecated 
with Python 3's bytes/text separation anyway.

Acknowledgements
================

Thanks to Stephen Turnbull, Eryk Sun, Victor Stinner and Random832 for 
their significant contributions and willingness to engage, and to 
everyone else on python-ideas for contributing to the discussion.


From dholth at gmail.com  Fri Aug 19 15:25:24 2016
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 19 Aug 2016 19:25:24 +0000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
Message-ID: <CAG8k2+6a5g23nGUM4ZwYEy+LOAks3d6fbhSEm+aQEsM7iHwPUw@mail.gmail.com>

#1 sounds like a great idea. I suppose surrogatepass solves approximately
the same problem of Rust's WTF-8, which is a way to round-trip bad UCS-2?
https://simonsapin.github.io/wtf-8/

#2 sounds like it would leave several problems, since mbcs is not the same
as a normal text encoding, IIUC it depends on the active code page. So if
your active code page is Russian you might not be able to encode Japanese
characters into MBCS.

Solution #2a Modify Windows so utf-8 is a valid value for the current MBCS
code page.

On Fri, Aug 19, 2016 at 3:01 PM Steve Dower <steve.dower at python.org> wrote:

> Hi python-dev
>
> About a week ago I proposed on python-ideas making some changes to how
> Python deals with encodings on Windows, specifically in relation to how
> Python interacts with the operating system.
>
> Changes to the console were uncontroversial, and I have posted patches
> at http://bugs.python.org/issue1602 and
> http://bugs.python.org/issue17620 to enable the full range of Unicode
> input to be used at interactive stdin/stdout.
>
> However, changes to sys.getfilesystemencoding(), which determines how
> the os module (and most filesystem functions in general) interpret bytes
> parameters, were more heatedly discussed. I've summarised the discussion
> in this email
>
> I'll declare up front that my preferred change is to treat bytes as
> utf-8 in Python 3.6, and I've posted a patch to do that at
> http://bugs.python.org/issue27781. Hopefully I haven't been too biased
> in my presentation of the alternatives, but this is so you at least know
> which way I'm biased.
>
> I'm looking for some agreement on the answers to the questions I pose in
> the summary.
>
> There is much more detail about them presented after that, as there are
> a number of non-obvious issues at play here. I suspect this will
> eventually become a PEP, but it's presented here as a summary of a
> discussion and not a PEP.
>
> Cheers,
> Steve
>
> Summary
> =======
>
> Representing file system paths on Windows as bytes may result in data
> loss due to the way Windows encodes/decodes strings via its bytes API.
>
> We can mitigate this by only using Window's Unicode API and doing our
> own encoding and decoding (i.e. within posixmodule.c's path converter).
> Invalid characters could cause encoding exceptions rather than data loss.
>
> We can go further to fix this by declaring the encoding of bytes paths
> on Windows must be utf-8, which would also prevent encoding exceptions,
> as utf-8 can fully represent all paths on Windows (natively utf-16-le).
>
> Even though using bytes for paths on Windows has been deprecated for
> three releases, this is not widely known and it may be too soon to
> change the behaviour.
>
> Questions:
> * should we always use Window's Unicode APIs instead of switching
> between bytes/Unicode based on parameter type?
> * should we allow users to pass bytes and interpret them as utf-8 rather
> than letting Windows do the decoding?
> * should we do it in 3.6, 3.7 or 3.8?
>
> Background
> ==========
>
> File system paths are almost universally represented as text in some
> encoding determined by the file system. In Python, we expose these paths
> via a number of interfaces, such as the os and io modules. Paths may be
> passed either direction across these interfaces, that is, from the
> filesystem to the application (for example, os.listdir()), or from the
> application to the filesystem (for example, os.unlink()).
>
> When paths are passed between the filesystem and the application, they
> are either passed through as a bytes blob or converted to/from str using
> sys.getfilesystemencoding(). The result of encoding a string with
> sys.getfilesystemencoding() is a blob of bytes in the native format for
> the default file system.
>
> On Windows, the native format for the filesystem is utf-16-le. The
> recommended platform APIs for accessing the filesystem all accept and
> return text encoded in this format. However, prior to Windows NT (and
> possibly further back), the native format was a configurable machine
> option and a separate set of APIs existed to accept this format. The
> option (the "active code page") and these APIs (the "*A functions")
> still exist in recent versions of Windows for backwards compatibility,
> though new functionality often only has a utf-16-le API (the "*W
> functions").
>
> In Python, we recommend using str as the default format because (with
> the surrogateescape handling on POSIX), it can correctly round-trip all
> characters used in paths. On Windows this is strongly recommended
> because the legacy OS support for bytes cannot round-trip all characters
> used in paths. Our support for bytes explicitly uses the *A functions
> and hence the encoding for the bytes is "whatever the active code page
> is". Since the active code page cannot represent all Unicode characters,
> the conversion of a path into bytes can lose information without warning
> (and we can't get a warning from the OS here - more on this later).
>
> As a demonstration of this:
>
>  >>> open('test\uAB00.txt', 'wb').close()
>  >>> import glob
>  >>> glob.glob('test*')
> ['test\uab00.txt']
>  >>> glob.glob(b'test*')
> [b'test?.txt']
>
> The Unicode character in the second call to glob has been replaced by a
> '?', which means passing the path back into the filesystem will result
> in a FileNotFoundError (though ironically, passing it back into glob()
> will find the file again, since '?' is a single-character wildcard). You
> can observe the same results in os.listdir() or any function that
> matches the return type to the parameter type.
>
> Why is this a problem?
> ======================
>
> While the obvious and correct answer is to just use str everywhere, in
> general on POSIX systems there is no possibility of confusion when using
> bytes exclusively. Even if the encoding is "incorrect" by some standard,
> the file system can still map the bytes back to the file. Making use of
> this avoids the cost of decoding and reencoding, such that
> (theoretically, and only on POSIX), code like below is faster because of
> the use of `b'.'`:
>
>  >>> for f in os.listdir(b'.'):
> ...     os.stat(f)
> ...
>
> On Windows, if a filename exists that cannot be encoding with the active
> code page, you will receive an error from the above code. These errors
> are why in Python 3.3 the use of bytes paths on Windows was deprecated
> (listed in the What's New, but not clearly obvious in the documentation
> - more on this later). The above code produces multiple deprecation
> warnings in 3.3, 3.4 and 3.5 on Windows.
>
> However, we still keep seeing libraries use bytes paths, which can cause
> unexpected issues on Windows (well, all platforms, but less and less
> common on POSIX as systems move to utf-8 - Windows long ago decided to
> move to utf-16 for the same reason, but Python's bytes interface did not
> keep up). Given the current approach of not-very-aggressively
> recommending that library developers either write their code twice (once
> for bytes and once for str) or use str exclusively are not working, we
> should consider alternative mitigations.
>
> Proposals
> =========
>
> There are two dimensions here - the fix and the timing. We can basically
> choose any fix and any timing.
>
> The main differences between the fixes are the balance between incorrect
> behaviour and backwards-incompatible behaviour. The main issue with
> respect to timing is whether or not we believe using bytes as paths on
> Windows was correctly deprecated in 3.3 and sufficiently advertised
> since to allow us to change the behaviour in 3.6.
>
> Fixes
> -----
>
> Fix #1: Change sys.getfilesystemencoding() to utf-8 on Windows
>
> Currently the default filesystem encoding is 'mbcs', which is a
> meta-encoder that uses the active code page. However, when bytes are
> passed to the filesystem they go through the *A APIs and the operating
> system handles encoding. In this case, paths are always encoded using
> the equivalent of 'mbcs:replace' - we have no ability to change this
> (though there is a user/machine configuration option to change the
> encoding from CP_ACP to CP_OEM, so it won't necessarily always match
> mbcs...)
>
> This proposal would remove all use of the *A APIs and only ever call the
> *W APIs. When Windows returns paths to Python as str, they will be
> decoded from utf-16-le and returned as text. When paths are to be
> returned as bytes, we would decode from utf-16-le to utf-8 using
> surrogatepass (as Windows does not validate surrogate pairs, so it is
> possible to have invalid surrogates in filenames). Equally, when paths
> are provided as bytes, they are decoded from utf-8 into utf-16-le and
> passed to the *W APIs.
>
> The use of utf-8 will not be configurable, with the possible exception
> of a "legacy mode" environment variable or Xflag.
>
> surrogateescape does not apply here, as we are not concerned about
> keeping arbitrary bytes in the path. Any bytes path returned from the
> operating system will be valid; any bytes path created by the user may
> raise a decoding error (currently it would raise a file not found or
> similar OSError).
>
> The choice of utf-8 (as opposed to returning utf-16-le bytes) is to
> ensure the ability to round-trip, while also allowing basic manipulation
> of paths - essentially just slicing and concatenating at '\' characters.
> Applications doing this have to ensure that their encoding matches
> sys.getfilesystemencoding(), or just use str everywhere.
>
> It is debated, but I believe this is not a backwards compatibility issue
> because:
> * byte paths in Python are specified as being encoded by
> sys.getfilesystemencoding()
> * byte paths on Windows have been deprecated for three versions
>
> Unfortunately, the deprecation is not explicitly called out anywhere in
> the docs apart from the What's New page, so there is an argument that it
> shouldn't be counted despite the warnings in the interpreter. However,
> this is more directly addressed in the discussion of timing below.
>
> Equally, sys.getfilesystemencoding() documents the specific return
> values for various platforms, as well as that it is part of the protocol
> for using bytes to represent filesystem strings.
>
> I believe both of these arguments are invalid, that the only code that
> will break as a result of this change is relying on deprecated
> functionality and incorrect encoding, and that the (probably noisy)
> breakage that will occur is less bad than the silent breakage that
> currently exists.
>
> As far as implementation goes, there is already a patch for this at
> http://bugs.python.org/issue27781. In short, we update the path
> converter to decode bytes (path->narrow) to Unicode (path->wide) and
> remove all the code that would call *A APIs. In my patch I've changed
> path->narrow to a flag that indicates whether to convert back to bytes
> on return, and also to prevent compilation of code that tries to use
> ->narrow as a string on Windows (maybe that will get too annoying for
> contributors? good discussion for the tracker IMHO).
>
>
> Fix #2: Do the mbcs decoding ourselves
>
> This is essentially the same as fix #1, but instead of changing to utf-8
> we keep mbcs as the encoding.
>
> This approach will allow us to utilise new functionality that is only
> available as *W APIs, and also lets us be more strict about
> encoding/decoding to bytes. For example, rather than silently replacing
> Unicode characters with '?', we could warn or fail the operation,
> potentially modifying that behaviour with an environment variable or flag.
>
> Compared to fix #1, this will enable some new functionality but will not
> fix any of the problems immediately. New runtime errors may cause some
> problems to be more obvious and lead to fixes, provided library
> maintainers are interested in supporting Windows and adding a separate
> code path to treat filesystem paths as strings.
>
> This is a middle-ground proposal. On the positive side, it significantly
> reduces the code we have to maintain in CPython (e.g. posixmodule.c), as
> we won't require separate code paths to call the *A APIs. However, it
> doesn't really improve things for users apart from giving more
> exceptions, which are likely unexpected (people probably handle OSError
> but not UnicodeDecodeError when accessing the file system).
>
>
> Fix #3: Make bytes paths on Windows an error
>
> By preventing the use of bytes paths on Windows completely we prevent
> users from hitting encoding issues. However, we do this at the expense
> of usability. Obviously the deprecation concerns also play a big role in
> whether this is feasible.
>
> I don't have numbers of libraries that will simply fail on Windows if
> this "fix" is made, but given I've already had people directly email me
> and tell me about their problems we can safely assume it's non-zero.
>
> I'm really not a fan of this fix, because it doesn't actually make
> things better in a practical way, despite being more "pure".
>
>
> Timing #1: Change it in 3.6
>
> This timing assumes that we believe the deprecation of using bytes for
> paths in Python 3.3 was sufficiently well advertised that we can freely
> make changes in 3.6. A typical deprecation cycle would be two versions
> before removal (though we also often leave things in forever when they
> aren't fundamentally broken), so we have passed that point and
> theoretically can remove or change the functionality without breaking it.
>
> In this case, we would announce in 3.6 that using bytes as paths on
> Windows is no longer deprecated, and that the encoding used is whatever
> is returned by sys.getfilesystemencoding().
>
>
> Timing #2: Change it in 3.7
>
> This timing assumes that the deprecation in 3.3 was valid, but
> acknowledges that it was not well publicised. For 3.6, we aggressively
> make it known that only strings should be used to represent paths on
> Windows and bytes are invalid and going to change in 3.7. (It has been
> suggested that I could use a keynote at PyCon to publicise this, and
> while I'd totally accept a keynote, I'd hate to subject a crowd to just
> this issue for an hour :) ).
>
> My concern with this approach is that there is no benefit to the change
> at all. If we aggressively publicise the fact that libraries that don't
> handle Unicode paths on Windows properly are using deprecated
> functionality and need to be fixed by 3.7 in order to avoid breaking
> (more precisely - continuing to be broken, but with a different error
> message), then we will alienate non-Windows developers further from the
> platform (net loss for the ecosystem) and convince some to switch to str
> everywhere (net gain for the ecosystem). It doesn't
>
> For those who listen and change to str, it removes the need to make any
> change in 3.7 at all, so we would really just be making noise about
> something that some people may not have noticed without necessarily
> going in and fixing anything. For those who don't listen, the change in
> 3.7 is going to break them just as much as if we made the change in 3.6.
>
>
> Timing #3: Change it in 3.8
>
> This timing assumes that the deprecation in 3.3 was not sufficient and
> we need to start a new deprecation cycle. This is strengthened by the
> fact that the deprecation announcement does not explicitly include the
> io module or the builtin open() function, and so some developers may
> believe that using bytes for paths with these is okay despite the os
> module being deprecated.
>
> The one upside to this approach is that it would also allow us to change
> locale.getpreferredencoding() to utf-8 on Windows (to affect the default
> behaviour of open(..., 'r') ), which I don't believe is going to be
> possible without a new deprecation cycle. There is a strong argument
> that the following code should also round-trip regardless of platform:
>
>  >>> with open('list.txt', 'w') as f:
> ...     for i in os.listdir('.'):
> ...         print(i, file=f)
> ...
>  >>> with open('list.txt', 'r') as f:
> ...     files = list(f)
> ...
>
> Currently, the default encoding for open() cannot represent all
> filenames that may be returned from listdir(). This may affect makefiles
> and configuration files that contain paths. Currently they will work
> correctly for paths that can be represented in the machine's active code
> page (though it should be noted that the *A APIs may be changed in a
> process by user/machine configuration to use the OEM code page rather
> than the active code page, which would potentially lead to encoding
> issues even for CP_ACP compatible names).
>
> Possibly resolving both issues simultaneously is worth waiting for two
> more releases? I'm not convinced the change to getfilesystemencoding()
> needs to wait for getpreferredencoding() to also change, or that they
> necessarily need to match, but it would not be hugely surprising to see
> the changes bundled together.
>
> I'll also note that there has been limited discussion about changing
> getpreferredencoding() so far, though there have been a number of "+1"
> votes alongside some "+1 with significant concerns" votes. Changing the
> default encoding of the contents of data files is pretty scary, so I'm
> not in any rush to force it in. On the other hand, changing the encoding
> for paths without changing the default encoding for text files may break
> "bytes in, bytes through, bytes out" for some files (especially
> makefiles and .ini files). Arguably this idea was already deprecated
> with Python 3's bytes/text separation anyway.
>
> Acknowledgements
> ================
>
> Thanks to Stephen Turnbull, Eryk Sun, Victor Stinner and Random832 for
> their significant contributions and willingness to engage, and to
> everyone else on python-ideas for contributing to the discussion.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160819/2838883e/attachment-0001.html>

From brett at python.org  Fri Aug 19 15:33:14 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 19 Aug 2016 19:33:14 +0000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAPTjJmp9fDkTXqpj6ZuE-aW-QLQX-Wjhvxkvp4u=w1wtgphGfA@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
 <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
 <4a611645-db9e-5da6-ede5-15e87427447a@python.org>
 <CAPTjJmp9fDkTXqpj6ZuE-aW-QLQX-Wjhvxkvp4u=w1wtgphGfA@mail.gmail.com>
Message-ID: <CAP1=2W5ivydxkT4Zj647wZpB5E1pHW25BnFmSRHBrBFUW86Zqg@mail.gmail.com>

On Fri, 19 Aug 2016 at 09:47 Chris Angelico <rosuav at gmail.com> wrote:

> On Sat, Aug 20, 2016 at 2:25 AM, Steve Dower <steve.dower at python.org>
> wrote:
> > On 19Aug2016 0910, Chris Angelico wrote:
> >>
> >> On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower <steve.dower at python.org>
> >> wrote:
> >>>
> >>> Check any .pth files you can find. I suspect mpl_toolkits has some
> magic
> >>> in
> >>> it to make the namespace package work on 2.7.
> >>
> >>
> >> $ cat
> >> /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
> >> import sys, types, os;p =
> >> os.path.join(sys._getframe(1).f_locals['sitedir'],
> >> *('mpl_toolkits',));ie =
> >> os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
> >> sys.modules.setdefault('mpl_toolkits',
> >> types.ModuleType('mpl_toolkits'));mp = (m or []) and
> >> m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
> >>
> >> Is it possible that that's the cause? And if so, should "python -I"
> >> remove /usr/local/lib/python3.6/site-packages from sys.path?
> >
> >
> > That explains the extra modules - types.py imports collections and
> > collections.abc, which would seem to account for the rest.
> >
> > I'm not sure whether that's causing the problem or not, but temporarily
> > renaming that pth file so it isn't picked up may help you track down the
> > test issue.
>
> Sure enough, that did fix the problem - that is to say, renaming it
> did allow test_site to pass. Thanks!
>
> > "python -S" is the only way to avoid .pth files, but that's a pretty
> heavy
> > hammer.
>
> Hmmmmmmm. So the question is, what is this test testing?


It's making sure people who work on the modules that are imported during
startup don't accidentally add another module dependency to the startup
sequence. Since module imports impact startup time we have historically
worked hard to minimize import dependencies for startup (hence things like
certain functions in the os module doing imports within a function even
though we discourage that practice in general).


> For
> reference, here's the code:
>
> class StartupImportTests(unittest.TestCase):
>
>     def test_startup_imports(self):
>         # This tests checks which modules are loaded by Python when it
>         # initially starts upon startup.
>         popen = subprocess.Popen([sys.executable, '-I', '-v', '-c',
>                                   'import sys; print(set(sys.modules))'],
>                                  stdout=subprocess.PIPE,
>                                  stderr=subprocess.PIPE)
>         stdout, stderr = popen.communicate()
>         stdout = stdout.decode('utf-8')
>         stderr = stderr.decode('utf-8')
>         modules = eval(stdout)
>
>         self.assertIn('site', modules)
>
>         # http://bugs.python.org/issue19205
>         re_mods = {'re', '_sre', 'sre_compile', 'sre_constants',
> 'sre_parse'}
>         # _osx_support uses the re module in many placs
>         if sys.platform != 'darwin':
>             self.assertFalse(modules.intersection(re_mods), stderr)
>         # http://bugs.python.org/issue9548
>         self.assertNotIn('locale', modules, stderr)
>         if sys.platform != 'darwin':
>             # http://bugs.python.org/issue19209
>             self.assertNotIn('copyreg', modules, stderr)
>         # http://bugs.python.org/issue19218>
>         collection_mods = {'_collections', 'collections', 'functools',
>                            'heapq', 'itertools', 'keyword', 'operator',
>                            'reprlib', 'types', 'weakref'
>                           }.difference(sys.builtin_module_names)
>         self.assertFalse(modules.intersection(collection_mods), stderr)
>
>
> It asserts that 'site' is in the list of loaded modules, so simply
> adding '-S' to the subpython parameters doesn't work. But this test,
> as written, depends on the behaviour of arbitrary third-party modules
> installed with pip, which seems wrong for a standard library test.
> AIUI this is trying to catch changes that accidentally cause large
> modules to be loaded on startup, correct? Which is all well and good,
> but Python shouldn't have test failures due to matplotlib being
> installed.
>
> I see two (or three) ways to deal with this.
>
> 1) Declare it a bug in matplotlib, and demand that .pth files not
> import anything that wouldn't otherwise be imported.
>
> 2) Change the test to add '-S' and remove the assertion about 'site'
> being present; this means that site.py itself is no longer covered by
> the test.
>
> 3) Accept the status quo: this test can fail if .pth files are present.
>
> Incidentally, where's it documented that .pth files can be executed?

It's not in the site.py docs [1] or docstring, only in the addpackage
> function within that (which I found by browsing the source).


Actually it is documented in the site documentation: "Lines starting with
import (followed by space or tab) are executed". It's subtle, but
"executed" literally means execution in this context; the line is passed
into exec():
https://github.com/python/cpython/blob/b505c47044fb5b8f03b9708ee1f210a4c935baf5/Lib/site.py#L167
.
It's an abuse that e.g. setuptools relies on to make eggs work.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160819/383a898d/attachment.html>

From steve.dower at python.org  Fri Aug 19 15:33:58 2016
From: steve.dower at python.org (Steve Dower)
Date: Fri, 19 Aug 2016 12:33:58 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAG8k2+6a5g23nGUM4ZwYEy+LOAks3d6fbhSEm+aQEsM7iHwPUw@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAG8k2+6a5g23nGUM4ZwYEy+LOAks3d6fbhSEm+aQEsM7iHwPUw@mail.gmail.com>
Message-ID: <ad708e76-41ed-3e0f-e4b9-02897727e714@python.org>

On 19Aug2016 1225, Daniel Holth wrote:
> #1 sounds like a great idea. I suppose surrogatepass solves
> approximately the same problem of Rust's WTF-8, which is a way to
> round-trip bad UCS-2? https://simonsapin.github.io/wtf-8/

Yep.

> #2 sounds like it would leave several problems, since mbcs is not the
> same as a normal text encoding, IIUC it depends on the active code page.
> So if your active code page is Russian you might not be able to encode
> Japanese characters into MBCS.

That's correct. In 99% (or more) of cases, mbcs is going to be the same 
as what we currently have. The difference is that when we encode/decode 
in CPython we can use a different handler than 'replace' and at least 
prevent the _silent_ data loss.

> Solution #2a Modify Windows so utf-8 is a valid value for the current
> MBCS code page.

Presumably a joke, but won't happen because too many applications assume 
that the active code page is one byte per character, which it isn't, but 
it's close enough that most of the time you never notice. (Incidentally, 
this is also the problem with utf-16, since many applications also 
assume that it's always one wchar_t per character and get away with it. 
At least with utf-8 you encounter multi-byte sequences often enough that 
you basically are forced to deal with them.)

Cheers,
Steve

From rosuav at gmail.com  Fri Aug 19 17:10:45 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Aug 2016 07:10:45 +1000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAP1=2W5ivydxkT4Zj647wZpB5E1pHW25BnFmSRHBrBFUW86Zqg@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
 <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
 <4a611645-db9e-5da6-ede5-15e87427447a@python.org>
 <CAPTjJmp9fDkTXqpj6ZuE-aW-QLQX-Wjhvxkvp4u=w1wtgphGfA@mail.gmail.com>
 <CAP1=2W5ivydxkT4Zj647wZpB5E1pHW25BnFmSRHBrBFUW86Zqg@mail.gmail.com>
Message-ID: <CAPTjJmo64s_reKYoF7mYiygXKA130KH=BrE6sFbKx=oQ3kZuRg@mail.gmail.com>

On Sat, Aug 20, 2016 at 5:33 AM, Brett Cannon <brett at python.org> wrote:
>> Hmmmmmmm. So the question is, what is this test testing?
>
>
> It's making sure people who work on the modules that are imported during
> startup don't accidentally add another module dependency to the startup
> sequence. Since module imports impact startup time we have historically
> worked hard to minimize import dependencies for startup (hence things like
> certain functions in the os module doing imports within a function even
> though we discourage that practice in general).

Fair enough. Which means it probably *should* import site, because
that's going to happen on all normal usage. So that still leaves a few
options:

1) Demand that .pth files restrict themselves to what's already
imported. This means startup is still fast even if you have a bunch of
pths. Downside: Third-party code can break Python's rules. Upside:
When they do, it can be reported as a bug, the fixing of which will
improve startup times when that's installed.

2) Change the test to somehow disable .pth execution while keeping the
rest of site.py intact. This gives more consistent test results, but
still is mostly applicable to normal usage.

3) Ignore the problem and assume that the Python test suite will be
run with no site-packages installed. Status quo, basically. Possibly
with some tweaks so running 'make test' ignores site-packages?

Is #2 doable? If not, can 'make test' somehow hack out site-packages?

ChrisA

From brett at python.org  Fri Aug 19 17:34:17 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 19 Aug 2016 21:34:17 +0000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAPTjJmo64s_reKYoF7mYiygXKA130KH=BrE6sFbKx=oQ3kZuRg@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
 <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
 <4a611645-db9e-5da6-ede5-15e87427447a@python.org>
 <CAPTjJmp9fDkTXqpj6ZuE-aW-QLQX-Wjhvxkvp4u=w1wtgphGfA@mail.gmail.com>
 <CAP1=2W5ivydxkT4Zj647wZpB5E1pHW25BnFmSRHBrBFUW86Zqg@mail.gmail.com>
 <CAPTjJmo64s_reKYoF7mYiygXKA130KH=BrE6sFbKx=oQ3kZuRg@mail.gmail.com>
Message-ID: <CAP1=2W5wQ+4fw+ijNAmLV53FM=5bAFnwZJ4Ut4d3LFm9QNc1eg@mail.gmail.com>

On Fri, 19 Aug 2016 at 14:11 Chris Angelico <rosuav at gmail.com> wrote:

> On Sat, Aug 20, 2016 at 5:33 AM, Brett Cannon <brett at python.org> wrote:
> >> Hmmmmmmm. So the question is, what is this test testing?
> >
> >
> > It's making sure people who work on the modules that are imported during
> > startup don't accidentally add another module dependency to the startup
> > sequence. Since module imports impact startup time we have historically
> > worked hard to minimize import dependencies for startup (hence things
> like
> > certain functions in the os module doing imports within a function even
> > though we discourage that practice in general).
>
> Fair enough. Which means it probably *should* import site, because
> that's going to happen on all normal usage. So that still leaves a few
> options:
>
> 1) Demand that .pth files restrict themselves to what's already
> imported. This means startup is still fast even if you have a bunch of
> pths. Downside: Third-party code can break Python's rules. Upside:
> When they do, it can be reported as a bug, the fixing of which will
> improve startup times when that's installed.
>
> 2) Change the test to somehow disable .pth execution while keeping the
> rest of site.py intact. This gives more consistent test results, but
> still is mostly applicable to normal usage.
>
> 3) Ignore the problem and assume that the Python test suite will be
> run with no site-packages installed. Status quo, basically. Possibly
> with some tweaks so running 'make test' ignores site-packages?
>
> Is #2 doable?


Anything is doable given enough time and effort. ;) As of right now the
only way is with -S.


> If not, can 'make test' somehow hack out site-packages?
>

The problem is you're dealing with startup which means it won't come into
effect until it's too late.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160819/29fe70d2/attachment.html>

From steve.dower at python.org  Fri Aug 19 17:51:08 2016
From: steve.dower at python.org (Steve Dower)
Date: Fri, 19 Aug 2016 14:51:08 -0700
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAPTjJmo64s_reKYoF7mYiygXKA130KH=BrE6sFbKx=oQ3kZuRg@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
 <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
 <4a611645-db9e-5da6-ede5-15e87427447a@python.org>
 <CAPTjJmp9fDkTXqpj6ZuE-aW-QLQX-Wjhvxkvp4u=w1wtgphGfA@mail.gmail.com>
 <CAP1=2W5ivydxkT4Zj647wZpB5E1pHW25BnFmSRHBrBFUW86Zqg@mail.gmail.com>
 <CAPTjJmo64s_reKYoF7mYiygXKA130KH=BrE6sFbKx=oQ3kZuRg@mail.gmail.com>
Message-ID: <ce3b2365-8904-3e08-8fd1-b05f0cdbfb29@python.org>

On 19Aug2016 1410, Chris Angelico wrote:
> 2) Change the test to somehow disable .pth execution while keeping the
> rest of site.py intact. This gives more consistent test results, but
> still is mostly applicable to normal usage.

If you start the process with "-S", then run:

     import site
     site.addpackage = lambda *a: None
     site.main()
     # rest of the test

Then that should do it. This will cut out all .pth processing, but 
shouldn't affect any of the other steps.

Cheers,
Steve



From rosuav at gmail.com  Fri Aug 19 17:55:35 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Aug 2016 07:55:35 +1000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAP1=2W5wQ+4fw+ijNAmLV53FM=5bAFnwZJ4Ut4d3LFm9QNc1eg@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
 <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
 <4a611645-db9e-5da6-ede5-15e87427447a@python.org>
 <CAPTjJmp9fDkTXqpj6ZuE-aW-QLQX-Wjhvxkvp4u=w1wtgphGfA@mail.gmail.com>
 <CAP1=2W5ivydxkT4Zj647wZpB5E1pHW25BnFmSRHBrBFUW86Zqg@mail.gmail.com>
 <CAPTjJmo64s_reKYoF7mYiygXKA130KH=BrE6sFbKx=oQ3kZuRg@mail.gmail.com>
 <CAP1=2W5wQ+4fw+ijNAmLV53FM=5bAFnwZJ4Ut4d3LFm9QNc1eg@mail.gmail.com>
Message-ID: <CAPTjJmr4W_A=LiQgGy5KndYJ3kUKsPFTA_9ww73dU09HNmgjcg@mail.gmail.com>

On Sat, Aug 20, 2016 at 7:34 AM, Brett Cannon <brett at python.org> wrote:
>> 1) Demand that .pth files restrict themselves to what's already
>> imported. This means startup is still fast even if you have a bunch of
>> pths. Downside: Third-party code can break Python's rules. Upside:
>> When they do, it can be reported as a bug, the fixing of which will
>> improve startup times when that's installed.
>>
>> 2) Change the test to somehow disable .pth execution while keeping the
>> rest of site.py intact. This gives more consistent test results, but
>> still is mostly applicable to normal usage.
>>
>> 3) Ignore the problem and assume that the Python test suite will be
>> run with no site-packages installed. Status quo, basically. Possibly
>> with some tweaks so running 'make test' ignores site-packages?
>>
>> Is #2 doable?
>
>
> Anything is doable given enough time and effort. ;) As of right now the only
> way is with -S.

Heh. I guess what I really mean is: Is #2 a good idea, or would it be
considered horrendously hacky? This is my itch, so I'm not afraid to
be the one to code a solution, but I don't want to make things overall
worse to fix one specific use case.

>> If not, can 'make test' somehow hack out site-packages?
>
>
> The problem is you're dealing with startup which means it won't come into
> effect until it's too late.

Yeah, but the 'make test' operation could set an environment variable
or something to say "there are no site-packages, kthxbye". Again,
there's currently no such facility, and probably that one is a bad
idea, as it'd have implications (possibly security implications) for
other runnings of Python.

Steve's suggestion would keep all the visible hackery in test_site.py,
and it's well known that tests often need to do dark and dangerous
things.

On Sat, Aug 20, 2016 at 7:51 AM, Steve Dower <steve.dower at python.org> wrote:
>
> If you start the process with "-S", then run:
>
>     import site
>     site.addpackage = lambda *a: None
>     site.main()
>     # rest of the test
>
> Then that should do it. This will cut out all .pth processing, but shouldn't
> affect any of the other steps.

If I make a patch to do that, would it have a good chance of being accepted?

ChrisA

From sjoerdjob at sjec.nl  Fri Aug 19 18:07:17 2016
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Sat, 20 Aug 2016 00:07:17 +0200
Subject: [Python-Dev] Failures in test_site.py - how to debug?
Message-ID: <3sGHBc3t2SzFrBj@mail.python.org>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160820/055a8af8/attachment-0001.html>

From rosuav at gmail.com  Fri Aug 19 18:07:24 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Aug 2016 08:07:24 +1000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAPTjJmr4W_A=LiQgGy5KndYJ3kUKsPFTA_9ww73dU09HNmgjcg@mail.gmail.com>
References: <CAPTjJmoM+4qTD=3qFpmx2ZQTBb3gk9RucCCCi9Kq1+KOExuD1A@mail.gmail.com>
 <E1balhu-0002B4-Gl@se2-syd.hostedmail.net.au>
 <CAPTjJmoFzfPNyC9kUgeijkGjUkcJFMjX4Ys5Q=v=NwTRs6T6+w@mail.gmail.com>
 <4a611645-db9e-5da6-ede5-15e87427447a@python.org>
 <CAPTjJmp9fDkTXqpj6ZuE-aW-QLQX-Wjhvxkvp4u=w1wtgphGfA@mail.gmail.com>
 <CAP1=2W5ivydxkT4Zj647wZpB5E1pHW25BnFmSRHBrBFUW86Zqg@mail.gmail.com>
 <CAPTjJmo64s_reKYoF7mYiygXKA130KH=BrE6sFbKx=oQ3kZuRg@mail.gmail.com>
 <CAP1=2W5wQ+4fw+ijNAmLV53FM=5bAFnwZJ4Ut4d3LFm9QNc1eg@mail.gmail.com>
 <CAPTjJmr4W_A=LiQgGy5KndYJ3kUKsPFTA_9ww73dU09HNmgjcg@mail.gmail.com>
Message-ID: <CAPTjJmrzsbk=Xgrnbisrhx3X7WQs2iOPts2k=MwaAN-oZVd9_g@mail.gmail.com>

On Sat, Aug 20, 2016 at 7:55 AM, Chris Angelico <rosuav at gmail.com> wrote:
> If I make a patch to do that, would it have a good chance of being accepted?
>

Shortcut. I've made the patch and put it on the tracker.

http://bugs.python.org/issue27807

Either it's accepted or it's not :)

Thank you all for the debugging help.

ChrisA

From rosuav at gmail.com  Fri Aug 19 18:10:36 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Aug 2016 08:10:36 +1000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <57b78318.d160c20a.3ab16.19e5SMTPIN_ADDED_MISSING@mx.google.com>
References: <57b78318.d160c20a.3ab16.19e5SMTPIN_ADDED_MISSING@mx.google.com>
Message-ID: <CAPTjJmrmzt_TCqJz_e_eshct-CB1xN2+TxJmyh1__CeTL=Jv-w@mail.gmail.com>

On Sat, Aug 20, 2016 at 8:07 AM, Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
> I'd like to re-iterate my suggestion in case it was missed: split the
> current test in 2 tests:
>
> * Running with `-S` which is for checking that by default the collections
> are not imported. (Which is what is currently tested)
> * Running without `-S` which is for checking that `site` gets imported.
>
> I hope that is clear enough. Do you see any problems with such an approach?

AIUI this wouldn't test that site is sufficiently slim. The test is
supposed to mimic normal startup, then assert that unnecessary modules
haven't been loaded. Splitting it would prove that a -S run is nice
and slim, but would allow bloat in site.py to pass unnoticed.

ChrisA

From random832 at fastmail.com  Fri Aug 19 22:55:32 2016
From: random832 at fastmail.com (Random832)
Date: Fri, 19 Aug 2016 22:55:32 -0400
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <CAPTjJmrmzt_TCqJz_e_eshct-CB1xN2+TxJmyh1__CeTL=Jv-w@mail.gmail.com>
References: <57b78318.d160c20a.3ab16.19e5SMTPIN_ADDED_MISSING@mx.google.com>
 <CAPTjJmrmzt_TCqJz_e_eshct-CB1xN2+TxJmyh1__CeTL=Jv-w@mail.gmail.com>
Message-ID: <1471661732.2772361.700731969.388E4413@webmail.messagingengine.com>

On Fri, Aug 19, 2016, at 18:10, Chris Angelico wrote:
> On Sat, Aug 20, 2016 at 8:07 AM, Sjoerd Job Postmus <sjoerdjob at sjec.nl>
> wrote:
> > I'd like to re-iterate my suggestion in case it was missed: split the
> > current test in 2 tests:
> >
> > * Running with `-S` which is for checking that by default the collections
> > are not imported. (Which is what is currently tested)
> > * Running without `-S` which is for checking that `site` gets imported.
> >
> > I hope that is clear enough. Do you see any problems with such an approach?
> 
> AIUI this wouldn't test that site is sufficiently slim. The test is
> supposed to mimic normal startup, then assert that unnecessary modules
> haven't been loaded. Splitting it would prove that a -S run is nice
> and slim, but would allow bloat in site.py to pass unnoticed.

What about -S and putting "import site" explicitly in the test code? Or
would that go back to importing everything on people who have packages
installed?

Really, there should be a "source-isolated" mode, which removes
/usr/lib/whatever from the path in the same way that -I removes the
user's stuff, but still runs cpython/Lib/site.py , and all tests (and
for that matter stuff like frozen module prep) should be run in this
mode.

I'm not sure I see a downside to having such a mode be the *default*
mode for "python detects it is being run from a source tree".

From rosuav at gmail.com  Fri Aug 19 23:00:10 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 20 Aug 2016 13:00:10 +1000
Subject: [Python-Dev] Failures in test_site.py - how to debug?
In-Reply-To: <1471661732.2772361.700731969.388E4413@webmail.messagingengine.com>
References: <57b78318.d160c20a.3ab16.19e5SMTPIN_ADDED_MISSING@mx.google.com>
 <CAPTjJmrmzt_TCqJz_e_eshct-CB1xN2+TxJmyh1__CeTL=Jv-w@mail.gmail.com>
 <1471661732.2772361.700731969.388E4413@webmail.messagingengine.com>
Message-ID: <CAPTjJmqOYrsTCawRDuDU3YCMDEmW19L8OP0f6tEDYaiFvKanVw@mail.gmail.com>

On Sat, Aug 20, 2016 at 12:55 PM, Random832 <random832 at fastmail.com> wrote:
>
> What about -S and putting "import site" explicitly in the test code? Or
> would that go back to importing everything on people who have packages
> installed?

I think so, yes.

> Really, there should be a "source-isolated" mode, which removes
> /usr/lib/whatever from the path in the same way that -I removes the
> user's stuff, but still runs cpython/Lib/site.py , and all tests (and
> for that matter stuff like frozen module prep) should be run in this
> mode.
>
> I'm not sure I see a downside to having such a mode be the *default*
> mode for "python detects it is being run from a source tree".

That sounds great, but I don't have the skills to implement that and
ensure that it's reliable.

ChrisA

From ncoghlan at gmail.com  Sat Aug 20 15:31:27 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Aug 2016 05:31:27 +1000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
Message-ID: <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>

On 20 August 2016 at 04:59, Steve Dower <steve.dower at python.org> wrote:
> Questions:
> * should we always use Window's Unicode APIs instead of switching between
> bytes/Unicode based on parameter type?

Yes

> * should we allow users to pass bytes and interpret them as utf-8 rather
> than letting Windows do the decoding?

Yes (eventually)

> * should we do it in 3.6, 3.7 or 3.8?

Reading your summary meant this finally clicked with something Victor
has been considering for a while: a "Force UTF-8" switch that told
Python to ignore the locale encoding on Linux, and instead assume
UTF-8 everywhere (command line parameter parsing, environment variable
processing, filesystem encoding, standard streams, etc)

It's essentially the same problem you have on Windows, just with
slightly different symptoms and consequences.

Prompted by that realisation, I'd like to suggest an option that
didn't come up on python-ideas: add such a flag to Python 3.6, and
then actively seek feedback from folks using non-UTF-8 encodings
before making a decision on what to do by default in Python 3.7.

This is a really hard problem for people to reason about abstractly,
but "try running Python with this new flag, and see if anything
breaks" is a much easier question to ask and answer.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve.dower at python.org  Sat Aug 20 16:31:59 2016
From: steve.dower at python.org (Steve Dower)
Date: Sat, 20 Aug 2016 13:31:59 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
Message-ID: <57B8BE3F.9090405@python.org>

On 20Aug2016 1231, Nick Coghlan wrote:
> I'd like to suggest an option that
> didn't come up on python-ideas: add such a flag to Python 3.6, and
> then actively seek feedback from folks using non-UTF-8 encodings
> before making a decision on what to do by default in Python 3.7.

My biggest concern is that it then falls onto users to know how to start 
Python with that flag. About the only way I'd be okay with this is going 
with option #2 (using mbcs:strict, rather than mbcs:replace) and putting 
a note in the exception to "restart Python with the -X:utf8 option, or 
contact whoever gave you the application if you don't know what this 
means", and hope that the user gets to see the exception rather than it 
being swallowed by the application (and then that the user knows how to 
start it via python.exe, as opposed to a script entry point, a shortcut 
or via a browser or other launcher).

Basically, we make breaking noisier and tell the user to fix it. Some 
users will think that means it is their fault.

On the other hand, having code opt-in or out of the new handling 
requires changing code (which is presumably not going to happen, or we 
wouldn't consider keeping the old behaviour and/or letting the user 
control it), and leads to conflicts between libraries that expect 
different behaviour. If we expect people to use the flag (and I do), we 
need to be really explicit that developers should handle two cases. 
Without the flag, we only need to be explicit about how developers 
should handle the one case.

> This is a really hard problem for people to reason about abstractly,
> but "try running Python with this new flag, and see if anything
> breaks" is a much easier question to ask and answer.

Personally I'm a big fan of doing this with beta releases. There's time 
to switch back to an opt-in model before the final release if things go 
badly. There's already been a lot of interest in the 3.6 prereleases 
(the 3.6.0a3 installer had over 80k downloads in its first fortnight), 
so I think this is viable.

To help, I could add an extra info page to the prerelease installer 
announcing some specific new changes and explicitly asking for feedback 
to bugs.p.o - the ask is already there, but we could beg more in 
prereleases.

Cheers,
Steve

From ncoghlan at gmail.com  Sun Aug 21 00:58:46 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Aug 2016 14:58:46 +1000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <57B8BE3F.9090405@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
Message-ID: <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>

On 21 August 2016 at 06:31, Steve Dower <steve.dower at python.org> wrote:
> My biggest concern is that it then falls onto users to know how to start
> Python with that flag.

Not necessarily, as this is one of the areas where commercial
redistributors can earn their revenue stream - by deciding that
flipping the default behaviour is the right thing to do for *their*
user base (which is inevitably only a subset of the overall Python
user base).

Making that possible doesn't mean redistributors will actually follow
through, but it's an option worth keeping in mind, as while it does
increase the ecosystem complexity in the near term (since default
behaviour may vary based on how you obtained your Python runtime), in
the longer term it can allow for better informed design decisions at
the reference interpreter level. (For business process wonks, it's
essentially like running through a deliberate divergence/convergence
cycle at the level of the entire language ecosystem:
http://theagilepirate.net/archives/1392 )

> About the only way I'd be okay with this is going
> with option #2 (using mbcs:strict, rather than mbcs:replace) and putting a
> note in the exception to "restart Python with the -X:utf8 option, or contact
> whoever gave you the application if you don't know what this means", and
> hope that the user gets to see the exception rather than it being swallowed
> by the application (and then that the user knows how to start it via
> python.exe, as opposed to a script entry point, a shortcut or via a browser
> or other launcher).
>
> Basically, we make breaking noisier and tell the user to fix it. Some users
> will think that means it is their fault.

I think that makes a good fallback plan if we run into problems during
the beta cycle.

> On the other hand, having code opt-in or out of the new handling requires
> changing code (which is presumably not going to happen, or we wouldn't
> consider keeping the old behaviour and/or letting the user control it), and
> leads to conflicts between libraries that expect different behaviour. If we
> expect people to use the flag (and I do), we need to be really explicit that
> developers should handle two cases. Without the flag, we only need to be
> explicit about how developers should handle the one case.

Library and framework developers will need to handle both cases
anyway, since it's going to be a while before anyone that isn't
bundling their language runtime as part of a fully integrated
application can assume Python 3.6 as their minimum Python version.

>> This is a really hard problem for people to reason about abstractly,
>> but "try running Python with this new flag, and see if anything
>> breaks" is a much easier question to ask and answer.
>
> Personally I'm a big fan of doing this with beta releases. There's time to
> switch back to an opt-in model before the final release if things go badly.
> There's already been a lot of interest in the 3.6 prereleases (the 3.6.0a3
> installer had over 80k downloads in its first fortnight), so I think this is
> viable.

Aye, that sounds like a plausible plan to me.

> To help, I could add an extra info page to the prerelease installer
> announcing some specific new changes and explicitly asking for feedback to
> bugs.p.o - the ask is already there, but we could beg more in prereleases.

I think you'll want to escalate this to a PEP as well - those aren't
just useful as within-development-team design discussion documents,
they also help to give changes more visibility to end users during the
beta process, and to provide an easily referenced "Why did this
change?" explanation for a particular design decision post-release.

That way, "Make it the default during the beta period, potentially
revert back to a '-X:utf-8' option in the final beta if we find
unresolvable-in-the-near-term problems with changing the default" can
be documented as part of the PEP, and folks with concerns about the
change will have something to point people to when trying to convince
them that they really need to test the 3.6 beta releases on their
systems in time to influence that decision.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From christian at python.org  Sun Aug 21 08:37:51 2016
From: christian at python.org (Christian Heimes)
Date: Sun, 21 Aug 2016 14:37:51 +0200
Subject: [Python-Dev] socket.setsockopt() with optval=NULL
Message-ID: <c9bbe651-af70-175f-d386-e81e791b2f44@python.org>

Hi,

the socket.setsockopt(level, optname, value) method has two calling
variants. When it is called with a buffer-like object as value, it calls
the C API function setsockopt() with optval=buffer.buf and
optlen=buffer.len. When value is an integer, setsockopt() packs it as
int32 and sends it with optlen=4.

---
# example.py
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,
    b'\x00\x00\x00\x00')
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
---

$ strace -e setsockopt ./python example.py
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0


For AF_ALG (Linux Kernel crypto) I need a way to call the C API function
setsockopt() with optval=NULL and optlen as any arbitrary number. I have
been playing with multiple ideas. So far I liked the idea of
value=(None, int) most.

setsockopt(socket.SOL_ALG, socket.ALG_SET_AEAD_AUTHSIZE, (None, taglen))

What do you think?

Christian

From tjreedy at udel.edu  Sun Aug 21 17:22:59 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 21 Aug 2016 17:22:59 -0400
Subject: [Python-Dev] Anyone know Brendan Scott, author of 'Python for Kids'?
Message-ID: <npd63k$4pv$1@blaine.gmane.org>

Brendan Scott, author of 'Python for Kids for Dummies' is revising his 
book to produce a Python 3 version.  Great so far.

Unfortunately, he thinks that Python 3 turned range() into a (generator) 
function that produces a generator, one that can be indexed and sliced. 
I left a comment on his blog explaining that range() still produces 
arithmetic sequence objects.  The difference is that in Py 3, the 
compact representation is not expanded to a verbose (and redundant) 
Python list until requested.  It is *not* like map and filter, which 
*were* changed to return iterators.

For whatever reason, my comment had no effect and today on
https://python4kids.brendanscott.com/2016/08/21/python-for-kids-python-3-project-7/
he still says, near the top

 >>> # range(10) is no longer a list. Rather, it's a generator
 >>> # so the [:] operator slices the generator. You can use list()
 >>> # to see what list the generator corresponds to.

I left a second comment, again with interactive code examples, but I 
fear the result (nothing) will be the same.  We cannot stop people from 
publishing such misleading dis-information,  but we can discourage it, 
and I think we should, especially if we can before paper copies are 
printed.  Beginners have enough problem with iterables versus iterators 
without a major beginner book confusing the two.  I imagine this leading 
to tutor-list and Stackoverflow questions like "Why doesn't slicing my 
generator work?  It did for range?"

So, if you agree with me, please either write Brendan personally if you 
know him, or just leave your own comment on the blog.

-- 
Terry Jan Reedy


From gvanrossum at gmail.com  Sun Aug 21 22:14:09 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sun, 21 Aug 2016 19:14:09 -0700
Subject: [Python-Dev] socket.setsockopt() with optval=NULL
In-Reply-To: <c9bbe651-af70-175f-d386-e81e791b2f44@python.org>
References: <c9bbe651-af70-175f-d386-e81e791b2f44@python.org>
Message-ID: <CAP7+vJ+TedQEfJ6R=ri3z=6tDW0tFnzT3A8s0Yb3WnhANKCKqA@mail.gmail.com>

Wouldn't

setsockopt(socket.SOL_ALG, socket.ALG_SET_AEAD_AUTHSIZE, None, taglen)

be more consistent?

--Guido (mobile)

On Aug 21, 2016 5:40 AM, "Christian Heimes" <christian at python.org> wrote:

> Hi,
>
> the socket.setsockopt(level, optname, value) method has two calling
> variants. When it is called with a buffer-like object as value, it calls
> the C API function setsockopt() with optval=buffer.buf and
> optlen=buffer.len. When value is an integer, setsockopt() packs it as
> int32 and sends it with optlen=4.
>
> ---
> # example.py
> import socket
> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,
>     b'\x00\x00\x00\x00')
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> ---
>
> $ strace -e setsockopt ./python example.py
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
>
>
> For AF_ALG (Linux Kernel crypto) I need a way to call the C API function
> setsockopt() with optval=NULL and optlen as any arbitrary number. I have
> been playing with multiple ideas. So far I liked the idea of
> value=(None, int) most.
>
> setsockopt(socket.SOL_ALG, socket.ALG_SET_AEAD_AUTHSIZE, (None, taglen))
>
> What do you think?
>
> Christian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160821/1cdae195/attachment.html>

From vadmium+py at gmail.com  Sun Aug 21 23:45:01 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Mon, 22 Aug 2016 03:45:01 +0000
Subject: [Python-Dev] socket.setsockopt() with optval=NULL
In-Reply-To: <c9bbe651-af70-175f-d386-e81e791b2f44@python.org>
References: <c9bbe651-af70-175f-d386-e81e791b2f44@python.org>
Message-ID: <CA+eR4cH_R3xo=+hMwOZ_A6P0FZ+kAqegf9Eo6ZMa_0iWd97jUg@mail.gmail.com>

On 21 August 2016 at 12:37, Christian Heimes <christian at python.org> wrote:
> the socket.setsockopt(level, optname, value) method has two calling
> variants. When it is called with a buffer-like object as value, it calls
> the C API function setsockopt() with optval=buffer.buf and
> optlen=buffer.len. When value is an integer, setsockopt() packs it as
> int32 and sends it with optlen=4.
>
> ---
> # example.py
> import socket
> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,
>     b'\x00\x00\x00\x00')
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> ---
>
> $ strace -e setsockopt ./python example.py
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
>
>
> For AF_ALG (Linux Kernel crypto) I need a way to call the C API function
> setsockopt() with optval=NULL and optlen as any arbitrary number. I have
> been playing with multiple ideas. So far I liked the idea of
> value=(None, int) most.
>
> setsockopt(socket.SOL_ALG, socket.ALG_SET_AEAD_AUTHSIZE, (None, taglen))

Would this new functionality be open-ended? What would happen if you did

sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, (None, 4))

On Linux, this seems to fail with errno = EFAULT, which would be fine.
But this does not seem to be specified by Posix, and I imagine other
platforms might crash. On the other hand, these sort of holes are
already present in the socket module. E.g. with MSG_TRUNC
<https://bugs.python.org/issue24933> you can apparently get Python to
copy from unallocated memory.

From victor.stinner at gmail.com  Mon Aug 22 04:01:38 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 22 Aug 2016 10:01:38 +0200
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
In-Reply-To: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
References: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
Message-ID: <CAMpsgwZx2r2UcHMaXFMKGyFQVmnFhTaX40QnZiWsST0FF6yaEg@mail.gmail.com>

Hi,

I pushed the most basic implementation of _PyObject_FastCall(), it
doesn't support keyword parameters yet:
https://hg.python.org/cpython/rev/a1a29d20f52d
https://bugs.python.org/issue27128

Then I patched a lot of call sites calling PyObject_Call(),
PyObject_CallObject(), PyEval_CallObject(), etc. with a temporary
tuple. Just one example:

-            args = PyTuple_Pack(1, match);
-            if (!args) {
-                Py_DECREF(match);
-                goto error;
-            }
-            item = PyObject_CallObject(filter, args);
-            Py_DECREF(args);
+            item = _PyObject_FastCall(filter, &match, 1, NULL);

The next step is to support keyword parameters. In fact, it's already
supported in all cases except of Python functions:
https://bugs.python.org/issue27809

Supporting keyword parameters will allow to patch much code to avoid
temporary tuples, but it is also required for a much more interesting
change:
https://bugs.python.org/issue27810
"Add METH_FASTCALL: new calling convention for C functions"

I propose to add a new METH_FASTCALL calling convention. The example
using METH_VARARGS | METH_KEYWORDS:
   PyObject* func(DirEntry *self, PyObject *args, PyObject *kwargs)
becomes:
   PyObject* func(DirEntry *self, PyObject **args, int nargs, PyObject *kwargs)

Later, Argument Clinic will be modified to *generate* code using the
new METH_FASTCALL calling convention. Code written with Argument
Clinic will only need to be updated by Argument Clinic to get the new
faster calling convention (avoid the creation of a temporary tuple for
positional arguments).

Victor

From turnbull.stephen.fw at u.tsukuba.ac.jp  Mon Aug 22 05:47:06 2016
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Mon, 22 Aug 2016 18:47:06 +0900
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
Message-ID: <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>

Nick Coghlan writes:
 > On 21 August 2016 at 06:31, Steve Dower <steve.dower at python.org> wrote:

 > > My biggest concern is that it then falls onto users to know how
 > > to start Python with that flag.

The users I'm most worried about belong to organizations where
concerted effort has been made to "purify" the environment so that
they *can* use bytes-oriented code.  That is, getfilesystemencoding()
== getpreferredencoding() == what is actually used throughout the
system.  Such organizations will be able to choose the flag correctly,
and implement it organization-wide, I'm pretty sure.  I doubt that all
will choose UTF-8 at this point in time, though I wish they would.

 > Not necessarily, as this is one of the areas where commercial
 > redistributors can earn their revenue stream - by deciding that
 > flipping the default behaviour is the right thing to do for *their*
 > user base (which is inevitably only a subset of the overall Python
 > user base).

This assumes that the Python applications are the mission-critical
ones for their clients.  What if they're not?  I think the commercial
redistributors will have to make their decisions on a client-by-client
basis, too.  They may be in a better position to do so, but why buy
trouble?  They'll be quite conservative (unless they're basically
monopoly IT supplier to a whole organization, but they'll still have
to face potential problems from existing files on users' storage, and
perhaps applications that they supply but don't "own").

I have real trouble seeing trying to force UTF-8 as a good idea until
the organization mandates UTF-8. :-(  This really is an organizational
decision, to be implemented with client resources.  We can't do it for
them, redistributors can't do it for them.  It needs to be an option
in Python.

Python itself is already ready for UTF-8, except that on Windows
getfilesystemencoding and getpreferredencoding can't honestly return
'utf-8', AIUI.  I understand that that is exactly what Steve wants to
change, but "honestly" is the rub.  What happens if Python 3.6 is only
part of a bytes-oriented system, receives a filename forced to UTF-8-
encoded bytes, and passes that over a pipe or in shared memory or in a
file to a non-Python-3.6 application that trusts the system defaults?
"Boom!", no?  Is there any experience anywhere in any implementation
language with systems used on Windows that use this approach of
pretending the Windows world is UTF-8?  If not, why is it a good idea
for Python to go first?

 > Making that possible doesn't mean redistributors will actually follow
 > through, but it's an option worth keeping in mind, as while it does
 > increase the ecosystem complexity in the near term (since default
 > behaviour may vary based on how you obtained your Python runtime), in
 > the longer term it can allow for better informed design decisions at
 > the reference interpreter level. (For business process wonks, it's
 > essentially like running through a deliberate divergence/convergence
 > cycle at the level of the entire language ecosystem:
 > http://theagilepirate.net/archives/1392 )

It's worse than "the entire language ecosystem" -- it's your whole
business.[1]  If the proposed change to getfilesystemencoding and file
system APIs creates issues at all, it matters because files on disk,
or other applications that receive bytes from Python, refer to
filenames encoded in the preferred encoding != UTF-8.  It's unlikely
in the extreme that all such files are exclusively used by Python,
which at best means individual users will need to manage encodings
file by file.  At worst, some of the filenames so encoded will be
shared with applications that expect the preferred encoding, and then
you've got a war on your hands.

 > > On the other hand, having code opt-in or out of the new handling
 > > requires changing code (which is presumably not going to happen,
 > > or we wouldn't consider keeping the old behaviour and/or letting
 > > the user control it),

I don't understand why this argument doesn't cut both ways equally.
If you believe that, you should also believe that the same people who
won't change code to opt in also won't use a Python containing fix #1,
and may not install it at all.  Doesn't that matter?

 > I think you'll want to escalate this to a PEP as well

+1 for the reasons Nick gives.  The conclusions of this discussion
need a canonical URL.


Footnotes: 
[1]  I'm assuming that readers are going to associated "language" <-->
"Python".  The blog post Nick refers to is about the whole business.



From ncoghlan at gmail.com  Mon Aug 22 07:31:52 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 Aug 2016 21:31:52 +1000
Subject: [Python-Dev] Anyone know Brendan Scott,
 author of 'Python for Kids'?
In-Reply-To: <npd63k$4pv$1@blaine.gmane.org>
References: <npd63k$4pv$1@blaine.gmane.org>
Message-ID: <CADiSq7fSWMMs1TkXd+LhHrCESCEFE0j_1-2-X_sfe2q2iExU+Q@mail.gmail.com>

On 22 August 2016 at 07:22, Terry Reedy <tjreedy at udel.edu> wrote:
> So, if you agree with me, please either write Brendan personally if you know
> him, or just leave your own comment on the blog.

Brendan spoke at the inaugural PyCon Australia Education Seminar last
year, so I've contacted him (cc you) to suggest making the fix.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve.dower at python.org  Mon Aug 22 11:58:31 2016
From: steve.dower at python.org (Steve Dower)
Date: Mon, 22 Aug 2016 08:58:31 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
Message-ID: <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>

On 22Aug2016 0247, Stephen J. Turnbull wrote:
> Nick Coghlan writes:
>  > On 21 August 2016 at 06:31, Steve Dower <steve.dower at python.org> wrote:
>
>  > > My biggest concern is that it then falls onto users to know how
>  > > to start Python with that flag.
>
> The users I'm most worried about belong to organizations where
> concerted effort has been made to "purify" the environment so that
> they *can* use bytes-oriented code.  That is, getfilesystemencoding()
> == getpreferredencoding() == what is actually used throughout the
> system.  Such organizations will be able to choose the flag correctly,
> and implement it organization-wide, I'm pretty sure.  I doubt that all
> will choose UTF-8 at this point in time, though I wish they would.

I think that these are also the people who are likely to read a PEP and 
enable an environment variable to preserve the current behaviour. I'm 
more concerned about uncontrolled environments where a library breaks on 
a random user's machine because random user downloaded a file from a 
foreign website.

I don't recall whether I mentioned an environment variable (i.e. 
PYTHONUSELEGACYENCODING or similar) to switch back to mbcs:ignore by 
default, but it was always my intent to have one.

> Python itself is already ready for UTF-8, except that on Windows
> getfilesystemencoding and getpreferredencoding can't honestly return
> 'utf-8', AIUI.  I understand that that is exactly what Steve wants to
> change, but "honestly" is the rub.  What happens if Python 3.6 is only
> part of a bytes-oriented system, receives a filename forced to UTF-8-
> encoded bytes, and passes that over a pipe or in shared memory or in a
> file to a non-Python-3.6 application that trusts the system defaults?
> "Boom!", no?  Is there any experience anywhere in any implementation
> language with systems used on Windows that use this approach of
> pretending the Windows world is UTF-8?  If not, why is it a good idea
> for Python to go first?

The Windows world is Unicode. Mostly represented in UTF-16, but UTF-8 is 
entirely equivalent.

All MSVC users have been pushed towards Unicode for many years. The .NET 
Framework has defaulted to UTF-8 its entire existence. The use of code 
pages has been discouraged for decades. We're not going first :)

>  > > On the other hand, having code opt-in or out of the new handling
>  > > requires changing code (which is presumably not going to happen,
>  > > or we wouldn't consider keeping the old behaviour and/or letting
>  > > the user control it),
>
> I don't understand why this argument doesn't cut both ways equally.
> If you believe that, you should also believe that the same people who
> won't change code to opt in also won't use a Python containing fix #1,
> and may not install it at all.  Doesn't that matter?

People already do this (e.g. Python 2.7). I don't think it should matter 
enough to prevent us from making changes in new versions of Python. 
Otherwise, why would we ever release new versions?

So I guess the question here is: for organisations who have already 
(incorrectly) assumed that the file system encoding and the active code 
page are always the same, have built solid infrastructure around this 
using bytes (including ensuring that their systems never encounter 
external paths in glob/listdir/etc.), are currently using 3.5 and want 
to migrate to 3.6 - is an environment variable to change back to mbcs 
sufficient to meet their needs?

Cheers,
Steve

From eryksun at gmail.com  Mon Aug 22 13:31:44 2016
From: eryksun at gmail.com (eryk sun)
Date: Mon, 22 Aug 2016 17:31:44 +0000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
Message-ID: <CACL+1autpt49b3SEnTHVYBS2NMTmWhWMbsmKGwUXOLDMOdUJ3A@mail.gmail.com>

On Mon, Aug 22, 2016 at 3:58 PM, Steve Dower <steve.dower at python.org> wrote:
> All MSVC users have been pushed towards Unicode for many years. The .NET
> Framework has defaulted to UTF-8 its entire existence. The use of code pages
> has been discouraged for decades. We're not going first :)

I just wrote a simple function to enumerate the 822 system locales on
my Windows box (using EnumSystemLocalesEx and GetLocaleInfoEx, which
are Unicode-only functions), and 36.7% of them lack an ANSI codepage.
They're Unicode-only locales. UTF-8 is the only way to support these
locales with a bytes API.

From vgr255 at live.ca  Mon Aug 22 21:16:21 2016
From: vgr255 at live.ca (Emanuel Barry)
Date: Tue, 23 Aug 2016 01:16:21 +0000
Subject: [Python-Dev] Deprecating invalid escape sequences: review request
Message-ID: <CY1PR13MB0234F3FF68D63CC20B1A4C1F91EB0@CY1PR13MB0234.namprd13.prod.outlook.com>

Hello Python-dev,
some time ago I went ahead and implemented a patch to deprecate the invalid
escape sequences (e.g. \c, \g, \h, etc.) in str and bytes literals. The
change itself is pretty straightforward, and shouldn't be hard to review.
The change was split in two patches; one which does the actual deprecation
and adds relevant tests, and the other fixes all invalid escape sequences in
the entire CPython distribution (this resulted in a substantial patch file
of over 2000 lines). I'd like to get this reviewed and merged in time, so
I'm asking here. Thanks in advance!

http://bugs.python.org/issue27364

-Emanuel

From turnbull.stephen.fw at u.tsukuba.ac.jp  Tue Aug 23 00:21:22 2016
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Tue, 23 Aug 2016 13:21:22 +0900
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
Message-ID: <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>

Steve Dower writes:

 > The Windows world is Unicode. Mostly represented in UTF-16, but UTF-8 is 
 > entirely equivalent.

Sort of, yes, and not for present purposes.

AFAICS, the Windows world is mostly application/* media that require
substantial developer effort to extract text from; character encoding
is a minor annoyance.  These are not Unicode, even if the embedded
text uses the Unicode coded character set.  When in comes to text/*
media (including file system names), my personal experience is that
non-Unicode encodings are used often, even where they're forbidden
(and, ironically enough, where forbidden only by Windows users[1]).

As far as the UTF in use, I concede your expertise.

UTF-8 is absolutely not equivalent to UTF-16 from the point of view of
developers. Passing it to Windows APIs requires decoding to UTF-16 (or
from a Python developer's point of view, decoding to str and use of
str APIs).  That fact is what got you started on this whole proposal!

 > All MSVC users have been pushed towards Unicode for many years.

But that "push" is due to the use of UTF-16-based *W APIs and
deprecation of ACP-based *A APIs, right?  The input to *W APIs must be
decoded from all text/* content "out there", including UTF-8 content.
I don't see evidence that users have been pushed toward *UTF-8* in that
statement; they may be decoding from something else.  Unicode != UTF-8
for our purposes!

In any case, I suspect lot of people use Python to avoid C, and so
existing Python users may not be affected by MSVC "pressure".

 > The .NET Framework has defaulted to UTF-8

Default != enforce, though.  Do you know that almost nobody changes
the default, and that behavior is fairly uniform across different
classes of organization (specifically by language)?  Or did you mean
"enforce"?

 > its entire existence. The use of code pages has been discouraged
 > for decades. We're not going first :)

The fact that a framework, which by definition provides a world-
within-a-world, can insist on UTF-8 from the start is very different
from a generic programming language, which has deliberately provided
multiscript capability for decades.  People who buy in to .NET do so
because the disadvantages (which may include character encoding
conversion at the boundary, or "purification" of the environment to
use only UTF-8) are outweighed by both the individual features of the
framework and their packaging into a consistent whole.  This is
closely related to my idea about "effective monopoly IT providers".

On the contrary, people who use Python may very well have done to
*avoid* the Unicode strictures of .NET (or at least consider it a
convenience compared to changing user behavior to conform to .NET),
perhaps "localized" to a particular department or use case.  I believe
I've mentioned that my employers' various downloadable database
queries (course catalog, student rosters) are mostly structured as CSV
files, with the option to encode as UTF-8 or Shift-JIS.  I suspect
that is very common in Japanese universities because of the popularity
of Macs among educators, professionals, and students.  I don't know
about business and government, which is very Windows-oriented.  There,
I suspect Shift-JIS is the rule for text/* media, but Excel for data
tables and Word, Powerpoint, and PDF for "rich text" may be used almost
exclusively, so text/* may not be relevant in information interchange.

 > > I don't understand why this argument doesn't cut both ways
 > > equally.  If you believe that, you should also believe that the
 > > same people who won't change code to opt in also won't use a
 > > Python containing fix #1, and may not install it at all.  Doesn't
 > > that matter?
 > 
 > People already do this (e.g. Python 2.7). I don't think it should
 > matter enough to prevent us from making changes in new versions of
 > Python.

Of course it shouldn't, for the generic idea of change.  But the
argument you made is that "if we don't *force* UTF-8, users who won't
change code won't get the benefit of UTF-8".  My rebuttal is that "if
we *do* force UTF-8, those same users lose the benefit of both Python
3.6 and UTF-8."  It matters how many are in that situation, but
unfortunately we'll just have to guess about that.

 > So I guess the question here is: for organisations who have already
 > (incorrectly) assumed that the file system encoding and the active
 > code page are always the same,

Stop bashing the users, please!  This "users are stupid, we know
better" is the attitude that scares me about this proposal.  In the
enterprises I'm talking about, that is an organizational decision, not
an assumption.  (It is likely to be "close enough" to true in some
cases that lack such a policy, too.)  Or are you telling me that
Windows will change the active code page behind the users' backs even
if it's told not to do so?

Now, you can argue that few organizations actually have such policies,
and you may be right.  I don't know, and you don't know.  The damage
to Python's reputation if even *one* such gets screwed by forcing
UTF-8 will be large, though.

 > have built solid infrastructure around this using bytes (including
 > ensuring that their systems never encounter external paths in
 > glob/listdir/etc.), are currently using 3.5 and want to migrate to
 > 3.6 - is an environment variable to change back to mbcs sufficient
 > to meet their needs?

I should hope so!  As you surely know, the amount of technical
knowledge and organizational discipline required to to build solid
infrastructure around non-UTF-8 encodings is great.  The set of
applications that use bytes and need the setting should be finite, and
the decision to migrate them to Python 3.6 is unlikely to be
thoughtless.

To be clear: asking users who want backward-compatible behavior to set
an environment variable does not count as a "screw" -- some will
complain, but "the defaults always suck for somebody".  Reasonable
people know that, and we can't do anything about the hysterics.

The questions then are, what are the costs and benefits to various
classes of user, and how big are those classes?  Here's how I see the
costs and benefits playing out:

1.  Organizations which behave like ".NET users" already have pure
    UTF-8 environments.  They win from Python defaulting to UTF-8,
    since Windows won't let them do it for themselves.  Now they can
    plug in bytes-oriented code written for the POSIX environment
    straight from upstream.

    Is that correct?  Ie, without transcoding, they can't now use
    bytes because their environment hands them UTF-8 but when Python
    hands those bytes to Windows, it assumes anything else but UTF-8?

BTW, I wonder how those organizations manage to get pure UTF-8
environments, given that Windows itself won't default to that.  Is it
just that they live in .NET and other applications that default to
producing UTF-8 text (in the rare(?) case that text is generated at
all, vs some application/* medium), and so never get near applications
that produce text in the active code page, and especially not near
applications that embed file system names encoded in a non-UTF-8
encoding in text/* media?

2.  Organizations with a mixed environment will get a different set of
    "random" failures when using bytes-oriented code from before.
    Bytes-oriented code still represents a substantial risk with the
    UTF-8 default.

3.  Organizations with pure "other" encoding environments in the short
    run will have to change Python's defaults (or use older Python
    versions, if UTF-8 is forced) for bytes-oriented code (which they
    may already have installed).

I guess from Nick's (and Victor's) point of view, we would also like
to know if we're going to be able to recruit more Windows-based
developers from group 1.

Overall, I think Nick's hybrid strategy is the way to go.  First, give
users the choice of 'mbcs' or 'utf-8' for the Windows encoding.  I see
no reason not to do this for locale.getpreferredencoding() at the same
time, as long as it's an option.  Then, default them to 'utf-8' for
the betas, document how to change the defaults prominently, reserve
the right to change defaults for the rcs and the release.  Now we see
how many and who screams, and what they do about the pain -- reset
defaults or mandate UTF-8 (or both for a transition period).

It would be a good idea to have a short list of libraries using bytes-
oriented code and their applications that users can easily install to
try out, too.  Our working assumption has to be that few Windows users
do have them installed already, because they haven't worked to date.


Footnotes: 
[1]  Users who *happen* to be Windows users.  Windows didn't make them
do these horrible things, but the software that does is used only on
Windows.

[2]  I wonder how they manage that, given that Windows itself won't
let them set the preferred encoding to UTF-8.  Just how does .NET
manage the non-UTF-8 content that it must occasionally encounter?


From benjamin at python.org  Tue Aug 23 00:41:42 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 22 Aug 2016 21:41:42 -0700
Subject: [Python-Dev] socket.setsockopt() with optval=NULL
In-Reply-To: <c9bbe651-af70-175f-d386-e81e791b2f44@python.org>
References: <c9bbe651-af70-175f-d386-e81e791b2f44@python.org>
Message-ID: <1471927302.930455.703183209.0326E0BD@webmail.messagingengine.com>

Another option would be add a setalg method with whatever (nice,
pythonic) API we want. Emulating the crummy C-level API needn't be a
goal I think.

On Sun, Aug 21, 2016, at 05:37, Christian Heimes wrote:
> Hi,
> 
> the socket.setsockopt(level, optname, value) method has two calling
> variants. When it is called with a buffer-like object as value, it calls
> the C API function setsockopt() with optval=buffer.buf and
> optlen=buffer.len. When value is an integer, setsockopt() packs it as
> int32 and sends it with optlen=4.
> 
> ---
> # example.py
> import socket
> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,
>     b'\x00\x00\x00\x00')
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> ---
> 
> $ strace -e setsockopt ./python example.py
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> 
> 
> For AF_ALG (Linux Kernel crypto) I need a way to call the C API function
> setsockopt() with optval=NULL and optlen as any arbitrary number. I have
> been playing with multiple ideas. So far I liked the idea of
> value=(None, int) most.
> 
> setsockopt(socket.SOL_ALG, socket.ALG_SET_AEAD_AUTHSIZE, (None, taglen))
> 
> What do you think?
> 
> Christian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org

From turnbull.stephen.fw at u.tsukuba.ac.jp  Tue Aug 23 04:46:47 2016
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Tue, 23 Aug 2016 17:46:47 +0900
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CACL+1autpt49b3SEnTHVYBS2NMTmWhWMbsmKGwUXOLDMOdUJ3A@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <CACL+1autpt49b3SEnTHVYBS2NMTmWhWMbsmKGwUXOLDMOdUJ3A@mail.gmail.com>
Message-ID: <22460.3447.823676.65278@turnbull.sk.tsukuba.ac.jp>

eryk sun writes:

 > I just wrote a simple function to enumerate the 822 system locales on
 > my Windows box (using EnumSystemLocalesEx and GetLocaleInfoEx, which
 > are Unicode-only functions), and 36.7% of them lack an ANSI codepage.
 > They're Unicode-only locales. UTF-8 is the only way to support these
 > locales with a bytes API.

Are the users of those locales banging on our door demanding such an API?

Apparently not; such banging would have resulted in a patch.  (That's
how you know it's a bang and not a whimper!)  Instead, Steve had to
volunteer one.

Pragmatically, I don't see anyone rushing to *supply* bytes-oriented
APIs, bytes-oriented networking stacks, or bytes-oriented applications
to the Windows world.  I doubt there are all that many purely bytes-
oriented libraries out there that are plug-compatible with existing
Windows libraries of similar functionality, and obviously superior.
So somebody's going to have to do some work to exploit this new
feature.

Who, and when?  If the answers are "uh, I dunno" and "eventually",
what's the big rush?  Making it possible to test such software on
Windows in the public release version of Python should be our goal for
3.6.  We can do that with an option to set the default codecs to
'utf-8', and the default being the backward-compatible 'mbcs'.  How we
deal with the existing deprecation, I don't really care ("now is
better than never", and everything currently on the table will need a
policy kludge).

If in 9 months after release of 3.6, there are apps targeting Windows
and using UTF-8 bytes APIs in beta (or nearing it), then we have
excellent reason to default to 'utf-8' for 3.7.

And of course the patch eliminating use of the *A APIs with their lack
of error-handling deserves nothing but a round of applause!

Steve

From yselivanov.ml at gmail.com  Tue Aug 23 11:56:25 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 23 Aug 2016 11:56:25 -0400
Subject: [Python-Dev] PEP 525
Message-ID: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>

Hi,

I think it's time to discuss PEP 525 on python-dev (pasted below).

There were no changes in the PEP since I posted it to python-ideas
a couple of weeks ago.

One really critical thing that will block PEP acceptance is
asynchronous generators (AG) finalization.  The problem is
to provide a reliable way to correctly close all AGs on program
shutdown.

To recap: PEP 492 requires an event loop or a coroutine runner
to run coroutines.  PEP 525 defines AGs using asynchronous
iteration protocol, also defined in PEP 492.  AGs require an
'async for' statement to iterate over them, which in turn can
only be used in a coroutine.  Therefore, AGs also require an
event loop or a coroutine runner to operate.

The finalization problem is related to partial iteration.
For instance, let's look at an ordinary synchronous generator:

   def gen():
     try:
       while True:
         yield 42
     finally:
       print("done")

   g = gen()
   next(g)
   next(g)
   del g

In the above example, when 'g' is GCed, the interpreter will
try to close the generator.  It will do that by throwing a
GeneratorExit exception into 'g', which would trigger the 'finally'
statement.

For AGs we have a similar problem.  Except that they can have
`await` expressions in their `finally` statements, which means
that the interpreter can't close them on its own.  An event
loop is required to run an AG, and an event loop is required to
close it correctly.

To enable correct AGs finalization, PEP 525 proposes to add a
`sys.set_asyncgen_finalizer` API.  The idea is to have a finalizer
callback assigned to each AG, and when it's time to close the AG,
that callback will be called.  The callback will be installed by
the event loop (or coroutine runner), and should schedule a
correct asynchronous finalization of the AG (remember, AGs can
have 'await' expressions in their finally statements).

The problem with 'set_asyncgen_finalizer' is that the event loop
doesn't know about AGs until they are GCed.  This can be a problem
if we want to write a program that gracefully closes all AGs
when the event loop is being closed.

There is an alternative approach to finalization of AGs: instead
of assigning a finalizer callback to an AG, we can add an API to
intercept AG first iteration.  That would allow event loops to
have weak references to all AGs running under their control:

1. that would make it possible to intercept AGs garbage collection
similarly to the currently proposed set_asyncgen_finalizer

2. it would also allow us to implement 'loop.shutdown' coroutine,
which would try to asynchronously close all open AGs.

The second approach gives event loops more control and allows to
implement APIs to collect open resources gracefully.  The only
downside is that it's a bit harder for event loops to work with.

Let's discuss.


PEP: 525
Title: Asynchronous Generators
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov <yury at magic.io>
Discussions-To: <python-dev at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 28-Jul-2016
Python-Version: 3.6
Post-History: 02-Aug-2016


Abstract
========

PEP 492 introduced support for native coroutines and ``async``/``await``
syntax to Python 3.5.  It is proposed here to extend Python's
asynchronous capabilities by adding support for
*asynchronous generators*.


Rationale and Goals
===================

Regular generators (introduced in PEP 255) enabled an elegant way of
writing complex *data producers* and have them behave like an iterator.

However, currently there is no equivalent concept for the *asynchronous
iteration protocol* (``async for``).  This makes writing asynchronous
data producers unnecessarily complex, as one must define a class that
implements ``__aiter__`` and ``__anext__`` to be able to use it in
an ``async for`` statement.

Essentially, the goals and rationale for PEP 255, applied to the
asynchronous execution case, hold true for this proposal as well.

Performance is an additional point for this proposal: in our testing of
the reference implementation, asynchronous generators are **2x** faster
than an equivalent implemented as an asynchronous iterator.

As an illustration of the code quality improvement, consider the
following class that prints numbers with a given delay once iterated::

     class Ticker:
         """Yield numbers from 0 to `to` every `delay` seconds."""

         def __init__(self, delay, to):
             self.delay = delay
             self.i = 0
             self.to = to

         def __aiter__(self):
             return self

         async def __anext__(self):
             i = self.i
             if i >= self.to:
                 raise StopAsyncIteration
             self.i += 1
             if i:
                 await asyncio.sleep(self.delay)
             return i


The same can be implemented as a much simpler asynchronous generator::

     async def ticker(delay, to):
         """Yield numbers from 0 to `to` every `delay` seconds."""
         for i in range(to):
             yield i
             await asyncio.sleep(delay)


Specification
=============

This proposal introduces the concept of *asynchronous generators* to
Python.

This specification presumes knowledge of the implementation of
generators and coroutines in Python (PEP 342, PEP 380 and PEP 492).


Asynchronous Generators
-----------------------

A Python *generator* is any function containing one or more ``yield``
expressions::

     def func():            # a function
         return

     def genfunc():         # a generator function
         yield

We propose to use the same approach to define
*asynchronous generators*::

     async def coro():      # a coroutine function
         await smth()

     async def asyncgen():  # an asynchronous generator function
         await smth()
         yield 42

The result of calling an *asynchronous generator function* is
an *asynchronous generator object*, which implements the asynchronous
iteration protocol defined in PEP 492.

It is a ``SyntaxError`` to have a non-empty ``return`` statement in an
asynchronous generator.


Support for Asynchronous Iteration Protocol
-------------------------------------------

The protocol requires two special methods to be implemented:

1. An ``__aiter__`` method returning an *asynchronous iterator*.
2. An ``__anext__`` method returning an *awaitable* object, which uses
    ``StopIteration`` exception to "yield" values, and
    ``StopAsyncIteration`` exception to signal the end of the iteration.

Asynchronous generators define both of these methods.  Let's manually
iterate over a simple asynchronous generator::

     async def genfunc():
         yield 1
         yield 2

     gen = genfunc()

     assert gen.__aiter__() is gen

     assert await gen.__anext__() == 1
     assert await gen.__anext__() == 2

     await gen.__anext__()  # This line will raise StopAsyncIteration.


Finalization
------------

PEP 492 requires an event loop or a scheduler to run coroutines.
Because asynchronous generators are meant to be used from coroutines,
they also require an event loop to run and finalize them.

Asynchronous generators can have ``try..finally`` blocks, as well as
``async with``.  It is important to provide a guarantee that, even
when partially iterated, and then garbage collected, generators can
be safely finalized.  For example::

     async def square_series(con, to):
         async with con.transaction():
             cursor = con.cursor(
                 'SELECT generate_series(0, $1) AS i', to)
             async for row in cursor:
                 yield row['i'] ** 2

     async for i in square_series(con, 1000):
         if i == 100:
             break

The above code defines an asynchronous generator that uses
``async with`` to iterate over a database cursor in a transaction.
The generator is then iterated over with ``async for``, which interrupts
the iteration at some point.

The ``square_series()`` generator will then be garbage collected,
and without a mechanism to asynchronously close the generator, Python
interpreter would not be able to do anything.

To solve this problem we propose to do the following:

1. Implement an ``aclose`` method on asynchronous generators
    returning a special *awaitable*.  When awaited it
    throws a ``GeneratorExit`` into the suspended generator and
    iterates over it until either a ``GeneratorExit`` or
    a ``StopAsyncIteration`` occur.

    This is very similar to what the ``close()`` method does to regular
    Python generators, except that an event loop is required to execute
    ``aclose()``.

2. Raise a ``RuntimeError``, when an asynchronous generator executes
    a ``yield`` expression in its ``finally`` block (using ``await``
    is fine, though)::

         async def gen():
             try:
                 yield
             finally:
                 await asyncio.sleep(1)   # Can use 'await'.

                 yield                    # Cannot use 'yield',
                                          # this line will trigger a
                                          # RuntimeError.

3. Add two new methods to the ``sys`` module:
    ``set_asyncgen_finalizer()`` and ``get_asyncgen_finalizer()``.

The idea behind ``sys.set_asyncgen_finalizer()`` is to allow event
loops to handle generators finalization, so that the end user
does not need to care about the finalization problem, and it just
works.

When an asynchronous generator is iterated for the first time,
it stores a reference to the current finalizer.  If there is none,
a ``RuntimeError`` is raised.  This provides a strong guarantee that
every asynchronous generator object will always have a finalizer
installed by the correct event loop.

When an asynchronous generator is about to be garbage collected,
it calls its cached finalizer.  The assumption is that the finalizer
will schedule an ``aclose()`` call with the loop that was active
when the iteration started.

For instance, here is how asyncio is modified to allow safe
finalization of asynchronous generators::

    # asyncio/base_events.py

    class BaseEventLoop:

        def run_forever(self):
            ...
            old_finalizer = sys.get_asyncgen_finalizer()
            sys.set_asyncgen_finalizer(self._finalize_asyncgen)
            try:
                ...
            finally:
                sys.set_asyncgen_finalizer(old_finalizer)
                ...

        def _finalize_asyncgen(self, gen):
            self.create_task(gen.aclose())

``sys.set_asyncgen_finalizer()`` is thread-specific, so several event
loops running in parallel threads can use it safely.


Asynchronous Generator Object
-----------------------------

The object is modeled after the standard Python generator object.
Essentially, the behaviour of asynchronous generators is designed
to replicate the behaviour of synchronous generators, with the only
difference in that the API is asynchronous.

The following methods and properties are defined:

1. ``agen.__aiter__()``: Returns ``agen``.

2. ``agen.__anext__()``: Returns an *awaitable*, that performs one
    asynchronous generator iteration when awaited.

3. ``agen.asend(val)``: Returns an *awaitable*, that pushes the
    ``val`` object in the ``agen`` generator.  When the ``agen`` has
    not yet been iterated, ``val`` must be ``None``.

    Example::

        async def gen():
            await asyncio.sleep(0.1)
            v = yield 42
            print(v)
            await asyncio.sleep(0.2)

        g = gen()

        await g.asend(None)      # Will return 42 after sleeping
                                 # for 0.1 seconds.

        await g.asend('hello')   # Will print 'hello' and
                                 # raise StopAsyncIteration
                                 # (after sleeping for 0.2 seconds.)

4. ``agen.athrow(typ, [val, [tb]])``: Returns an *awaitable*, that
    throws an exception into the ``agen`` generator.

    Example::

        async def gen():
            try:
                await asyncio.sleep(0.1)
                yield 'hello'
            except ZeroDivisionError:
                await asyncio.sleep(0.2)
                yield 'world'

        g = gen()
        v = await g.asend(None)
        print(v)                # Will print 'hello' after
                                # sleeping for 0.1 seconds.

        v = await g.athrow(ZeroDivisionError)
        print(v)                # Will print 'world' after
                                $ sleeping 0.2 seconds.

5. ``agen.aclose()``: Returns an *awaitable*, that throws a
    ``GeneratorExit`` exception into the generator.  The *awaitable* can
    either return a yielded value, if ``agen`` handled the exception,
    or ``agen`` will be closed and the exception will propagate back
    to the caller.

6. ``agen.__name__`` and ``agen.__qualname__``: readable and writable
    name and qualified name attributes.

7. ``agen.ag_await``: The object that ``agen`` is currently *awaiting*
    on, or ``None``.  This is similar to the currently available
    ``gi_yieldfrom`` for generators and ``cr_await`` for coroutines.

8. ``agen.ag_frame``, ``agen.ag_running``, and ``agen.ag_code``:
    defined in the same way as similar attributes of standard generators.

``StopIteration`` and ``StopAsyncIteration`` are not propagated out of
asynchronous generators, and are replaced with a ``RuntimeError``.


Implementation Details
----------------------

Asynchronous generator object (``PyAsyncGenObject``) shares the
struct layout with ``PyGenObject``.  In addition to that, the
reference implementation introduces three new objects:

1. ``PyAsyncGenASend``: the awaitable object that implements
    ``__anext__`` and ``asend()`` methods.

2. ``PyAsyncGenAThrow``: the awaitable object that implements
    ``athrow()`` and ``aclose()`` methods.

3. ``_PyAsyncGenWrappedValue``: every directly yielded object from an
    asynchronous generator is implicitly boxed into this structure. This
    is how the generator implementation can separate objects that are
    yielded using regular iteration protocol from objects that are
    yielded using asynchronous iteration protocol.

``PyAsyncGenASend`` and ``PyAsyncGenAThrow`` are awaitables (they have
``__await__`` methods returning ``self``) and are coroutine-like objects
(implementing ``__iter__``, ``__next__``, ``send()`` and ``throw()``
methods).  Essentially, they control how asynchronous generators are
iterated:

.. image:: pep-0525-1.png
    :align: center
    :width: 80%


PyAsyncGenASend and PyAsyncGenAThrow
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``PyAsyncGenASend`` is a coroutine-like object that drives ``__anext__``
and ``asend()`` methods and implements the asynchronous iteration
protocol.

``agen.asend(val)`` and ``agen.__anext__()`` return instances of
``PyAsyncGenASend`` (which hold references back to the parent
``agen`` object.)

The data flow is defined as follows:

1. When ``PyAsyncGenASend.send(val)`` is called for the first time,
    ``val`` is pushed to the parent ``agen`` object (using existing
    facilities of ``PyGenObject``.)

    Subsequent iterations over the ``PyAsyncGenASend`` objects, push
    ``None`` to ``agen``.

    When a ``_PyAsyncGenWrappedValue`` object is yielded, it
    is unboxed, and a ``StopIteration`` exception is raised with the
    unwrapped value as an argument.

2. When ``PyAsyncGenASend.throw(*exc)`` is called for the first time,
    ``*exc`` is throwed into the parent ``agen`` object.

    Subsequent iterations over the ``PyAsyncGenASend`` objects, push
    ``None`` to ``agen``.

    When a ``_PyAsyncGenWrappedValue`` object is yielded, it
    is unboxed, and a ``StopIteration`` exception is raised with the
    unwrapped value as an argument.

3. ``return`` statements in asynchronous generators raise
    ``StopAsyncIteration`` exception, which is propagated through
    ``PyAsyncGenASend.send()`` and ``PyAsyncGenASend.throw()`` methods.

``PyAsyncGenAThrow`` is very similar to ``PyAsyncGenASend``.  The only
difference is that ``PyAsyncGenAThrow.send()``, when called first time,
throws an exception into the parent ``agen`` object (instead of pushing
a value into it.)


New Standard Library Functions and Types
----------------------------------------

1. ``types.AsyncGeneratorType`` -- type of asynchronous generator
    object.

2. ``sys.set_asyncgen_finalizer()`` and ``sys.get_asyncgen_finalizer()``
    methods to set up asynchronous generators finalizers in event loops.

3. ``inspect.isasyncgen()`` and ``inspect.isasyncgenfunction()``
    introspection functions.


Backwards Compatibility
-----------------------

The proposal is fully backwards compatible.

In Python 3.5 it is a ``SyntaxError`` to define an ``async def``
function with a ``yield`` expression inside, therefore it's safe to
introduce asynchronous generators in 3.6.


Performance
===========

Regular Generators
------------------

There is no performance degradation for regular generators.
The following micro benchmark runs at the same speed on CPython with
and without asynchronous generators::

     def gen():
         i = 0
         while i < 100000000:
             yield i
             i += 1

     list(gen())


Improvements over asynchronous iterators
----------------------------------------

The following micro-benchmark shows that asynchronous generators
are about **2.3x faster** than asynchronous iterators implemented in
pure Python::

     N = 10 ** 7

     async def agen():
         for i in range(N):
             yield i

     class AIter:
         def __init__(self):
             self.i = 0

         def __aiter__(self):
             return self

         async def __anext__(self):
             i = self.i
             if i >= N:
                 raise StopAsyncIteration
             self.i += 1
             return i


Design Considerations
=====================


``aiter()`` and ``anext()`` builtins
------------------------------------

Originally, PEP 492 defined ``__aiter__`` as a method that should
return an *awaitable* object, resulting in an asynchronous iterator.

However, in CPython 3.5.2, ``__aiter__`` was redefined to return
asynchronous iterators directly.  To avoid breaking backwards
compatibility, it was decided that Python 3.6 will support both
ways: ``__aiter__`` can still return an *awaitable* with
a ``DeprecationWarning`` being issued.

Because of this dual nature of ``__aiter__`` in Python 3.6, we cannot
add a synchronous implementation of ``aiter()`` built-in. Therefore,
it is proposed to wait until Python 3.7.


Asynchronous list/dict/set comprehensions
-----------------------------------------

Syntax for asynchronous comprehensions is unrelated to the asynchronous
generators machinery, and should be considered in a separate PEP.


Asynchronous ``yield from``
---------------------------

While it is theoretically possible to implement ``yield from`` support
for asynchronous generators, it would require a serious redesign of the
generators implementation.

``yield from`` is also less critical for asynchronous generators, since
there is no need provide a mechanism of implementing another coroutines
protocol on top of coroutines.  And to compose asynchronous generators a
simple ``async for`` loop can be used::

     async def g1():
         yield 1
         yield 2

     async def g2():
         async for v in g1():
             yield v


Why the ``asend()`` and ``athrow()`` methods are necessary
----------------------------------------------------------

They make it possible to implement concepts similar to
``contextlib.contextmanager`` using asynchronous generators.
For instance, with the proposed design, it is possible to implement
the following pattern::

     @async_context_manager
     async def ctx():
         await open()
         try:
             yield
         finally:
             await close()

     async with ctx():
         await ...

Another reason is that it is possible to push data and throw exceptions
into asynchronous generators using the object returned from
``__anext__`` object, but it is hard to do that correctly.  Adding
explicit ``asend()`` and ``athrow()`` will pave a safe way to
accomplish that.

In terms of implementation, ``asend()`` is a slightly more generic
version of ``__anext__``, and ``athrow()`` is very similar to
``aclose()``.  Therefore having these methods defined for asynchronous
generators does not add any extra complexity.


Example
=======

A working example with the current reference implementation (will
print numbers from 0 to 9 with one second delay)::

     async def ticker(delay, to):
         for i in range(to):
             yield i
             await asyncio.sleep(delay)


     async def run():
         async for i in ticker(1, 10):
             print(i)


     import asyncio
     loop = asyncio.get_event_loop()
     try:
         loop.run_until_complete(run())
     finally:
         loop.close()


Implementation
==============

The complete reference implementation is available at [1]_.


References
==========

.. [1] https://github.com/1st1/cpython/tree/async_gen


Copyright
=========

This document has been placed in the public domain.

..
    Local Variables:
    mode: indented-text
    indent-tabs-mode: nil
    sentence-end-double-space: t
    fill-column: 70
    coding: utf-8
    End:


From steve.dower at python.org  Tue Aug 23 12:08:00 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 23 Aug 2016 09:08:00 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
Message-ID: <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>

I've trimmed fairly aggressively for the sake of not causing the rest of 
the list to mute our discussion (again :) ). Stephen - feel free to 
email me off list if I go too far or misrepresent you.

As a summary for people who don't want to read on (and Stephen will 
correct me if I misquote):

* we agree on removing use of the *A APIs within Python, which means 
Python will have to decode bytes before passing them to the operating system
* we agree on allowing users to switch the encoding between utf-8 and 
mbcs:replace (the current default)
* we agree on making utf-8 the default for 3.6.0b1 and closely 
monitoring the reaction
* Stephen sees "no reason not to change locale.getpreferredencoding()" 
(default encoding for open()) at the same time with the same switches, 
while I'm not quite as confident. Do users generally specify an encoding 
these days? I know I always put utf-8 there.

Does anyone else have concerns or questions?



On 22Aug2016 2121, Stephen J. Turnbull wrote:
> UTF-8 is absolutely not equivalent to UTF-16 from the point of view of
> developers. Passing it to Windows APIs requires decoding to UTF-16 (or
> from a Python developer's point of view, decoding to str and use of
> str APIs).  That fact is what got you started on this whole proposal!

As encoded bytes, that's true, but as far as correctly encoding text, 
they are equivalent.

>  > All MSVC users have been pushed towards Unicode for many years.
>
> But that "push" is due to the use of UTF-16-based *W APIs and
> deprecation of ACP-based *A APIs, right?  The input to *W APIs must be
> decoded from all text/* content "out there", including UTF-8 content.
> I don't see evidence that users have been pushed toward *UTF-8* in that
> statement; they may be decoding from something else.  Unicode != UTF-8
> for our purposes!

Yes, the operating system pushes people towards *W APIs, and the 
languages commonly used on that operating system follow.

Windows has (for as long as it matters) always been UTF-16 for paths and 
bytes for content. Nowhere does the operating system tell you how to 
read your text file except as raw bytes, and content types are meant to 
provide the encoding information you need. Languages each determine how 
to read files in "text" mode, but that's not bound to or enforced by the 
operating system in any way.

>  > The .NET Framework has defaulted to UTF-8
>
> Default != enforce, though.  Do you know that almost nobody changes
> the default, and that behavior is fairly uniform across different
> classes of organization (specifically by language)?  Or did you mean
> "enforce"?

This will also not enforce anything that the operating system doesn't 
enforce. Windows uses Unicode to represent paths and requires them to be 
passed as UTF-16 encoded bytes. If you don't do that, it'll convert for 
you. My proposal is for Python to do the conversion instead.

(In .NET, users have to decode a byte array if they want to get a 
string. There aren't any APIs that take byte[] as if it were text, so 
it's basically the same separation between bytes/str that Python 3 
introduced, except without any allowance for bytes to still be used in 
places where text is needed.)

> To be clear: asking users who want backward-compatible behavior to set
> an environment variable does not count as a "screw" -- some will
> complain, but "the defaults always suck for somebody".  Reasonable
> people know that, and we can't do anything about the hysterics.

Good. Glad we agree on this.


> 1.  Organizations which behave like ".NET users" already have pure
>     UTF-8 environments.  They win from Python defaulting to UTF-8,
>     since Windows won't let them do it for themselves.  Now they can
>     plug in bytes-oriented code written for the POSIX environment
>     straight from upstream.
>
>     Is that correct?  Ie, without transcoding, they can't now use
>     bytes because their environment hands them UTF-8 but when Python
>     hands those bytes to Windows, it assumes anything else but UTF-8?

If you give Windows anything but UTF-16 as a path, it will convert to 
UTF-16. The change is to convert to UTF-16 ourselves, so Windows will 
never see the original bytes. To do that conversion, we need to know 
what encoding the incoming bytes are encoded with.

Python users will either transcode from bytes in encoding X to str, 
transcode from bytes in encoding X to bytes in UTF-8, or keep their 
bytes in UTF-8 if that's how they started.

(I feel like there's some other misunderstanding going on here, because 
I know you understand how encoding works, but I can't figure out what it 
is or what I need to say to trigger clarity. :( )

Windows does not support using UTF-8 encoded bytes as text. UTF-16 is 
the universal encoding. (Basically the only thing you can reliably do 
with UTF-8 bytes in the Windows API is convert them to UTF-16 - see the 
MultiByteToWideChar function. Everything else just treats it like a blob 
of meaningless data.)

> BTW, I wonder how those organizations manage to get pure UTF-8
> environments, given that Windows itself won't default to that.  Is it
> just that they live in .NET and other applications that default to
> producing UTF-8 text (in the rare(?) case that text is generated at
> all, vs some application/* medium), and so never get near applications
> that produce text in the active code page, and especially not near
> applications that embed file system names encoded in a non-UTF-8
> encoding in text/* media?

I doubt they're pure UTF-8, but they pay attention to what files are 
encoded with and explicitly decode into a common internal encoding.

> Overall, I think Nick's hybrid strategy is the way to go.  First, give
> users the choice of 'mbcs' or 'utf-8' for the Windows encoding.  I see
> no reason not to do this for locale.getpreferredencoding() at the same
> time, as long as it's an option.

The thing about this is that it's always been an option (the encoding 
argument to open() et al.), and specifically, an option that's required 
on all platforms. So I see one reason to not do it - users can (and do) 
override it in a cross-platform compatible way.

The biggest difference from the file system encoding is that the 
encoding for file contents is entirely the business of the application 
(and whichever other applications it talks to), while the OS is the main 
recipient of file system encoded text and so it gets a say in the chosen 
encoding.

I'm happy for this to be on the table though, but *I* need convincing 
that it's a good idea to do it now.

Cheers,
Steve


From rajiv.kumar at gmail.com  Tue Aug 23 22:38:27 2016
From: rajiv.kumar at gmail.com (Rajiv Kumar)
Date: Tue, 23 Aug 2016 19:38:27 -0700
Subject: [Python-Dev] PEP 525
In-Reply-To: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
References: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
Message-ID: <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>

Hi Yury,

I was playing with your implementation to gain a better understanding of
the operation of asend() and friends. Since I was explicitly trying to
"manually" advance the generators, I wasn't using asyncio or other event
loop. This meant that the first thing I ran into with my toy code was the
RuntimeError ("cannot iterate async generator without finalizer set").

As you have argued elsewhere, in practice the finalizer is likely to be set
by the event loop. Since the authors of event loops are likely to know that
they should set the finalizer, would it perhaps be acceptable to merely
issue a warning instead of an error if the finalizer is not set? That way
there isn't an extra hoop to jump through for simple examples.

In my case, I just called
    sys.set_asyncgen_finalizer(lambda g: 1)
to get around the error and continue playing :) (I realize that's a bad
thing to do but it didn't matter for the toy cases)

- Rajiv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160823/e6608445/attachment.html>

From turnbull.stephen.fw at u.tsukuba.ac.jp  Wed Aug 24 00:50:06 2016
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Wed, 24 Aug 2016 13:50:06 +0900
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
 <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
Message-ID: <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>

Steve Dower writes:

 > * Stephen sees "no reason not to change locale.getpreferredencoding()" 
 > (default encoding for open()) at the same time with the same switches, 
 > while I'm not quite as confident. Do users generally specify an encoding 
 > these days? I know I always put utf-8 there.

I was insufficiently specific.  "No reason not to" depends on separate
switches for file system encoding and preferred encoding.  That makes
things somewhat more complicated for implementation, and significantly
so for users.



From yselivanov.ml at gmail.com  Wed Aug 24 11:17:53 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 24 Aug 2016 11:17:53 -0400
Subject: [Python-Dev] PEP 525
In-Reply-To: <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>
References: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
 <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>
Message-ID: <b70893fe-8a82-1a2d-8d32-b3727bbce31c@gmail.com>

Hi Rajiv,

On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
> Hi Yury,
>
> I was playing with your implementation to gain a better understanding 
> of the operation of asend() and friends. Since I was explicitly trying 
> to "manually" advance the generators, I wasn't using asyncio or other 
> event loop. This meant that the first thing I ran into with my toy 
> code was the RuntimeError ("cannot iterate async generator without 
> finalizer set").
>
> As you have argued elsewhere, in practice the finalizer is likely to 
> be set by the event loop. Since the authors of event loops are likely 
> to know that they should set the finalizer, would it perhaps be 
> acceptable to merely issue a warning instead of an error if the 
> finalizer is not set? That way there isn't an extra hoop to jump 
> through for simple examples.
>
> In my case, I just called
>     sys.set_asyncgen_finalizer(lambda g: 1)
> to get around the error and continue playing :) (I realize that's a 
> bad thing to do but it didn't matter for the toy cases)

Yeah, maybe warning would be sufficient.  I just find it's highly 
unlikely that a lot of people would use async generators without a 
loop/coroutine runner, as it's a very tedious process.

Thank you,
Yury

From steve.dower at python.org  Wed Aug 24 11:44:07 2016
From: steve.dower at python.org (Steve Dower)
Date: Wed, 24 Aug 2016 08:44:07 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
 <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
 <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>
Message-ID: <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>

On 23Aug2016 2150, Stephen J. Turnbull wrote:
> Steve Dower writes:
>
>  > * Stephen sees "no reason not to change locale.getpreferredencoding()"
>  > (default encoding for open()) at the same time with the same switches,
>  > while I'm not quite as confident. Do users generally specify an encoding
>  > these days? I know I always put utf-8 there.
>
> I was insufficiently specific.  "No reason not to" depends on separate
> switches for file system encoding and preferred encoding.  That makes
> things somewhat more complicated for implementation, and significantly
> so for users.

Yes, it does, but it's about the only possible migration path.

I know Nick and Victor like the idea of a -X flag (or a direct -utf8 
flag), but I prefer more specific environment variables:

- PYTHONWINDOWSLEGACYSTDIO (for the console changes)
- PYTHONWINDOWSLEGACYPATHENCODING (assuming getfilesystemencoding() is utf8)
- PYTHONWINDOWSLEGACYLOCALEENCODING (assuming getpreferredencoding() is 
utf8)

I'm open to dropping "WINDOWS" from these if others don't think it's 
necessary. Swap "LEGACY" for "UNICODE" if we just offer the option to 
upgrade now rather than changing the default.

(I could also see the possibility of PYTHONWINDOWSSTRICT* options to use 
the default encoding but raise if decoding bytes fails - mbcs:strict 
rather than mbcs:replace. For utf-8 mode I'd want to use surrogatepass 
throughout, so it will always raise on invalid encoding, but there will 
be far fewer occurrences than for mbcs.)

I'll transform my earlier post into a PEP (or probably three PEPs - one 
for each change), since it seems like the paper trail is going to be 
more valuable than discussing it now. Without an actual build to test 
it's pretty hard to evaluate the impact.

Cheers,
Steve


From guido at python.org  Wed Aug 24 12:35:20 2016
From: guido at python.org (Guido van Rossum)
Date: Wed, 24 Aug 2016 09:35:20 -0700
Subject: [Python-Dev] PEP 525
In-Reply-To: <b70893fe-8a82-1a2d-8d32-b3727bbce31c@gmail.com>
References: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
 <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>
 <b70893fe-8a82-1a2d-8d32-b3727bbce31c@gmail.com>
Message-ID: <CAP7+vJKxDCuNH04vO3a45X3MSG+eN3FZHNx50qxu4oRj=QwKQg@mail.gmail.com>

On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov <yselivanov.ml at gmail.com>
wrote:

> On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
>
>> I was playing with your implementation to gain a better understanding of
>> the operation of asend() and friends. Since I was explicitly trying to
>> "manually" advance the generators, I wasn't using asyncio or other event
>> loop. This meant that the first thing I ran into with my toy code was the
>> RuntimeError ("cannot iterate async generator without finalizer set").
>>
>> As you have argued elsewhere, in practice the finalizer is likely to be
>> set by the event loop. Since the authors of event loops are likely to know
>> that they should set the finalizer, would it perhaps be acceptable to
>> merely issue a warning instead of an error if the finalizer is not set?
>> That way there isn't an extra hoop to jump through for simple examples.
>>
>> In my case, I just called
>>     sys.set_asyncgen_finalizer(lambda g: 1)
>> to get around the error and continue playing :) (I realize that's a bad
>> thing to do but it didn't matter for the toy cases)
>>
>
> Yeah, maybe warning would be sufficient.  I just find it's highly unlikely
> that a lot of people would use async generators without a loop/coroutine
> runner, as it's a very tedious process.
>

Heh, I had the same reaction as Rajiv. I think the tediousness is actually
a good argument that there's no reason to forbid this. I don't even think a
warning is needed. People who don't use a coroutine runner are probably
just playing around (maybe even in the REPL) and they shouldn't get advice
unasked.

Would it be possible to print a warning only when an async generator is
being finalized and doesn't run straight to the end without suspending or
yielding? For regular generators we have a similar exception (although I
don't recall whether we actually warn) -- if you call close() and it tries
to yield another value it is just GC'ed without giving the frame more
control. For an async generator there are two cases: either it tries to
yield another value (the first time this happens you can throw an error
back into it) or it tries to await -- in that case you can also throw an
error back into it, and if the error comes out unhandled you can print the
error (in both cases actually).

It's probably to specify all this behavior using some kind of default
finalizer (though you don't have to implement it that way).

Hopefully there will be other discussion as well, otherwise I'll have to
accept the PEP once this issue is cleared up. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160824/3755c2cf/attachment.html>

From yselivanov.ml at gmail.com  Wed Aug 24 15:00:19 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 24 Aug 2016 15:00:19 -0400
Subject: [Python-Dev] PEP 525
In-Reply-To: <CAP7+vJKxDCuNH04vO3a45X3MSG+eN3FZHNx50qxu4oRj=QwKQg@mail.gmail.com>
References: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
 <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>
 <b70893fe-8a82-1a2d-8d32-b3727bbce31c@gmail.com>
 <CAP7+vJKxDCuNH04vO3a45X3MSG+eN3FZHNx50qxu4oRj=QwKQg@mail.gmail.com>
Message-ID: <ba78fcec-8500-50ec-7c70-8aa5bf1620ce@gmail.com>

On 2016-08-24 12:35 PM, Guido van Rossum wrote:

> On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov 
> <yselivanov.ml at gmail.com <mailto:yselivanov.ml at gmail.com>> wrote:
>
>     On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
>
>         I was playing with your implementation to gain a better
>         understanding of the operation of asend() and friends. Since I
>         was explicitly trying to "manually" advance the generators, I
>         wasn't using asyncio or other event loop. This meant that the
>         first thing I ran into with my toy code was the RuntimeError
>         ("cannot iterate async generator without finalizer set").
>
>         As you have argued elsewhere, in practice the finalizer is
>         likely to be set by the event loop. Since the authors of event
>         loops are likely to know that they should set the finalizer,
>         would it perhaps be acceptable to merely issue a warning
>         instead of an error if the finalizer is not set? That way
>         there isn't an extra hoop to jump through for simple examples.
>
>         In my case, I just called
>             sys.set_asyncgen_finalizer(lambda g: 1)
>         to get around the error and continue playing :) (I realize
>         that's a bad thing to do but it didn't matter for the toy cases)
>
>
>     Yeah, maybe warning would be sufficient.  I just find it's highly
>     unlikely that a lot of people would use async generators without a
>     loop/coroutine runner, as it's a very tedious process.
>
>
> Heh, I had the same reaction as Rajiv. I think the tediousness is 
> actually a good argument that there's no reason to forbid this. I 
> don't even think a warning is needed. People who don't use a coroutine 
> runner are probably just playing around (maybe even in the REPL) and 
> they shouldn't get advice unasked.

Good point.

>
> Would it be possible to print a warning only when an async generator 
> is being finalized and doesn't run straight to the end without 
> suspending or yielding? For regular generators we have a similar 
> exception (although I don't recall whether we actually warn) -- if you 
> call close() and it tries to yield another value it is just GC'ed 
> without giving the frame more control.

Yes, we can implement the exact same semantics for AGs:

- A ResourceWarning will be issued if an AG is GCed and cannot be 
synchronously closed (that will happen if no finalizer is set and there 
are 'await' expressions in 'finally').

- A RuntimeError is issued when an AG is yielding (asynchronously) in 
its 'finally' block.

I think both of those things are already there in the reference 
implementation.  So we can just lift the requirement for asynchronous 
finalizer being set before you iterate an AG.

> For an async generator there are two cases: either it tries to yield 
> another value (the first time this happens you can throw an error back 
> into it) or it tries to await -- in that case you can also throw an 
> error back into it, and if the error comes out unhandled you can print 
> the error (in both cases actually).
>
> It's probably to specify all this behavior using some kind of default 
> finalizer (though you don't have to implement it that way).
>
> Hopefully there will be other discussion as well, otherwise I'll have 
> to accept the PEP once this issue is cleared up. :-)

Curious to hear your thoughts on two different approaches to 
finalization.  At this point, I'm inclined to change the PEP to use the 
second approach.  I think it gives much more power to event loops, and 
basically means that any kind of APIs to control AG (or to finalize the 
loop) is possible.

Thank you,
Yury

From srkunze at mail.de  Wed Aug 24 15:01:01 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 24 Aug 2016 21:01:01 +0200
Subject: [Python-Dev] PEP 525
In-Reply-To: <CAP7+vJKxDCuNH04vO3a45X3MSG+eN3FZHNx50qxu4oRj=QwKQg@mail.gmail.com>
References: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
 <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>
 <b70893fe-8a82-1a2d-8d32-b3727bbce31c@gmail.com>
 <CAP7+vJKxDCuNH04vO3a45X3MSG+eN3FZHNx50qxu4oRj=QwKQg@mail.gmail.com>
Message-ID: <a87bf011-c129-62aa-d207-ef4806437064@mail.de>

On 24.08.2016 18:35, Guido van Rossum wrote:
> On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov 
> <yselivanov.ml at gmail.com <mailto:yselivanov.ml at gmail.com>> wrote:
>
>     On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
>
>         I was playing with your implementation to gain a better
>         understanding of the operation of asend() and friends. Since I
>         was explicitly trying to "manually" advance the generators, I
>         wasn't using asyncio or other event loop. This meant that the
>         first thing I ran into with my toy code was the RuntimeError
>         ("cannot iterate async generator without finalizer set").
>
>         As you have argued elsewhere, in practice the finalizer is
>         likely to be set by the event loop. Since the authors of event
>         loops are likely to know that they should set the finalizer,
>         would it perhaps be acceptable to merely issue a warning
>         instead of an error if the finalizer is not set? That way
>         there isn't an extra hoop to jump through for simple examples.
>
>         In my case, I just called
>             sys.set_asyncgen_finalizer(lambda g: 1)
>         to get around the error and continue playing :) (I realize
>         that's a bad thing to do but it didn't matter for the toy cases)
>
>
>     Yeah, maybe warning would be sufficient.  I just find it's highly
>     unlikely that a lot of people would use async generators without a
>     loop/coroutine runner, as it's a very tedious process.
>
>
> Heh, I had the same reaction as Rajiv. I think the tediousness is 
> actually a good argument that there's no reason to forbid this. I 
> don't even think a warning is needed. People who don't use a coroutine 
> runner are probably just playing around (maybe even in the REPL) and 
> they shouldn't get advice unasked.

I also was irritated as Yury said there were absolutely no changes after 
python-ideas. He said he might consider a clearer warning for those 
examples at the beginning of the PEP to make them work for the reader.

>
> Would it be possible to print a warning only when an async generator 
> is being finalized and doesn't run straight to the end without 
> suspending or yielding? For regular generators we have a similar 
> exception (although I don't recall whether we actually warn) -- if you 
> call close() and it tries to yield another value it is just GC'ed 
> without giving the frame more control. For an async generator there 
> are two cases: either it tries to yield another value (the first time 
> this happens you can throw an error back into it) or it tries to await 
> -- in that case you can also throw an error back into it, and if the 
> error comes out unhandled you can print the error (in both cases 
> actually).
>
> It's probably to specify all this behavior using some kind of default 
> finalizer (though you don't have to implement it that way).

Does a default finalizer solve the "event loop does not know its AGs" 
problem?

> Hopefully there will be other discussion as well, otherwise I'll have 
> to accept the PEP once this issue is cleared up. :-)
>
> -- 
> --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160824/f8e247c3/attachment.html>

From yselivanov.ml at gmail.com  Wed Aug 24 15:05:02 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 24 Aug 2016 15:05:02 -0400
Subject: [Python-Dev] PEP 525
In-Reply-To: <a87bf011-c129-62aa-d207-ef4806437064@mail.de>
References: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
 <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>
 <b70893fe-8a82-1a2d-8d32-b3727bbce31c@gmail.com>
 <CAP7+vJKxDCuNH04vO3a45X3MSG+eN3FZHNx50qxu4oRj=QwKQg@mail.gmail.com>
 <a87bf011-c129-62aa-d207-ef4806437064@mail.de>
Message-ID: <8c11f607-4e4e-51a2-e117-8509fa7cf41e@gmail.com>

On 2016-08-24 3:01 PM, Sven R. Kunze wrote:

> On 24.08.2016 18:35, Guido van Rossum wrote:
>> On Wed, Aug 24, 2016 at 8:17 AM, Yury Selivanov 
>> <yselivanov.ml at gmail.com <mailto:yselivanov.ml at gmail.com>> wrote:
>>
>>     On 2016-08-23 10:38 PM, Rajiv Kumar wrote:
>>
>>         I was playing with your implementation to gain a better
>>         understanding of the operation of asend() and friends. Since
>>         I was explicitly trying to "manually" advance the generators,
>>         I wasn't using asyncio or other event loop. This meant that
>>         the first thing I ran into with my toy code was the
>>         RuntimeError ("cannot iterate async generator without
>>         finalizer set").
>>
>>         As you have argued elsewhere, in practice the finalizer is
>>         likely to be set by the event loop. Since the authors of
>>         event loops are likely to know that they should set the
>>         finalizer, would it perhaps be acceptable to merely issue a
>>         warning instead of an error if the finalizer is not set? That
>>         way there isn't an extra hoop to jump through for simple
>>         examples.
>>
>>         In my case, I just called
>>             sys.set_asyncgen_finalizer(lambda g: 1)
>>         to get around the error and continue playing :) (I realize
>>         that's a bad thing to do but it didn't matter for the toy cases)
>>
>>
>>     Yeah, maybe warning would be sufficient.  I just find it's highly
>>     unlikely that a lot of people would use async generators without
>>     a loop/coroutine runner, as it's a very tedious process.
>>
>>
>> Heh, I had the same reaction as Rajiv. I think the tediousness is 
>> actually a good argument that there's no reason to forbid this. I 
>> don't even think a warning is needed. People who don't use a 
>> coroutine runner are probably just playing around (maybe even in the 
>> REPL) and they shouldn't get advice unasked.
>
> I also was irritated as Yury said there were absolutely no changes 
> after python-ideas. He said he might consider a clearer warning for 
> those examples at the beginning of the PEP to make them work for the 
> reader.

Sorry for making you irritated.  Please feel free to remind me about any 
concrete changes to the PEP that I promised to add on python-ideas.  
I'll go and re-read that thread right now anyways.

Yury

From srkunze at mail.de  Wed Aug 24 15:07:12 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 24 Aug 2016 21:07:12 +0200
Subject: [Python-Dev] PEP 525
In-Reply-To: <ba78fcec-8500-50ec-7c70-8aa5bf1620ce@gmail.com>
References: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
 <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>
 <b70893fe-8a82-1a2d-8d32-b3727bbce31c@gmail.com>
 <CAP7+vJKxDCuNH04vO3a45X3MSG+eN3FZHNx50qxu4oRj=QwKQg@mail.gmail.com>
 <ba78fcec-8500-50ec-7c70-8aa5bf1620ce@gmail.com>
Message-ID: <a5750d8b-fec6-e35f-0068-0c684a1be061@mail.de>

On 24.08.2016 21:00, Yury Selivanov wrote:
>
>> For an async generator there are two cases: either it tries to yield 
>> another value (the first time this happens you can throw an error 
>> back into it) or it tries to await -- in that case you can also throw 
>> an error back into it, and if the error comes out unhandled you can 
>> print the error (in both cases actually).
>>
>> It's probably to specify all this behavior using some kind of default 
>> finalizer (though you don't have to implement it that way).
>>
>> Hopefully there will be other discussion as well, otherwise I'll have 
>> to accept the PEP once this issue is cleared up. :-)
>
> Curious to hear your thoughts on two different approaches to 
> finalization.  At this point, I'm inclined to change the PEP to use 
> the second approach.  I think it gives much more power to event loops, 
> and basically means that any kind of APIs to control AG (or to 
> finalize the loop) is possible.

I think your alternative approach is the better one. It feels more 
integrated even though it's harder for event loop implementors (which 
are rarer than normal event loop users). Also AG finalization is 
something that's not really custom to each AG but makes more sense at 
the event loop level, I think.

Best,
Sven

From srkunze at mail.de  Wed Aug 24 15:38:10 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 24 Aug 2016 21:38:10 +0200
Subject: [Python-Dev] PEP 525
In-Reply-To: <8c11f607-4e4e-51a2-e117-8509fa7cf41e@gmail.com>
References: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
 <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>
 <b70893fe-8a82-1a2d-8d32-b3727bbce31c@gmail.com>
 <CAP7+vJKxDCuNH04vO3a45X3MSG+eN3FZHNx50qxu4oRj=QwKQg@mail.gmail.com>
 <a87bf011-c129-62aa-d207-ef4806437064@mail.de>
 <8c11f607-4e4e-51a2-e117-8509fa7cf41e@gmail.com>
Message-ID: <d2935e58-253e-2305-7458-fbe8932a56b6@mail.de>

On 24.08.2016 21:05, Yury Selivanov wrote:
> Sorry for making you irritated.  Please feel free to remind me about 
> any concrete changes to the PEP that I promised to add on 
> python-ideas.  I'll go and re-read that thread right now anyways.

No problem as it seems I wasn't the only one. So, it doesn't matter 
anymore. :)

Best,
Sven

From victor.stinner at gmail.com  Wed Aug 24 18:53:06 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 25 Aug 2016 00:53:06 +0200
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
In-Reply-To: <CAMpsgwZx2r2UcHMaXFMKGyFQVmnFhTaX40QnZiWsST0FF6yaEg@mail.gmail.com>
References: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
 <CAMpsgwZx2r2UcHMaXFMKGyFQVmnFhTaX40QnZiWsST0FF6yaEg@mail.gmail.com>
Message-ID: <CAMpsgwZ6mgChOF+W--LojGS53gfd1BZRfsSLDYU0O9-o=RkYeg@mail.gmail.com>

2016-08-22 10:01 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:
> The next step is to support keyword parameters. In fact, it's already
> supported in all cases except of Python functions:
> https://bugs.python.org/issue27809

Serhiy Storchaka proposed to use a single C array for positional and
keyword arguments. Keyword arguments are passed as (key, value) pairs.
I just added this function:

     PyAPI_FUNC(PyObject *) _PyObject_FastCallKeywords(
                                PyObject *func,
                                PyObject **stack,
                                Py_ssize_t nargs,
                                Py_ssize_t nkwargs);

The function is not used yet. Serhiy proposed to enhance the functions
to parse arguments to support this format to pass arguments which
would allow to avoid the creation of a temporary dictionary in many
cases.

I proposed to use this format (instead of (PyObject **stack,
Py_ssize_t nargs, PyObject *kwargs)) for a new METH_FASTCALL calling
convention for C functions:
https://bugs.python.org/issue27810

Victor

From victor.stinner at gmail.com  Wed Aug 24 18:56:38 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 25 Aug 2016 00:56:38 +0200
Subject: [Python-Dev] New calling convention to avoid temporarily tuples
 when calling functions
In-Reply-To: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
References: <CAMpsgwY+Hg2LpMEE2upAfwxvRwecfR=wQXS_DKh+pEGwMUURhw@mail.gmail.com>
Message-ID: <CAMpsgwbo_VPD_Tdrb6v9ESG8D+4wtvswR=9ci0pqWxOtmh5WUA@mail.gmail.com>

Oh, I found a nice pice of CPython history in Modules/_pickle.c.
Extract of Python 3.3:
-----------------
/* A temporary cleaner API for fast single argument function call.

   XXX: Does caching the argument tuple provides any real performance benefits?

   A quick benchmark, on a 2.0GHz Athlon64 3200+ running Linux 2.6.24 with
   glibc 2.7, tells me that it takes roughly 20,000,000 PyTuple_New(1) calls
   when the tuple is retrieved from the freelist (i.e, call PyTuple_New() then
   immediately DECREF it) and 1,200,000 calls when allocating brand new tuples
   (i.e, call PyTuple_New() and store the returned value in an array), to save
   one second (wall clock time). Either ways, the loading time a pickle stream
   large enough to generate this number of calls would be massively
   overwhelmed by other factors, like I/O throughput, the GC traversal and
   object allocation overhead. So, I really doubt these functions provide any
   real benefits.

   On the other hand, oprofile reports that pickle spends a lot of time in
   these functions. But, that is probably more related to the function call
   overhead, than the argument tuple allocation.

   XXX: And, what is the reference behavior of these? Steal, borrow? At first
   glance, it seems to steal the reference of 'arg' and borrow the reference
   of 'func'. */
static PyObject *
_Pickler_FastCall(PicklerObject *self, PyObject *func, PyObject *arg)
-----------------

Extract of Python 3.4 (same function):
-----------------
    /* Note: this function used to reuse the argument tuple. This used to give
       a slight performance boost with older pickle implementations where many
       unbuffered reads occurred (thus needing many function calls).

       However, this optimization was removed because it was too complicated
       to get right. It abused the C API for tuples to mutate them which led
       to subtle reference counting and concurrency bugs. Furthermore, the
       introduction of protocol 4 and the prefetching optimization via peek()
       significantly reduced the number of function calls we do. Thus, the
       benefits became marginal at best. */
-----------------

It recalls me the story of property_descr_get() optimizations :-)

I hope that the new generic "fastcall" functions will provide a safe
and reliable optimization for the pickle module, property_descr_get()
and others optimized functions.

Victor

From ncoghlan at gmail.com  Thu Aug 25 07:40:44 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Aug 2016 21:40:44 +1000
Subject: [Python-Dev] PEP 525
In-Reply-To: <ba78fcec-8500-50ec-7c70-8aa5bf1620ce@gmail.com>
References: <eb49441b-2936-2264-8dce-c48b9c4d620e@gmail.com>
 <CAGWVM=j5=d_o3U-=hZFGyC8OHnnQHVWjiMimxC7MCWEn2EMmHQ@mail.gmail.com>
 <b70893fe-8a82-1a2d-8d32-b3727bbce31c@gmail.com>
 <CAP7+vJKxDCuNH04vO3a45X3MSG+eN3FZHNx50qxu4oRj=QwKQg@mail.gmail.com>
 <ba78fcec-8500-50ec-7c70-8aa5bf1620ce@gmail.com>
Message-ID: <CADiSq7eCb=mfPxgUyTpRKaXr+PnB=sr9FOv-Kmui0eWyEQrAPA@mail.gmail.com>

On 25 August 2016 at 05:00, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> On 2016-08-24 12:35 PM, Guido van Rossum wrote:
>> Hopefully there will be other discussion as well, otherwise I'll have to
>> accept the PEP once this issue is cleared up. :-)
>
> Curious to hear your thoughts on two different approaches to finalization.
> At this point, I'm inclined to change the PEP to use the second approach.  I
> think it gives much more power to event loops, and basically means that any
> kind of APIs to control AG (or to finalize the loop) is possible.

The notification/callback approach where the event loop is given a
chance to intercept the first iteration of any given coroutine seems
nicer to me, since it opens up more opportunities for event loops to
experiment with new ideas. As a very simple example, they could emit a
debugging message every time a new coroutine is started.

asyncio could provide a default notification hook that just mapped
weakref finalisation to asynchronous execution of aclose().

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From status at bugs.python.org  Fri Aug 26 12:08:44 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 26 Aug 2016 18:08:44 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160826160844.807A15672F@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-08-19 - 2016-08-26)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5606 (+12)
  closed 34018 (+53)
  total  39624 (+65)

Open issues with patches: 2436 


Issues opened (44)
==================

#12713: argparse: allow abbreviation of sub commands by users
https://bugs.python.org/issue12713  reopened by zach.ware

#27803: ctypes automatic byref failing on custom classes attributes
https://bugs.python.org/issue27803  opened by lepaperwan

#27805: In Python 3, open('/dev/stdout', 'a') raises OSError with errn
https://bugs.python.org/issue27805  opened by hathawsh

#27806: 2.7 32-bit builds fail on future releases of OS X due to depen
https://bugs.python.org/issue27806  opened by ned.deily

#27807: Prevent site-packages .pth files from causing test_site failur
https://bugs.python.org/issue27807  opened by Rosuav

#27808: os.waitpid does not return (0,0) when child has not exited (Fr
https://bugs.python.org/issue27808  opened by Casey Lucas

#27809: _PyObject_FastCall(): add support for keyword arguments
https://bugs.python.org/issue27809  opened by haypo

#27810: Add METH_FASTCALL: new calling convention for C functions
https://bugs.python.org/issue27810  opened by haypo

#27811: _PyGen_Finalize() should not fail with an exception
https://bugs.python.org/issue27811  opened by arigo

#27812: PyFrameObject.f_gen can be left pointing to a dangling generat
https://bugs.python.org/issue27812  opened by arigo

#27815: Make SSL suppress_ragged_eofs default more secure
https://bugs.python.org/issue27815  opened by martin.panter

#27816: sock.proto does not reflect actual protocol
https://bugs.python.org/issue27816  opened by christian.heimes

#27817: tkinter string variable misinterpreted as boolean
https://bugs.python.org/issue27817  opened by Andreas Bolsch

#27818: Speed up number format spec parsing
https://bugs.python.org/issue27818  opened by scoder

#27820: Possible bug in smtplib when initial_response_ok=False
https://bugs.python.org/issue27820  opened by Dario D'Amico

#27822: Fail to create _SelectorTransport with unbound socket
https://bugs.python.org/issue27822  opened by Paul McGuire

#27823: Change bare AttributeError messages to be more informative
https://bugs.python.org/issue27823  opened by ebarry

#27824: update ConfigParser docs regarding in-line comments
https://bugs.python.org/issue27824  opened by jahschwa

#27825: Make the documentation for statistics' data argument clearer.
https://bugs.python.org/issue27825  opened by nedbat

#27827: pathlib is_reserved fails for some reserved paths on Windows
https://bugs.python.org/issue27827  opened by eryksun

#27829: test.regrtest: changed environment variables are not logged
https://bugs.python.org/issue27829  opened by haypo

#27830: Add _PyObject_FastCallKeywords(): avoid the creation of a temp
https://bugs.python.org/issue27830  opened by haypo

#27831: Python 3.4.5 leaks memory when attempting to load non-existent
https://bugs.python.org/issue27831  opened by geeknik

#27833: Process is locked when try to execute Queue.put() inside
https://bugs.python.org/issue27833  opened by Guni

#27835: SystemExit in setUpClass etc. terminates test run
https://bugs.python.org/issue27835  opened by alexander.sturm

#27838: test_os.test_chown() random failure on "AMD64 FreeBSD CURRENT 
https://bugs.python.org/issue27838  opened by haypo

#27839: "Python [...] calls sys.displayhook() on unhandled exceptions"
https://bugs.python.org/issue27839  opened by jwilk

#27842: Order CSV header fields
https://bugs.python.org/issue27842  opened by holdenweb

#27844: Python-3.6a4 build messages to stderr (on AIX and xlc compiler
https://bugs.python.org/issue27844  opened by Michael.Felt

#27845: Optimize update_keyword_args() function
https://bugs.python.org/issue27845  opened by haypo

#27846: Base64 expansion factor is 4 to 3, not 6 to 4
https://bugs.python.org/issue27846  opened by Jens.J??rgen.Mortensen

#27847: os.set_inheritable() looks to be broken on OpenIndiana, regres
https://bugs.python.org/issue27847  opened by haypo

#27850: Remove 3DES from cipher list (sweet32 CVE-2016-2183)
https://bugs.python.org/issue27850  opened by christian.heimes

#27853: Add title to examples in importlib docs
https://bugs.python.org/issue27853  opened by brett.cannon

#27854: Installed 2.7: IDLE Help disabled because help.html is missing
https://bugs.python.org/issue27854  opened by terry.reedy

#27855: 2to3: Wrong code output w/ has_key without trailing space
https://bugs.python.org/issue27855  opened by rippey.e

#27859: argparse - subparsers does not retain namespace
https://bugs.python.org/issue27859  opened by nickpapior

#27860: Improvements to ipaddress module
https://bugs.python.org/issue27860  opened by moritzs

#27861: sqlite3 type confusion and multiple frees
https://bugs.python.org/issue27861  opened by tehybel

#27863: multiple issues in _elementtree module
https://bugs.python.org/issue27863  opened by tehybel

#27864: test_socket: unknown thread blocks forever on "AMD64 FreeBSD 9
https://bugs.python.org/issue27864  opened by haypo

#27865: WeakMethod does not support builtin methods
https://bugs.python.org/issue27865  opened by Antony.Lee

#27866: ssl: get list of enabled ciphers
https://bugs.python.org/issue27866  opened by christian.heimes

#27867: various issues due to misuse of PySlice_GetIndicesEx
https://bugs.python.org/issue27867  opened by tehybel



Most recent 15 issues with no replies (15)
==========================================

#27867: various issues due to misuse of PySlice_GetIndicesEx
https://bugs.python.org/issue27867

#27866: ssl: get list of enabled ciphers
https://bugs.python.org/issue27866

#27865: WeakMethod does not support builtin methods
https://bugs.python.org/issue27865

#27864: test_socket: unknown thread blocks forever on "AMD64 FreeBSD 9
https://bugs.python.org/issue27864

#27860: Improvements to ipaddress module
https://bugs.python.org/issue27860

#27859: argparse - subparsers does not retain namespace
https://bugs.python.org/issue27859

#27854: Installed 2.7: IDLE Help disabled because help.html is missing
https://bugs.python.org/issue27854

#27853: Add title to examples in importlib docs
https://bugs.python.org/issue27853

#27844: Python-3.6a4 build messages to stderr (on AIX and xlc compiler
https://bugs.python.org/issue27844

#27839: "Python [...] calls sys.displayhook() on unhandled exceptions"
https://bugs.python.org/issue27839

#27835: SystemExit in setUpClass etc. terminates test run
https://bugs.python.org/issue27835

#27827: pathlib is_reserved fails for some reserved paths on Windows
https://bugs.python.org/issue27827

#27825: Make the documentation for statistics' data argument clearer.
https://bugs.python.org/issue27825

#27815: Make SSL suppress_ragged_eofs default more secure
https://bugs.python.org/issue27815

#27812: PyFrameObject.f_gen can be left pointing to a dangling generat
https://bugs.python.org/issue27812



Most recent 15 issues waiting for review (15)
=============================================

#27861: sqlite3 type confusion and multiple frees
https://bugs.python.org/issue27861

#27860: Improvements to ipaddress module
https://bugs.python.org/issue27860

#27850: Remove 3DES from cipher list (sweet32 CVE-2016-2183)
https://bugs.python.org/issue27850

#27842: Order CSV header fields
https://bugs.python.org/issue27842

#27835: SystemExit in setUpClass etc. terminates test run
https://bugs.python.org/issue27835

#27830: Add _PyObject_FastCallKeywords(): avoid the creation of a temp
https://bugs.python.org/issue27830

#27823: Change bare AttributeError messages to be more informative
https://bugs.python.org/issue27823

#27822: Fail to create _SelectorTransport with unbound socket
https://bugs.python.org/issue27822

#27818: Speed up number format spec parsing
https://bugs.python.org/issue27818

#27812: PyFrameObject.f_gen can be left pointing to a dangling generat
https://bugs.python.org/issue27812

#27811: _PyGen_Finalize() should not fail with an exception
https://bugs.python.org/issue27811

#27809: _PyObject_FastCall(): add support for keyword arguments
https://bugs.python.org/issue27809

#27807: Prevent site-packages .pth files from causing test_site failur
https://bugs.python.org/issue27807

#27799: Fix base64-codec and bz2-codec incremental decoders
https://bugs.python.org/issue27799

#27781: Change sys.getfilesystemencoding() on Windows to UTF-8
https://bugs.python.org/issue27781



Top 10 most discussed issues (10)
=================================

#27809: _PyObject_FastCall(): add support for keyword arguments
https://bugs.python.org/issue27809  17 msgs

#27850: Remove 3DES from cipher list (sweet32 CVE-2016-2183)
https://bugs.python.org/issue27850  14 msgs

#27128: Add _PyObject_FastCall()
https://bugs.python.org/issue27128  12 msgs

#27643: test_ctypes fails on AIX with xlc
https://bugs.python.org/issue27643   9 msgs

#27768: ssl: get CPU cap flags for AESNI and PCLMULQDQ
https://bugs.python.org/issue27768   9 msgs

#27805: In Python 3, open('/dev/stdout', 'a') raises OSError with errn
https://bugs.python.org/issue27805   9 msgs

#12319: [http.client] HTTPConnection.request not support "chunked" Tra
https://bugs.python.org/issue12319   8 msgs

#25658: PyThread assumes pthread_key_t is an integer, which is against
https://bugs.python.org/issue25658   7 msgs

#12713: argparse: allow abbreviation of sub commands by users
https://bugs.python.org/issue12713   6 msgs

#25532: infinite loop when running inspect.unwrap over unittest.mock.c
https://bugs.python.org/issue25532   6 msgs



Issues closed (50)
==================

#2466: os.path.ismount doesn't work for mounts the user doesn't have 
https://bugs.python.org/issue2466  closed by r.david.murray

#6057: sqlite3 error classes should be documented
https://bugs.python.org/issue6057  closed by berker.peksag

#12982: Document that importing .pyo files needs python -O
https://bugs.python.org/issue12982  closed by berker.peksag

#19504: Change "customise" to "customize".
https://bugs.python.org/issue19504  closed by rhettinger

#20124: The documentation for the atTime parameter of TimedRotatimeFil
https://bugs.python.org/issue20124  closed by python-dev

#21718: sqlite3 cursor.description seems to rely on incomplete stateme
https://bugs.python.org/issue21718  closed by berker.peksag

#23746: sysconfg.is_python_build() is buggy
https://bugs.python.org/issue23746  closed by python-dev

#25564: Document that IDLE -python difference for `del __builtins__`
https://bugs.python.org/issue25564  closed by terry.reedy

#25604: [Minor] bug in integer true division algorithm
https://bugs.python.org/issue25604  closed by mark.dickinson

#25916: wave module readframes now returns bytes not str
https://bugs.python.org/issue25916  closed by r.david.murray

#26266: add classattribute to enum to handle non-Enum attributes
https://bugs.python.org/issue26266  closed by ethan.furman

#26488: hashlib command line interface
https://bugs.python.org/issue26488  closed by rhettinger

#26907: Add missing getsockopt constants
https://bugs.python.org/issue26907  closed by r.david.murray

#26984: int() can return not exact int instance
https://bugs.python.org/issue26984  closed by serhiy.storchaka

#27323: ncurses putwin() fails in test_module_funcs
https://bugs.python.org/issue27323  closed by serhiy.storchaka

#27487: -m switch regression in Python 3.5.2 (under rare circumstances
https://bugs.python.org/issue27487  closed by ncoghlan

#27539: negative Fraction ** negative int not normalized
https://bugs.python.org/issue27539  closed by mark.dickinson

#27587: Issues, reported by PVS-Studio static analyzer
https://bugs.python.org/issue27587  closed by berker.peksag

#27595: Document PEP 495 (Local Time Disambiguation) features
https://bugs.python.org/issue27595  closed by python-dev

#27598: Add Collection to collections.abc and typing
https://bugs.python.org/issue27598  closed by gvanrossum

#27614: Race in test_docxmlrpc.py
https://bugs.python.org/issue27614  closed by martin.panter

#27662: Check against PY_SSIZE_T_MAX instead of PY_SIZE_MAX in List_Ne
https://bugs.python.org/issue27662  closed by mark.dickinson

#27692: Clean serveral unnecessary NULL checks in exceptions.c
https://bugs.python.org/issue27692  closed by serhiy.storchaka

#27713: Spurious "platform dependent libraries" warnings when running 
https://bugs.python.org/issue27713  closed by ned.deily

#27714: some test_idle tests are not re-runnable, producing false fail
https://bugs.python.org/issue27714  closed by terry.reedy

#27782: Multi-phase extension module initialization, inconsistent exce
https://bugs.python.org/issue27782  closed by ncoghlan

#27787: Avoid test_main() in test_httplib; gc.collect() dangling threa
https://bugs.python.org/issue27787  closed by martin.panter

#27802: Add __eq__ and __ne__ to collections.abc.Sequence.
https://bugs.python.org/issue27802  closed by gvanrossum

#27804: IDLE 3.5.2 crashes when typing ^ on keyboard
https://bugs.python.org/issue27804  closed by ned.deily

#27813: When I assign for a item which list of single item
https://bugs.python.org/issue27813  closed by martin.panter

#27814: contextlib.suppress: Add whitelist argument
https://bugs.python.org/issue27814  closed by ncoghlan

#27819: Always distribute sdist packages as gztar.
https://bugs.python.org/issue27819  closed by jason.coombs

#27821: IDLE custom keymaps don't work anymore (Windows, Python 3.6.0a
https://bugs.python.org/issue27821  closed by terry.reedy

#27826: Null-pointer dereference in tuplehash() function
https://bugs.python.org/issue27826  closed by rhettinger

#27828: Regression in http.cookies parsing with single key without val
https://bugs.python.org/issue27828  closed by SilentGhost

#27832: fractions.Fraction with 3 arguments: error passes silently
https://bugs.python.org/issue27832  closed by mark.dickinson

#27834: test_datetime fails on "x86 Gentoo Non-Debug with X 3.x" build
https://bugs.python.org/issue27834  closed by belopolsky

#27836: VCRUNTIME140.DLL not found
https://bugs.python.org/issue27836  closed by steve.dower

#27837: Logging Cookbook Improvement
https://bugs.python.org/issue27837  closed by vinay.sajip

#27840: functools.partial: don't copy keywoard arguments in partial_ca
https://bugs.python.org/issue27840  closed by ned.deily

#27841: Use fast call in method_call() and slot_tp_new()
https://bugs.python.org/issue27841  closed by haypo

#27843: Spaces in filenames in setuptools
https://bugs.python.org/issue27843  closed by brett.cannon

#27848: C function calls: use Py_ssize_t rather than C int for number 
https://bugs.python.org/issue27848  closed by haypo

#27849: Spam
https://bugs.python.org/issue27849  closed by berker.peksag

#27851: re.sub() bug with IGNORECASE
https://bugs.python.org/issue27851  closed by r.david.murray

#27852: itertools -> flatten_all()
https://bugs.python.org/issue27852  closed by rhettinger

#27856: Issue while passing the parameter
https://bugs.python.org/issue27856  closed by berker.peksag

#27857: n identical objects divide into m different groups
https://bugs.python.org/issue27857  closed by terry.reedy

#27858: Add identity function to functools
https://bugs.python.org/issue27858  closed by rhettinger

#27862: Make urllib.parse runnable as a script
https://bugs.python.org/issue27862  closed by SilentGhost

From tjreedy at udel.edu  Sat Aug 27 18:25:40 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 27 Aug 2016 18:25:40 -0400
Subject: [Python-Dev] What do we do about bad slicing and possible crashes
 (issue 27867)
Message-ID: <npt414$mdm$1@blaine.gmane.org>

Slicing can be made to malfunction and even crash with an 'evil' 
__index__ method. https://bugs.python.org/issue27867

The crux of the problem is this: PySlice_GetIndicesEx
receives a slice object and a sequence length.  Calling __index__ on the 
start, stop, and step components can mutate the sequence and invalidate 
the length.  Adjusting the int values of start and stop according to an 
invalid length (in particular, one that is too long) will result in 
invalid results or a crash.

Possible actions -- very briefly.  For more see end of
https://bugs.python.org/issue27867?@ok_message=msg 273801
0. Do nothing.
1. Detect length change and raise.
2. Retrieve length after any possible changes and proceed as normal.

Possible implementation strategies for 1. and 2.
A. Change all functions that call PySlice_GetIndicesEx.
B. Add PySlice_GetIndicesEx2 (or ExEx?), which would receive *collection 
instead of length, so the length could be retrieved after the __index__ 
calls.  Change calls. Deprecate PySlice_GetIndicesEx.

Which of the 4 possible patches, if any, would be best?
I personally prefer 2B.

And what versions should be patched?

-- 
Terry Jan Reedy


From songofacandy at gmail.com  Sat Aug 27 22:59:41 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Sun, 28 Aug 2016 11:59:41 +0900
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
Message-ID: <CAEfz+TzRW1xjMPJCeMeW7-1EutFguOQ8LPmebBd6hF+rNsbnFg@mail.gmail.com>

Last call.  There are only two weeks until 3.6 beta.
Please review it if possible.

On Tue, Aug 9, 2016 at 10:12 PM, INADA Naoki <songofacandy at gmail.com> wrote:
> Hi, devs.
>
> I've implemented compact and ordered dictionary [1], which PyPy
> implemented in 2015 [2].
>
> Since it is my first large patch, I would like to have enough time for
> review cycle by Python 3.6 beta1.
>
> Could someone review it?
>
> [1] http://bugs.python.org/issue27350
> [2] https://morepypy.blogspot.jp/2015/01/faster-more-memory-efficient-and-more.html
>
> --
> INADA Naoki  <songofacandy at gmail.com>

-- 
INADA Naoki  <songofacandy at gmail.com>

From guido at python.org  Sun Aug 28 01:05:48 2016
From: guido at python.org (Guido van Rossum)
Date: Sat, 27 Aug 2016 22:05:48 -0700
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <CAEfz+TzRW1xjMPJCeMeW7-1EutFguOQ8LPmebBd6hF+rNsbnFg@mail.gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
 <CAEfz+TzRW1xjMPJCeMeW7-1EutFguOQ8LPmebBd6hF+rNsbnFg@mail.gmail.com>
Message-ID: <CAP7+vJJNq4+SFcg77bi-qHiP9at16BxMd7DNzOOs7A49YmPJ9g@mail.gmail.com>

Hopefully some core dev(s) can work on this during the core sprint, which
is from Sept 5-9.

On Sat, Aug 27, 2016 at 7:59 PM, INADA Naoki <songofacandy at gmail.com> wrote:

> Last call.  There are only two weeks until 3.6 beta.
> Please review it if possible.
>
> On Tue, Aug 9, 2016 at 10:12 PM, INADA Naoki <songofacandy at gmail.com>
> wrote:
> > Hi, devs.
> >
> > I've implemented compact and ordered dictionary [1], which PyPy
> > implemented in 2015 [2].
> >
> > Since it is my first large patch, I would like to have enough time for
> > review cycle by Python 3.6 beta1.
> >
> > Could someone review it?
> >
> > [1] http://bugs.python.org/issue27350
> > [2] https://morepypy.blogspot.jp/2015/01/faster-more-memory-
> efficient-and-more.html
> >
> > --
> > INADA Naoki  <songofacandy at gmail.com>
>
> --
> INADA Naoki  <songofacandy at gmail.com>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160827/d02df57a/attachment.html>

From songofacandy at gmail.com  Sun Aug 28 03:16:21 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Sun, 28 Aug 2016 16:16:21 +0900
Subject: [Python-Dev] Review request: issue 27350, compact ordered dict
In-Reply-To: <CAP7+vJJNq4+SFcg77bi-qHiP9at16BxMd7DNzOOs7A49YmPJ9g@mail.gmail.com>
References: <CAEfz+TyUj8=ePQAEmd8xTb1jg1dSpJa5hbWUFZJAXkgFgN3sOg@mail.gmail.com>
 <CAEfz+TzRW1xjMPJCeMeW7-1EutFguOQ8LPmebBd6hF+rNsbnFg@mail.gmail.com>
 <CAP7+vJJNq4+SFcg77bi-qHiP9at16BxMd7DNzOOs7A49YmPJ9g@mail.gmail.com>
Message-ID: <CAEfz+TzHWmGkGuVHjsrW1YV6kHYmHnRF1w43jwmxj77oL0QUxA@mail.gmail.com>

On Sun, Aug 28, 2016 at 2:05 PM, Guido van Rossum <guido at python.org> wrote:
> Hopefully some core dev(s) can work on this during the core sprint, which is
> from Sept 5-9.
>

OK. While I'm in Japan (UTC+9) and cannot join the sprint, I'll be
active as possible
while the sprint.

Thank you!


>
> --
> --Guido van Rossum (python.org/~guido)

-- 
INADA Naoki  <songofacandy at gmail.com>

From ncoghlan at gmail.com  Sun Aug 28 12:26:59 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Aug 2016 02:26:59 +1000
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <npt414$mdm$1@blaine.gmane.org>
References: <npt414$mdm$1@blaine.gmane.org>
Message-ID: <CADiSq7f+kAnouZjn9e0vGdzzVM9Y=jfznPuuVHrHk5BPqAtFOA@mail.gmail.com>

On 28 August 2016 at 08:25, Terry Reedy <tjreedy at udel.edu> wrote:
> Slicing can be made to malfunction and even crash with an 'evil' __index__
> method. https://bugs.python.org/issue27867
>
> The crux of the problem is this: PySlice_GetIndicesEx
> receives a slice object and a sequence length.  Calling __index__ on the
> start, stop, and step components can mutate the sequence and invalidate the
> length.  Adjusting the int values of start and stop according to an invalid
> length (in particular, one that is too long) will result in invalid results
> or a crash.
>
> Possible actions -- very briefly.  For more see end of
> https://bugs.python.org/issue27867?@ok_message=msg 273801
> 0. Do nothing.
> 1. Detect length change and raise.

I suggest taking this path - it's the lowest impact, and akin to the
"dictionary changed size during iteration" runtime error.

__index__ having side effects is pathological code behaviour, so we
really just need to prevent the interpreter crash, rather than trying
to make it sense of it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tritium-list at sdamon.com  Sun Aug 28 16:39:33 2016
From: tritium-list at sdamon.com (tritium-list at sdamon.com)
Date: Sun, 28 Aug 2016 16:39:33 -0400
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
 <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
 <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>
 <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>
Message-ID: <04df01d2016c$4918e630$db4ab290$@hotmail.com>



> -----Original Message-----
> From: Python-Dev [mailto:python-dev-bounces+tritium-
> list=sdamon.com at python.org] On Behalf Of Steve Dower
> Sent: Wednesday, August 24, 2016 11:44 AM
> To: Stephen J. Turnbull <turnbull.stephen.fw at u.tsukuba.ac.jp>
> Cc: Nick Coghlan <ncoghlan at gmail.com>; Python Dev <python-
> dev at python.org>
> Subject: Re: [Python-Dev] File system path encoding on Windows
> 
> On 23Aug2016 2150, Stephen J. Turnbull wrote:
> > Steve Dower writes:
> >
> >  > * Stephen sees "no reason not to change
locale.getpreferredencoding()"
> >  > (default encoding for open()) at the same time with the same
switches,
> >  > while I'm not quite as confident. Do users generally specify an
encoding
> >  > these days? I know I always put utf-8 there.
> >
> > I was insufficiently specific.  "No reason not to" depends on separate
> > switches for file system encoding and preferred encoding.  That makes
> > things somewhat more complicated for implementation, and significantly
> > so for users.
> 
> Yes, it does, but it's about the only possible migration path.
> 
> I know Nick and Victor like the idea of a -X flag (or a direct -utf8
> flag), but I prefer more specific environment variables:
> 
> - PYTHONWINDOWSLEGACYSTDIO (for the console changes)
> - PYTHONWINDOWSLEGACYPATHENCODING (assuming
> getfilesystemencoding() is utf8)
> - PYTHONWINDOWSLEGACYLOCALEENCODING (assuming
> getpreferredencoding() is
> utf8)

Once you get to var lengths like that, arcane single character flags start
looking preferable.  How about "PYTHONWINLEGACY" to just turn it all on or
off.  If the code breaks on one thing, it obviously isn't written to use the
other two, so might as well shut them all off.


> I'm open to dropping "WINDOWS" from these if others don't think it's
> necessary. Swap "LEGACY" for "UNICODE" if we just offer the option to
> upgrade now rather than changing the default.
> 
> (I could also see the possibility of PYTHONWINDOWSSTRICT* options to use
> the default encoding but raise if decoding bytes fails - mbcs:strict
> rather than mbcs:replace. For utf-8 mode I'd want to use surrogatepass
> throughout, so it will always raise on invalid encoding, but there will
> be far fewer occurrences than for mbcs.)
> 
> I'll transform my earlier post into a PEP (or probably three PEPs - one
> for each change), since it seems like the paper trail is going to be
> more valuable than discussing it now. Without an actual build to test
> it's pretty hard to evaluate the impact.
> 
> Cheers,
> Steve
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-
> list%40sdamon.com


From christian at python.org  Sun Aug 28 16:40:11 2016
From: christian at python.org (Christian Heimes)
Date: Sun, 28 Aug 2016 22:40:11 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
Message-ID: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>

Hi,

we need to talk about OpenSSL and LibreSSL before the next release of
Python. I'm working on a PEP. Most likely it won't be ready before the
feature freeze.

But first let me start with some good news. OpenSSL 1.1 was released a
couple of days ago. It changed a lot of aspects of its internal API,
e.g. all structs are opaque and must be allocated / freed with OpenSSL
API calls. Since I have been tracking changes in OpenSSL for the last
half year and have submitted a couple of patches to OpenSSL, we are in a
good shape. My patch https://bugs.python.org/issue26470 makes Python 2
and 3 compatible with OpenSSL 0.9.8 to 1.1.0 and with LibreSSL, too. It
needs to go through review, though. I have asked Alex to verify my patch.

Now to the bad news. The SSL module is a mess. It looks like a junk room
owned by collector of ancient OpenSSL versions. For example it contains
version checks for OpenSSL 0.9.5 -- which was decommissioned in 2000!
That pre-dates new style classes!

I like to reduce the maintenance burden and list of supported OpenSSL
versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
will reach EOL by the end of this year,
https://www.openssl.org/policies/releasestrat.html . However OpenSSL
0.9.8 is still required for some platforms (OSX).


Here is the deal for 2.7 to 3.5:

1) All versions older than 0.9.8 are completely out-of-scope and no
longer supported.

2) 0.9.8 is semi-support. Python will still compile and work with 0.9.8.
However we do NOT promise that is secure to run 0.9.8. We also require a
recent version. Patch level 0.9.8zc from October 2014 is reasonable
because it comes with SCSV fallback (CVE-2014-3566).

3) 1.0.0 is irrelevant. Users are either stuck on 0.9.8 or are able to
upgrade to 1.0.1+. Let's not support it.

4) 1.0.1 is discouraged but still supported until its EOL.

5) 1.0.2 is the recommend version.

6) 1.1 support will be added by #26470 soon.

7) LibreSSL 2.3 is supported but with a slightly limited feature set.
LibreSSL removed some features like SSL_CERT_FILE and OPENSSL_CONF env vars.


For upcoming 3.6 I would like to limit support to 1.0.2+ and require
1.0.2 features for 3.7. What is the status of Python.org's OSX builds?
Is it possible to drop 0.9.8?

Christian

From benjamin at python.org  Sun Aug 28 19:06:37 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 28 Aug 2016 16:06:37 -0700
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
Message-ID: <1472425597.2349179.708635705.4A77A4A6@webmail.messagingengine.com>



On Sun, Aug 28, 2016, at 13:40, Christian Heimes wrote:
> Here is the deal for 2.7 to 3.5:
> 
> 1) All versions older than 0.9.8 are completely out-of-scope and no
> longer supported.

+1

> 
> 2) 0.9.8 is semi-support. Python will still compile and work with 0.9.8.
> However we do NOT promise that is secure to run 0.9.8. We also require a
> recent version. Patch level 0.9.8zc from October 2014 is reasonable
> because it comes with SCSV fallback (CVE-2014-3566).

I think we should support 0.9.8 for 2.7 and drop it for 3.6.

> 
> 3) 1.0.0 is irrelevant. Users are either stuck on 0.9.8 or are able to
> upgrade to 1.0.1+. Let's not support it.
> 
> 4) 1.0.1 is discouraged but still supported until its EOL.
> 
> 5) 1.0.2 is the recommend version.
> 
> 6) 1.1 support will be added by #26470 soon.

Thanks for writing this patch!

From cory at lukasa.co.uk  Sun Aug 28 20:01:41 2016
From: cory at lukasa.co.uk (Cory Benfield)
Date: Sun, 28 Aug 2016 20:01:41 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
Message-ID: <6DA21CB5-BC27-439B-8A3F-5B2BFBA4BF96@lukasa.co.uk>


> On 28 Aug 2016, at 16:40, Christian Heimes <christian at python.org> wrote:
> 
> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> 1.0.2 features for 3.7. What is the status of Python.org's OSX builds?
> Is it possible to drop 0.9.8?

I strongly support this change. Python with old OpenSSL versions is an enormous source of difficult-to-debug problems for users attempting to work with the web, and puts our users at totally unnecessary risk. Let?s move on.

Cory


From ncoghlan at gmail.com  Sun Aug 28 22:54:35 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Aug 2016 12:54:35 +1000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <04df01d2016c$4918e630$db4ab290$@hotmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
 <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
 <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>
 <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>
 <04df01d2016c$4918e630$db4ab290$@hotmail.com>
Message-ID: <CADiSq7fsdH4hkuFyi-_0RbMUvtZy6JHZh9MZZSdM206=POq3yw@mail.gmail.com>

On 29 August 2016 at 06:39,  <tritium-list at sdamon.com> wrote:
>> -----Original Message-----
>> From: Python-Dev [mailto:python-dev-bounces+tritium-
>> list=sdamon.com at python.org] On Behalf Of Steve Dower
>> Sent: Wednesday, August 24, 2016 11:44 AM
>> To: Stephen J. Turnbull <turnbull.stephen.fw at u.tsukuba.ac.jp>
>> Cc: Nick Coghlan <ncoghlan at gmail.com>; Python Dev <python-
>> dev at python.org>
>> Subject: Re: [Python-Dev] File system path encoding on Windows
>>
>> On 23Aug2016 2150, Stephen J. Turnbull wrote:
>> > Steve Dower writes:
>> >
>> >  > * Stephen sees "no reason not to change
> locale.getpreferredencoding()"
>> >  > (default encoding for open()) at the same time with the same
> switches,
>> >  > while I'm not quite as confident. Do users generally specify an
> encoding
>> >  > these days? I know I always put utf-8 there.
>> >
>> > I was insufficiently specific.  "No reason not to" depends on separate
>> > switches for file system encoding and preferred encoding.  That makes
>> > things somewhat more complicated for implementation, and significantly
>> > so for users.
>>
>> Yes, it does, but it's about the only possible migration path.
>>
>> I know Nick and Victor like the idea of a -X flag (or a direct -utf8
>> flag),

Command line flags and environment variables aren't mutually exclusive
- sometimes you have easy access to add command line arguments,
sometimes you have easy access to set environment variables, sometimes
you have equally easy access to both.

The idea of a "-X" flag is to have an easy way to try out new default
settings that say "Ignore the nominal default encoding and just assume
UTF-8 everywhere", such that folks can get Python behaving that way,
even if they haven't quite figured out how to configure their
operating system itself to use those defaults (this is particularly
relevant for non-systemd based Linux systems, as other init systems
generally don't have the ability to universally override the default
POSIX locale, but even systemd based systems can still do the wrong
thing if "LANG=C" is explicitly specified instead of "LANG=C.UTF-8",
if the particular distro in use doesn't have the latter locale
available, or if a client's locale settings have been forwarded to a
server SSH session)

>> but I prefer more specific environment variables:
>>
>> - PYTHONWINDOWSLEGACYSTDIO (for the console changes)
>> - PYTHONWINDOWSLEGACYPATHENCODING (assuming
>> getfilesystemencoding() is utf8)
>> - PYTHONWINDOWSLEGACYLOCALEENCODING (assuming
>> getpreferredencoding() is
>> utf8)
>
> Once you get to var lengths like that, arcane single character flags start
> looking preferable.  How about "PYTHONWINLEGACY" to just turn it all on or
> off.  If the code breaks on one thing, it obviously isn't written to use the
> other two, so might as well shut them all off.

+1 for a single global on-off switch that means we (and everyone else)
only have two configurations to test (the old way and the new way),
rather than a combinatorial explosion of 8 or more possible settings.
If we get demand for more configurability, *then* we can increase the
complexity, but we shouldn't inflict that level of QA pain on
ourselves in the absence of vocal user demand.

If the more fine-grained settings are considered useful for debugging
purposes, then they can be added with underscore prefixes and
documented accordingly (i.e. as a way of figuring out precisely which
aspect of the new defaults is causing a problem, rather than as a
permanently supported variant configuration).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Aug 28 23:05:48 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Aug 2016 13:05:48 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
Message-ID: <CADiSq7e7qEOgNJiZxGeUN1a7FYCMRxC5KoNE1_tANBO35XgZmg@mail.gmail.com>

On 29 August 2016 at 06:40, Christian Heimes <christian at python.org> wrote:
> Hi,
>
> we need to talk about OpenSSL and LibreSSL before the next release of
> Python. I'm working on a PEP. Most likely it won't be ready before the
> feature freeze.

If it's just drafting work that you need help with on that front, feel
free to send me what you have and I can work it up into PEP form so
folks can see a consolidated list of the proposed changes.

> I like to reduce the maintenance burden and list of supported OpenSSL
> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
> will reach EOL by the end of this year,
> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> 0.9.8 is still required for some platforms (OSX).

Back when I wrote PEP 466, Ned indicated he was in favour of switching
to static linking for the Mac OS X installers:
https://mail.python.org/pipermail/python-dev/2014-March/133347.html

So for 3.6, I agree with Benjamin's suggestion that we drop 0.9.8
support as well.

For 2.7, I think we should defer the decision on what to do to a
follow-up to PEP 466 that resyncs 2.7 with the Python 3.6 network
security stack (while 466 got 2.7 to parity with 3.4.3, even that's
starting to show its age now)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From nad at python.org  Sun Aug 28 22:38:20 2016
From: nad at python.org (Ned Deily)
Date: Sun, 28 Aug 2016 22:38:20 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <1472425597.2349179.708635705.4A77A4A6@webmail.messagingengine.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <1472425597.2349179.708635705.4A77A4A6@webmail.messagingengine.com>
Message-ID: <2EFA23B2-B60C-415A-84CA-405F601FEC62@python.org>

On Aug 28, 2016, at 19:06, Benjamin Peterson <benjamin at python.org> wrote:
> On Sun, Aug 28, 2016, at 13:40, Christian Heimes wrote:
>> Here is the deal for 2.7 to 3.5:
>> 
>> 1) All versions older than 0.9.8 are completely out-of-scope and no
>> longer supported.
> +1
>> 2) 0.9.8 is semi-support. Python will still compile and work with 0.9.8.
>> However we do NOT promise that is secure to run 0.9.8. We also require a
>> recent version. Patch level 0.9.8zc from October 2014 is reasonable
>> because it comes with SCSV fallback (CVE-2014-3566).
> I think we should support 0.9.8 for 2.7 and drop it for 3.6.

Sounds good to me, too.  I think we should also not change things for 3.5.x at this point, e.g. continue to support 0.9.8 there.

>> 3) 1.0.0 is irrelevant. Users are either stuck on 0.9.8 or are able to
>> upgrade to 1.0.1+. Let's not support it.
>> 
>> 4) 1.0.1 is discouraged but still supported until its EOL.
>> 
>> 5) 1.0.2 is the recommend version.
>> 
>> 6) 1.1 support will be added by #26470 soon.

[...]

>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>> 1.0.2 features for 3.7.

It's not clear to me what you are proposing as the differences between 3.6 ("limit support to 1.0.2+") and 3.7 ("require 1.0.2 features").  Could you elaborate?

>> What is the status of Python.org's OSX builds?
>> Is it possible to drop 0.9.8?

I think we can safely drop 0.9.8 support in 3.6.  If anyone is aware of any supported platform where this will would cause a problem, please speak up now.

With regard to OS X (or macOS, as the upcoming next major release is called), the 3.6.0 python.org OS X installer will supply a private copy of OpenSSL 1.0.2+.  Most other third-party distributors of Python on OS X already do not use the Apple-suplied deprecated system OpenSSL libs.  As of the current OS X 10.11 El Capitan, Apple no longer supplies the header files for OpenSSL in either Xcode macosx SDK or in the optional Command Line Tools /usr/include headers so, if you want to build Python on OS X, you now need to use a non-system copy of OpenSSL anyway (the devguide explains how to build with OpenSSL libs from either Homebrew or MacPorts).  The shared libs are still supplied for the benefit of applications built on older releases and for the Apple-supplied system Pythons (2.6.x and 2.7.x, still no 3.x).

> Thanks for writing this patch!

Ditto!

--
  Ned Deily
  nad at python.org -- []


From turnbull.stephen.fw at u.tsukuba.ac.jp  Sun Aug 28 23:43:47 2016
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Mon, 29 Aug 2016 12:43:47 +0900
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <04df01d2016c$4918e630$db4ab290$@hotmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
 <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
 <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>
 <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>
 <04df01d2016c$4918e630$db4ab290$@hotmail.com>
Message-ID: <22467.44915.742948.292725@turnbull.sk.tsukuba.ac.jp>

tritium-list at sdamon.com writes:

 > Once you get to var lengths like that, arcane single character flags start
 > looking preferable.  How about "PYTHONWINLEGACY" to just turn it all on or
 > off.  If the code breaks on one thing, it obviously isn't written to use the
 > other two, so might as well shut them all off.

Since Steve is thinking about three separate PEPs (among other things,
they might be implemented on different timelines), that's not really
possible (placing the features under control of one switch at
different times would be an unacceptable compatibility break).

Anyway, it's not *obvious* that your premise is true, because code
isn't written to do any of those things.  It's written to process
bytes agnostically.  The question is what does the environment look
like.  Steve obviously has a perspective on environment which suggests
that these aspects are often decoupled because in Windows the actual
filesystem is never bytes-oriented.  I don't know if it's possible to
construct a coherent environment where these aspects are decoupled,
but I can't say it's impossible, either.


From christian at python.org  Mon Aug 29 01:42:05 2016
From: christian at python.org (Christian Heimes)
Date: Mon, 29 Aug 2016 07:42:05 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <2EFA23B2-B60C-415A-84CA-405F601FEC62@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <1472425597.2349179.708635705.4A77A4A6@webmail.messagingengine.com>
 <2EFA23B2-B60C-415A-84CA-405F601FEC62@python.org>
Message-ID: <1981f645-eb17-5088-cb65-f91d0c4d9fd8@python.org>

On 2016-08-29 04:38, Ned Deily wrote:
> On Aug 28, 2016, at 19:06, Benjamin Peterson <benjamin at python.org> wrote:
>> On Sun, Aug 28, 2016, at 13:40, Christian Heimes wrote:
>>> Here is the deal for 2.7 to 3.5:
>>>
>>> 1) All versions older than 0.9.8 are completely out-of-scope and no
>>> longer supported.
>> +1
>>> 2) 0.9.8 is semi-support. Python will still compile and work with 0.9.8.
>>> However we do NOT promise that is secure to run 0.9.8. We also require a
>>> recent version. Patch level 0.9.8zc from October 2014 is reasonable
>>> because it comes with SCSV fallback (CVE-2014-3566).
>> I think we should support 0.9.8 for 2.7 and drop it for 3.6.
> 
> Sounds good to me, too.  I think we should also not change things for 3.5.x at this point, e.g. continue to support 0.9.8 there.


In my proto-PEP I'm talking about different levels of support: full,
build-only and unsupported. Full support means that the combination of
Python and OpenSSL versions is reasonable secure and recommended.

On the other hand build-only support doesn't come with any security
promise. The ssl and hashlib module are source compatible with OpenSSL
0.9.8. You can still compile Python, do https connections but they might
not be secure. It's "Warranty void" mode.

>>> 3) 1.0.0 is irrelevant. Users are either stuck on 0.9.8 or are able to
>>> upgrade to 1.0.1+. Let's not support it.
>>>
>>> 4) 1.0.1 is discouraged but still supported until its EOL.
>>>
>>> 5) 1.0.2 is the recommend version.
>>>
>>> 6) 1.1 support will be added by #26470 soon.
> 
> [...]
> 
>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>> 1.0.2 features for 3.7.
> 
> It's not clear to me what you are proposing as the differences between 3.6 ("limit support to 1.0.2+") and 3.7 ("require 1.0.2 features").  Could you elaborate?

For 3.6 I don't require any 1.0.2 feature yet. The 1.1.0 patch keeps
code compatible with 0.9.8zc to 1.1.0. But as soon as I use new
features, the ssl module will no longer be source and build compatible
with < 1.0.2. There is also the point of OpenSSL 1.0.1. It reaches
end-of-lifetime by the end if this year. 1.0.2 will be supported until 2019.

I'm tempted to require 1.0.2 for Python 3.6 but it's technically not
necessary yet.

#if OPENSSL_VERSION_INFO < 0x01000200L
#  error "OpenSSL 1.0.2+ required"
#endif


> 
>>> What is the status of Python.org's OSX builds?
>>> Is it possible to drop 0.9.8?
> 
> I think we can safely drop 0.9.8 support in 3.6.  If anyone is aware of any supported platform where this will would cause a problem, please speak up now.
> 
> With regard to OS X (or macOS, as the upcoming next major release is called), the 3.6.0 python.org OS X installer will supply a private copy of OpenSSL 1.0.2+.  Most other third-party distributors of Python on OS X already do not use the Apple-suplied deprecated system OpenSSL libs.  As of the current OS X 10.11 El Capitan, Apple no longer supplies the header files for OpenSSL in either Xcode macosx SDK or in the optional Command Line Tools /usr/include headers so, if you want to build Python on OS X, you now need to use a non-system copy of OpenSSL anyway (the devguide explains how to build with OpenSSL libs from either Homebrew or MacPorts).  The shared libs are still supplied for the benefit of applications built on older releases and for the Apple-supplied system Pythons (2.6.x and 2.7
>  .x, still no 3.x).

I'm glad to hear that. Thanks :)

Christian

From mal at egenix.com  Mon Aug 29 04:09:20 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 29 Aug 2016 10:09:20 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
Message-ID: <57C3EDB0.9030305@egenix.com>

On 28.08.2016 22:40, Christian Heimes wrote:
> ...
> I like to reduce the maintenance burden and list of supported OpenSSL
> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
> will reach EOL by the end of this year,
> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> 0.9.8 is still required for some platforms (OSX).
> ...
> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> 1.0.2 features for 3.7.
> ...

Hmm, that last part would mean that Python 3.7 will no longer compile
on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
Since 14.04 LTS is supported until 2019, I think it would be better
to only start requiring 1.0.2 in Python 3.8.

BTW: Are there any features in 1.0.2 that we need and would warrant
dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From christian at python.org  Mon Aug 29 04:24:42 2016
From: christian at python.org (Christian Heimes)
Date: Mon, 29 Aug 2016 10:24:42 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C3EDB0.9030305@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
Message-ID: <0c933915-3f6b-5907-b4ae-ed89d214cf74@python.org>

On 2016-08-29 10:09, M.-A. Lemburg wrote:
> On 28.08.2016 22:40, Christian Heimes wrote:
>> ...
>> I like to reduce the maintenance burden and list of supported OpenSSL
>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>> will reach EOL by the end of this year,
>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>> 0.9.8 is still required for some platforms (OSX).
>> ...
>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>> 1.0.2 features for 3.7.
>> ...
> 
> Hmm, that last part would mean that Python 3.7 will no longer compile
> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> Since 14.04 LTS is supported until 2019, I think it would be better
> to only start requiring 1.0.2 in Python 3.8.

No, LTS support should not be our concern. If you need a brand new
version of Python on an old LTS or Enterprise version of your OS, please
contact your vendor and buy support. You don't get to run old metal and
play with shiny new toys at the same time for free.

By the way I knew that something like this would come up from you. Thank
you that you satisfied my expectation. :p

> BTW: Are there any features in 1.0.2 that we need and would warrant
> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?

Yes, there are features I want to use, e.g. proper hostname
verification. Python's post-handshake verification is a hack and leads
to information disclosure.

Christian

From stefan at bytereef.org  Mon Aug 29 05:06:14 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Mon, 29 Aug 2016 11:06:14 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <0c933915-3f6b-5907-b4ae-ed89d214cf74@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <0c933915-3f6b-5907-b4ae-ed89d214cf74@python.org>
Message-ID: <20160829090614.GA4461@bytereef.org>

On Mon, Aug 29, 2016 at 10:24:42AM +0200, Christian Heimes wrote:
> On 2016-08-29 10:09, M.-A. Lemburg wrote:
> > On 28.08.2016 22:40, Christian Heimes wrote:
> >> ...
> >> I like to reduce the maintenance burden and list of supported OpenSSL
> >> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
> >> will reach EOL by the end of this year,
> >> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> >> 0.9.8 is still required for some platforms (OSX).
> >> ...
> >> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> >> 1.0.2 features for 3.7.
> >> ...
> > 
> > Hmm, that last part would mean that Python 3.7 will no longer compile
> > on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> > Since 14.04 LTS is supported until 2019, I think it would be better
> > to only start requiring 1.0.2 in Python 3.8.
> 
> No, LTS support should not be our concern. If you need a brand new
> version of Python on an old LTS or Enterprise version of your OS, please
> contact your vendor and buy support. You don't get to run old metal and
> play with shiny new toys at the same time for free.

Well, it's an additional annoyance for sure:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:        14.04
Codename:       trusty


I'm not in the habit of updating my OS constantly.

[Before this attracts some of the usual lectures, I know how to install
OpenSSL, thanks.]


Stefan Krah





From rosuav at gmail.com  Mon Aug 29 05:14:33 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 29 Aug 2016 19:14:33 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <0c933915-3f6b-5907-b4ae-ed89d214cf74@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <0c933915-3f6b-5907-b4ae-ed89d214cf74@python.org>
Message-ID: <CAPTjJmrGOdy2K7ZcD=4b0h33qtbpfKHuwN5YgZ8mi6gFsZ1OGw@mail.gmail.com>

On Mon, Aug 29, 2016 at 6:24 PM, Christian Heimes <christian at python.org> wrote:
> No, LTS support should not be our concern. If you need a brand new
> version of Python on an old LTS or Enterprise version of your OS, please
> contact your vendor and buy support. You don't get to run old metal and
> play with shiny new toys at the same time for free.

I think I agree with you, but I'm not fully convinced. The current
stable Debian (Jessie) also ships OpenSSL 1.0.1, although 1.0.2 is
available in backports. The next stable release (Stretch) is expected
some time 2017. That's not the same as "running an old LTS" - that's
running the latest non-dev release. Is it acceptable to tell people to
install a library from backports if they want to compile the latest
Python?

ChrisA

From ncoghlan at gmail.com  Mon Aug 29 06:52:08 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Aug 2016 20:52:08 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CAPTjJmrGOdy2K7ZcD=4b0h33qtbpfKHuwN5YgZ8mi6gFsZ1OGw@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <0c933915-3f6b-5907-b4ae-ed89d214cf74@python.org>
 <CAPTjJmrGOdy2K7ZcD=4b0h33qtbpfKHuwN5YgZ8mi6gFsZ1OGw@mail.gmail.com>
Message-ID: <CADiSq7ePoHJuVokuRG4TRX-T_COJ1Vg7f6Yfg+7Kg8HTqkst0g@mail.gmail.com>

On 29 August 2016 at 19:14, Chris Angelico <rosuav at gmail.com> wrote:
> On Mon, Aug 29, 2016 at 6:24 PM, Christian Heimes <christian at python.org> wrote:
>> No, LTS support should not be our concern. If you need a brand new
>> version of Python on an old LTS or Enterprise version of your OS, please
>> contact your vendor and buy support. You don't get to run old metal and
>> play with shiny new toys at the same time for free.
>
> I think I agree with you, but I'm not fully convinced. The current
> stable Debian (Jessie) also ships OpenSSL 1.0.1, although 1.0.2 is
> available in backports. The next stable release (Stretch) is expected
> some time 2017.

I agree keeping build compatibility with the latest Debian stable
release is a sensible baseline, but 3.7 won't land until some time in
2018, so Christian's proposed timeline still works on that front (but
jumping straight to requiring 1.0.2 in 3.6 wouldn't, suggesting that
supporting the latest Debian stable release will be a genuinely useful
guideline in helping to make decisions about minimum supported OpenSSL
versions for new CPython releases).

Assuming the checks for required OpenSSL features are able to be
implemented as feature guards rather than as a plain version check
(e.g. by issuing a warning if the detected SSL version is too low, but
otherwise letting the build continue), other distros will also be able
to keep new CPython versions building just by backporting the required
OpenSSL features.

More generally, it's important for folks to keep in mind that where
most commercial Linux redistributors invest directly in upstream
maintenance by assigning people to work on the kernel and other low
level infrastructure pieces full time (including the community distros
where those components are most actively developed), Python's
commercial redistributors aren't as well behaved, so we currently have
just the one full time developer working on PyPI and related PyPA
tooling, and a few folks that have successfully negotiated part-time
upstream CPython involvement as a condition of their employment.

That disparity means we have to start getting much better at
offloading security-sensitive work from CPython's volunteers to paid
Linux developers in cases where it's feasible for us to do so, even if
that in turn means requiring that Linux distros (both community and
commercial) pay much closer attention to keeping their network
security libraries up to date if they want their users to be readily
able to build and run the latest Python feature releases.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rosuav at gmail.com  Mon Aug 29 07:05:05 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 29 Aug 2016 21:05:05 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CADiSq7ePoHJuVokuRG4TRX-T_COJ1Vg7f6Yfg+7Kg8HTqkst0g@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <0c933915-3f6b-5907-b4ae-ed89d214cf74@python.org>
 <CAPTjJmrGOdy2K7ZcD=4b0h33qtbpfKHuwN5YgZ8mi6gFsZ1OGw@mail.gmail.com>
 <CADiSq7ePoHJuVokuRG4TRX-T_COJ1Vg7f6Yfg+7Kg8HTqkst0g@mail.gmail.com>
Message-ID: <CAPTjJmqrowUU+KyMgO4KjFHGoVvv-ViZZB=HLXqWwM2ts87rtA@mail.gmail.com>

On Mon, Aug 29, 2016 at 8:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 29 August 2016 at 19:14, Chris Angelico <rosuav at gmail.com> wrote:
>> On Mon, Aug 29, 2016 at 6:24 PM, Christian Heimes <christian at python.org> wrote:
>>> No, LTS support should not be our concern. If you need a brand new
>>> version of Python on an old LTS or Enterprise version of your OS, please
>>> contact your vendor and buy support. You don't get to run old metal and
>>> play with shiny new toys at the same time for free.
>>
>> I think I agree with you, but I'm not fully convinced. The current
>> stable Debian (Jessie) also ships OpenSSL 1.0.1, although 1.0.2 is
>> available in backports. The next stable release (Stretch) is expected
>> some time 2017.
>
> I agree keeping build compatibility with the latest Debian stable
> release is a sensible baseline, but 3.7 won't land until some time in
> 2018, so Christian's proposed timeline still works on that front (but
> jumping straight to requiring 1.0.2 in 3.6 wouldn't, suggesting that
> supporting the latest Debian stable release will be a genuinely useful
> guideline in helping to make decisions about minimum supported OpenSSL
> versions for new CPython releases).

Cool. I may have slightly misunderstood this from the OP:

> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> 1.0.2 features for 3.7.

What does "limit support" mean? Will it be possible to build CPython
3.6 against OpenSSL 1.0.1?

For 3.7, sure. Once Stretch has been stable for a year, it's not
unreasonable to declare that the newest Python won't run on oldstable.
Just not so sure about 3.6, where Stretch won't be stable yet. But
even there, Jessie has been stable since Apr 2015 (so, 18 months or so
by the time Python 3.6 comes out - that's one entire Python release),
so I could well understand not wanting to be shackled by it. Saying
"the latest CPython demands that you enable backports in your apt
sources.list" wouldn't be unreasonable.

ChrisA

From ncoghlan at gmail.com  Mon Aug 29 07:16:47 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Aug 2016 21:16:47 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CAPTjJmqrowUU+KyMgO4KjFHGoVvv-ViZZB=HLXqWwM2ts87rtA@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <0c933915-3f6b-5907-b4ae-ed89d214cf74@python.org>
 <CAPTjJmrGOdy2K7ZcD=4b0h33qtbpfKHuwN5YgZ8mi6gFsZ1OGw@mail.gmail.com>
 <CADiSq7ePoHJuVokuRG4TRX-T_COJ1Vg7f6Yfg+7Kg8HTqkst0g@mail.gmail.com>
 <CAPTjJmqrowUU+KyMgO4KjFHGoVvv-ViZZB=HLXqWwM2ts87rtA@mail.gmail.com>
Message-ID: <CADiSq7e8Y=QA6Vr3cVpFwQnxonNgK07B3H-r=PJ36jmUU2m9dQ@mail.gmail.com>

On 29 August 2016 at 21:05, Chris Angelico <rosuav at gmail.com> wrote:
> On Mon, Aug 29, 2016 at 8:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>> 1.0.2 features for 3.7.
>
> What does "limit support" mean? Will it be possible to build CPython
> 3.6 against OpenSSL 1.0.1?

Christian clarified this later in the thread:

- full support is stating confidently that software running that way
is using network connections that are as secure as we know how to make
them
- build support is ensuring it builds, without vouching one way or the
other for the security of the resulting network connections
- no support is "it doesn't build, but even if it did, we wouldn't
vouch for the security of the resulting connections"

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rosuav at gmail.com  Mon Aug 29 07:48:43 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 29 Aug 2016 21:48:43 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CADiSq7e8Y=QA6Vr3cVpFwQnxonNgK07B3H-r=PJ36jmUU2m9dQ@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <0c933915-3f6b-5907-b4ae-ed89d214cf74@python.org>
 <CAPTjJmrGOdy2K7ZcD=4b0h33qtbpfKHuwN5YgZ8mi6gFsZ1OGw@mail.gmail.com>
 <CADiSq7ePoHJuVokuRG4TRX-T_COJ1Vg7f6Yfg+7Kg8HTqkst0g@mail.gmail.com>
 <CAPTjJmqrowUU+KyMgO4KjFHGoVvv-ViZZB=HLXqWwM2ts87rtA@mail.gmail.com>
 <CADiSq7e8Y=QA6Vr3cVpFwQnxonNgK07B3H-r=PJ36jmUU2m9dQ@mail.gmail.com>
Message-ID: <CAPTjJmq2rNd7tc=zM2756LiOkQjT9GgtZ3NP6_RCSYzk_KyeTg@mail.gmail.com>

On Mon, Aug 29, 2016 at 9:16 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 29 August 2016 at 21:05, Chris Angelico <rosuav at gmail.com> wrote:
>> On Mon, Aug 29, 2016 at 8:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>> 1.0.2 features for 3.7.
>>
>> What does "limit support" mean? Will it be possible to build CPython
>> 3.6 against OpenSSL 1.0.1?
>
> Christian clarified this later in the thread:
>
> - full support is stating confidently that software running that way
> is using network connections that are as secure as we know how to make
> them
> - build support is ensuring it builds, without vouching one way or the
> other for the security of the resulting network connections
> - no support is "it doesn't build, but even if it did, we wouldn't
> vouch for the security of the resulting connections"

Sorry, my bad for just skimming the thread. There are comments like this:

> I'm tempted to require 1.0.2 for Python 3.6 but it's technically not
> necessary yet.
>
> #if OPENSSL_VERSION_INFO < 0x01000200L
> #  error "OpenSSL 1.0.2+ required"
> #endif

that led me to think that 3.6 was planning to demand 1.0.2, but if the
intention is build support for 1.0.1, that would work.

Sorry for the noise!

ChrisA

From ethan at stoneleaf.us  Mon Aug 29 12:18:00 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 29 Aug 2016 09:18:00 -0700
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <CADiSq7f+kAnouZjn9e0vGdzzVM9Y=jfznPuuVHrHk5BPqAtFOA@mail.gmail.com>
References: <npt414$mdm$1@blaine.gmane.org>
 <CADiSq7f+kAnouZjn9e0vGdzzVM9Y=jfznPuuVHrHk5BPqAtFOA@mail.gmail.com>
Message-ID: <57C46038.1080107@stoneleaf.us>

On 08/28/2016 09:26 AM, Nick Coghlan wrote:
> On 28 August 2016 at 08:25, Terry Reedy <tjreedy at udel.edu> wrote:
>> Slicing can be made to malfunction and even crash with an 'evil' __index__
>> method. https://bugs.python.org/issue27867
>>
>> The crux of the problem is this: PySlice_GetIndicesEx
>> receives a slice object and a sequence length.  Calling __index__ on the
>> start, stop, and step components can mutate the sequence and invalidate the
>> length.  Adjusting the int values of start and stop according to an invalid
>> length (in particular, one that is too long) will result in invalid results
>> or a crash.
>>
>> Possible actions -- very briefly.  For more see end of
>> https://bugs.python.org/issue27867?@ok_message=msg 273801
>> 0. Do nothing.
>> 1. Detect length change and raise.
>
> I suggest taking this path - it's the lowest impact, and akin to the
> "dictionary changed size during iteration" runtime error.

+1.  Being able to do such strange things with list but not dict would be irritating and a nuisance (although maybe not attractive ;) .

> __index__ having side effects is pathological code behaviour, so we
> really just need to prevent the interpreter crash, rather than trying
> to make it sense of it.

Agreed.

--
~Ethan~

From steve.dower at python.org  Mon Aug 29 12:30:36 2016
From: steve.dower at python.org (Steve Dower)
Date: Mon, 29 Aug 2016 09:30:36 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <22467.44915.742948.292725@turnbull.sk.tsukuba.ac.jp>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
 <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
 <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>
 <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>
 <04df01d2016c$4918e630$db4ab290$@hotmail.com>
 <22467.44915.742948.292725@turnbull.sk.tsukuba.ac.jp>
Message-ID: <dbd4afa9-6d36-86ac-cb01-2f05fd280569@python.org>

On 28Aug2016 2043, Stephen J. Turnbull wrote:
> tritium-list at sdamon.com writes:
>
>  > Once you get to var lengths like that, arcane single character flags start
>  > looking preferable.  How about "PYTHONWINLEGACY" to just turn it all on or
>  > off.  If the code breaks on one thing, it obviously isn't written to use the
>  > other two, so might as well shut them all off.
>
> Since Steve is thinking about three separate PEPs (among other things,
> they might be implemented on different timelines), that's not really
> possible (placing the features under control of one switch at
> different times would be an unacceptable compatibility break).

Yeah, the likelihood of different timelines basically means three PEPs 
are going to be necessary. But I think we can have a single 
"PYTHONWINDOWSANSI" (or ...MBCS) flag to cover all three whenever they 
come in without it being a compatibility break, especially if (as Nick 
suggested) there are _PYTHONWINDOWSANSI(CONSOLE|PATH|LOCALE) flags too. 
But it does give us the ability to say "all ANSI or all UTF-8 are 
supported; mix-and-match at your own risk".

> Anyway, it's not *obvious* that your premise is true, because code
> isn't written to do any of those things.  It's written to process
> bytes agnostically.  The question is what does the environment look
> like.  Steve obviously has a perspective on environment which suggests
> that these aspects are often decoupled because in Windows the actual
> filesystem is never bytes-oriented.  I don't know if it's possible to
> construct a coherent environment where these aspects are decoupled,
> but I can't say it's impossible, either.

Actually, the three items are basically completely decoupled, though it 
isn't obvious.

* stdin/stdout/stderr are text wrappers by default (under my changes, 
using the console encoding when it's a console and the locale encoding 
when it's a file/pipe). There's no point reading bytes from the console, 
and redirected files or pipes are unaffected by the change.
* the file system encoding only affects paths passed into/returned from 
the OS as bytes, and...
* the locale encoding affects files opened in text mode, which means...
* if you open('rb') and read paths, the locale encoding has no effect on 
whether the bytes are the right encoding to be used as paths

So while there are scenarios that use multiple pieces of this, there 
should only be one change impacting any scenario:
* reading str paths from a file - locale encoding
* reading bytes paths from a file - filesystem encoding
* reading str paths from a pipe/redirected file - locale encoding
* reading bytes paths from a pipe/redirected file - filesystem encoding
* reading str paths from the console - console encoding
* reading bytes paths from the console (i.e. 
sys.stdin.buffer.raw.read()) - filesystem encoding

The last case doesn't make sense anyway right now, as 
sys.stdin.buffer.raw has no specified encoding and you can't reliably 
read paths from it. Perhaps there exist examples of where this is put to 
good use (bearing in mind it must be an actual console - not a 
redirection or pipe) - I would love to hear about them.

As far as I can tell, any other combination requires the Python 
developer to convert between str and bytes themselves, which may lead to 
errors if they have assumed that the encoding of the bytes would never 
change, but code that ignores encodings and uses bytes or str 
exclusively is only going to encounter one (bytes) or two (str) of the 
changes.

Cheers,
Steve


From cory at lukasa.co.uk  Mon Aug 29 12:33:05 2016
From: cory at lukasa.co.uk (Cory Benfield)
Date: Mon, 29 Aug 2016 12:33:05 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C3EDB0.9030305@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
Message-ID: <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>


> On 29 Aug 2016, at 04:09, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> On 28.08.2016 22:40, Christian Heimes wrote:
>> ...
>> I like to reduce the maintenance burden and list of supported OpenSSL
>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>> will reach EOL by the end of this year,
>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>> 0.9.8 is still required for some platforms (OSX).
>> ...
>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>> 1.0.2 features for 3.7.
>> ...
> 
> Hmm, that last part would mean that Python 3.7 will no longer compile
> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> Since 14.04 LTS is supported until 2019, I think it would be better
> to only start requiring 1.0.2 in Python 3.8.

Can someone explain to me why this is a use-case we care about?

The nature of a stable distribution is that all the packages are guaranteed to work together. Once you start compiling your own software, you are out in the cold: you are no longer covered by that guarantee. Why, then, should someone compiling a version of Python that was barely conceived when Ubuntu 14.04 was released expect that they can compile it from source using only dependencies available in their mainline distribution?

I absolutely understand wanting to support Ubuntu 14.04 for any release of Python that already compiles on Ubuntu 14.04: that makes sense. But new releases should not be shackled to what is in LTS releases of operating systems. If it were, a more coherent argument would be that we cannot drop 0.9.8 support in any Python released before the middle of 2017 because RHEL 5 ships it, and we will not be able to require OpenSSL 1.0.2 until almost 2021 because RHEL 6 ships 1.0.1. After all, why does Ubuntu 14.04 get privileged over RHEL? Both are supported LTS releases, after all.

I don?t believe that the Python dev team has ever promised that future versions of Python will compile on a given LTS release. Am I mistaken?

While I?m here, I should note that Ubuntu 14.04 goes EOL in 2019, while OpenSSL 1.0.1 loses support from upstream at the end of this year, which is probably a good reason to stop supporting it in new Python versions. OpenSSL 1.0.0 and 0.9.8 are already unsupported by upstream.

Cory


From brett at python.org  Mon Aug 29 15:29:58 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Aug 2016 19:29:58 +0000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
Message-ID: <CAP1=2W7a56-J=DbCLLkbC2neUj+ySCGwWgxKOA9Nc25+z87-vw@mail.gmail.com>

On Mon, 29 Aug 2016 at 09:34 Cory Benfield <cory at lukasa.co.uk> wrote:

>
> > On 29 Aug 2016, at 04:09, M.-A. Lemburg <mal at egenix.com> wrote:
> >
> > On 28.08.2016 22:40, Christian Heimes wrote:
> >> ...
> >> I like to reduce the maintenance burden and list of supported OpenSSL
> >> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
> >> will reach EOL by the end of this year,
> >> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> >> 0.9.8 is still required for some platforms (OSX).
> >> ...
> >> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> >> 1.0.2 features for 3.7.
> >> ...
> >
> > Hmm, that last part would mean that Python 3.7 will no longer compile
> > on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> > Since 14.04 LTS is supported until 2019, I think it would be better
> > to only start requiring 1.0.2 in Python 3.8.
>
> Can someone explain to me why this is a use-case we care about?
>
> The nature of a stable distribution is that all the packages are
> guaranteed to work together. Once you start compiling your own software,
> you are out in the cold: you are no longer covered by that guarantee. Why,
> then, should someone compiling a version of Python that was barely
> conceived when Ubuntu 14.04 was released expect that they can compile it
> from source using only dependencies available in their mainline
> distribution?
>

I also agree with this view. If you're on an LTS then you should expect
everything that comes with the LTS to work, not new software that you
pulled forward unless your OS distributor supports it, e.g. RHEL
collections.


>
> I absolutely understand wanting to support Ubuntu 14.04 for any release of
> Python that already compiles on Ubuntu 14.04: that makes sense. But new
> releases should not be shackled to what is in LTS releases of operating
> systems. If it were, a more coherent argument would be that we cannot drop
> 0.9.8 support in any Python released before the middle of 2017 because RHEL
> 5 ships it, and we will not be able to require OpenSSL 1.0.2 until almost
> 2021 because RHEL 6 ships 1.0.1. After all, why does Ubuntu 14.04 get
> privileged over RHEL? Both are supported LTS releases, after all.
>

No one gets special privileges beyond someone funding a core dev to put
into the effort to support a platform that won't shackle others.


>
> I don?t believe that the Python dev team has ever promised that future
> versions of Python will compile on a given LTS release. Am I mistaken?
>

No, literally the only support promise we make along these lines is that we
will support Windows versions that are still in extended support:
https://www.python.org/dev/peps/pep-0011/#id10 , and that promise only
works due to Windows not being a moving target in terms of
backwards-compatibility.


>
> While I?m here, I should note that Ubuntu 14.04 goes EOL in 2019, while
> OpenSSL 1.0.1 loses support from upstream at the end of this year, which is
> probably a good reason to stop supporting it in new Python versions.
> OpenSSL 1.0.0 and 0.9.8 are already unsupported by upstream.
>

I agree w/ Christian's proposal of matching what OpenSSL upstream supports
when releasing new versions of CPython. I don't see how supporting outdated
security code is good for anyone involved.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160829/6813b425/attachment.html>

From mal at egenix.com  Mon Aug 29 15:31:07 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 29 Aug 2016 21:31:07 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
Message-ID: <57C48D7B.4080200@egenix.com>

On 29.08.2016 18:33, Cory Benfield wrote:
> 
>> On 29 Aug 2016, at 04:09, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>> On 28.08.2016 22:40, Christian Heimes wrote:
>>> ...
>>> I like to reduce the maintenance burden and list of supported OpenSSL
>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>>> will reach EOL by the end of this year,
>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>>> 0.9.8 is still required for some platforms (OSX).
>>> ...
>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>> 1.0.2 features for 3.7.
>>> ...
>>
>> Hmm, that last part would mean that Python 3.7 will no longer compile
>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>> Since 14.04 LTS is supported until 2019, I think it would be better
>> to only start requiring 1.0.2 in Python 3.8.
> 
> Can someone explain to me why this is a use-case we care about?

Ubuntu 14.04 is a widely deployed system and newer Python version
should run on such widely deployed systems without having to
replace important vendor maintained system libraries such as
OpenSSL.

Python 3.7 starts shipping around June 2018 (assuming the 18 month
release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
be able to use Python 3.7 on such a system, you'd have to upgrade
to a more recent LTS version 10 months before the EOL date (with
all the associated issues) or lose vendor maintenance support and
run with your own copy of OpenSSL.

OTOH, the situations isn't all that bad: people on such systems
will likely not go straight for the 3.7.0 release of Python, but
instead wait for 3.7.2+ for the dust to settle.

> The nature of a stable distribution is that all the packages are guaranteed to work together. Once you start compiling your own software, you are out in the cold: you are no longer covered by that guarantee. Why, then, should someone compiling a version of Python that was barely conceived when Ubuntu 14.04 was released expect that they can compile it from source using only dependencies available in their mainline distribution?
> 
> I absolutely understand wanting to support Ubuntu 14.04 for any release of Python that already compiles on Ubuntu 14.04: that makes sense. But new releases should not be shackled to what is in LTS releases of operating systems. If it were, a more coherent argument would be that we cannot drop 0.9.8 support in any Python released before the middle of 2017 because RHEL 5 ships it, and we will not be able to require OpenSSL 1.0.2 until almost 2021 because RHEL 6 ships 1.0.1. After all, why does Ubuntu 14.04 get privileged over RHEL? Both are supported LTS releases, after all.
> 
> I don?t believe that the Python dev team has ever promised that future versions of Python will compile on a given LTS release. Am I mistaken?
> 
> While I?m here, I should note that Ubuntu 14.04 goes EOL in 2019, while OpenSSL 1.0.1 loses support from upstream at the end of this year, which is probably a good reason to stop supporting it in new Python versions. OpenSSL 1.0.0 and 0.9.8 are already unsupported by upstream.

Sure, but Ubuntu will continue to support OpenSSL 1.0.1
until 2019, backporting important security fixes as necessary and
that's what's important.

It's unfortunate that Python has to rely on a 3rd party library
for security, but we should at least make sure that our users
can rely on OS vendor support to keep the lib up to date with
security fixes.

On 29.08.2016 10:24, Christian Heimes wrote:
> By the way I knew that something like this would come up from you.
> Thank you that you satisfied my expectation. :p

Sure, I want Python to be used on as many systems as possible,
both in terms of architecture and OS. The more the better.
If we don't have to drop support early, why should we ?

>> BTW: Are there any features in 1.0.2 that we need and would warrant
>> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?
>
> Yes, there are features I want to use, e.g. proper hostname
> verification. Python's post-handshake verification is a hack and leads
> to information disclosure.

This doesn't sound like a feature worth breaking compatibility
to me.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From random832 at fastmail.com  Mon Aug 29 16:10:28 2016
From: random832 at fastmail.com (Random832)
Date: Mon, 29 Aug 2016 16:10:28 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C3EDB0.9030305@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
Message-ID: <1472501428.3664165.709582793.107C9983@webmail.messagingengine.com>

On Mon, Aug 29, 2016, at 04:09, M.-A. Lemburg wrote:
> Hmm, that last part would mean that Python 3.7 will no longer compile
> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> Since 14.04 LTS is supported until 2019, I think it would be better
> to only start requiring 1.0.2 in Python 3.8.

If someone's going to compile a new version of Python for Ubuntu LTS,
they can compile a new version of OpenSSL. How likely is it that someone
will want to use the packaged version of OpenSSL, but not use the
packaged version (which is 3.4) of Python?

From christian at python.org  Mon Aug 29 16:16:31 2016
From: christian at python.org (Christian Heimes)
Date: Mon, 29 Aug 2016 22:16:31 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C48D7B.4080200@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
Message-ID: <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>

On 2016-08-29 21:31, M.-A. Lemburg wrote:
> On 29.08.2016 18:33, Cory Benfield wrote:
>>
>>> On 29 Aug 2016, at 04:09, M.-A. Lemburg <mal at egenix.com> wrote:
>>>
>>> On 28.08.2016 22:40, Christian Heimes wrote:
>>>> ...
>>>> I like to reduce the maintenance burden and list of supported OpenSSL
>>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>>>> will reach EOL by the end of this year,
>>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>>>> 0.9.8 is still required for some platforms (OSX).
>>>> ...
>>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>>> 1.0.2 features for 3.7.
>>>> ...
>>>
>>> Hmm, that last part would mean that Python 3.7 will no longer compile
>>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>>> Since 14.04 LTS is supported until 2019, I think it would be better
>>> to only start requiring 1.0.2 in Python 3.8.
>>
>> Can someone explain to me why this is a use-case we care about?
> 
> Ubuntu 14.04 is a widely deployed system and newer Python version
> should run on such widely deployed systems without having to
> replace important vendor maintained system libraries such as
> OpenSSL.

"Widely deployed" is true for a lot of old operating systems including
Windows XP.

> Python 3.7 starts shipping around June 2018 (assuming the 18 month
> release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
> be able to use Python 3.7 on such a system, you'd have to upgrade
> to a more recent LTS version 10 months before the EOL date (with
> all the associated issues) or lose vendor maintenance support and
> run with your own copy of OpenSSL.

Why would you deploy an unsupported Python version on a LTS release? Why
should compatibility be our concern?

> Sure, but Ubuntu will continue to support OpenSSL 1.0.1
> until 2019, backporting important security fixes as necessary and
> that's what's important.

I see an easy solution here: either pay or make Canonical backport all
required features to OpenSSL 1.0.1. </sarcasm>

> It's unfortunate that Python has to rely on a 3rd party library
> for security, but we should at least make sure that our users
> can rely on OS vendor support to keep the lib up to date with
> security fixes.

No, it is a good thing that we can rely on 3rd party libraries for
security. Crypto and security is not our domain. It is incredible hard
to develop and maintain crypto code. Also my proposal enforces OS
vendors to supply up to date OpenSSL versions.

> 
> On 29.08.2016 10:24, Christian Heimes wrote:
>> By the way I knew that something like this would come up from you.
>> Thank you that you satisfied my expectation. :p
> 
> Sure, I want Python to be used on as many systems as possible,
> both in terms of architecture and OS. The more the better.
> If we don't have to drop support early, why should we ?

MAL, I don't like your attitude. It feels like you want me and other
contributors to waste time on this topic. That is not how this
discussion is going to end. If *you* want to keep support for outdated
OpenSSL versions, than it is *your* responsibility and *your* time. You
cannot and will not put this burden on me.

Python is running out of developers with OpenSSL expertise. It's Alex,
Antoine, Benjamin, Victor and me. Antoine and me haven't been active for
a while. Victor and Benjamin are mostly working on other topics. As far
as I can judge Alex, he rather works on PyCA than CPython stdlib.

I'm both interested and willing to improve Python's ssl stack, and I'm
going to do this in my own free time. Yes, I'm working for Red Hat's
security engineering, but I'm not getting paid to work on Python (except
for a couple of hours now and then when a bug is relevant for my daily
work). I will only contribute improvements and fixes on my own terms,
that means I'm not going to waste my time with outdated versions. In my
opinion it is more than reasonable to ditch 1.0.1 and earlier.

> This doesn't sound like a feature worth breaking compatibility
> to me.

It does.

Christian

From christian at python.org  Mon Aug 29 16:19:57 2016
From: christian at python.org (Christian Heimes)
Date: Mon, 29 Aug 2016 22:19:57 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <1472501428.3664165.709582793.107C9983@webmail.messagingengine.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <1472501428.3664165.709582793.107C9983@webmail.messagingengine.com>
Message-ID: <nq25dc$olc$1@blaine.gmane.org>

On 2016-08-29 22:10, Random832 wrote:
> On Mon, Aug 29, 2016, at 04:09, M.-A. Lemburg wrote:
>> Hmm, that last part would mean that Python 3.7 will no longer compile
>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>> Since 14.04 LTS is supported until 2019, I think it would be better
>> to only start requiring 1.0.2 in Python 3.8.
> 
> If someone's going to compile a new version of Python for Ubuntu LTS,
> they can compile a new version of OpenSSL. How likely is it that someone
> will want to use the packaged version of OpenSSL, but not use the
> packaged version (which is 3.4) of Python?

Please let me rephrase the question. How likely is it that somebody
won't use a container to deploy more recent versions? It's 2016.

Christian



From greg at krypto.org  Mon Aug 29 17:00:28 2016
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 29 Aug 2016 21:00:28 +0000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
Message-ID: <CAGE7PN+58CQUT2ikRhwVSK=31suoRY=ZKXV9_MY4QXgmC1xemA@mail.gmail.com>

On Mon, Aug 29, 2016 at 1:18 PM Christian Heimes <christian at python.org>
wrote:

> On 2016-08-29 21:31, M.-A. Lemburg wrote:
> > On 29.08.2016 18:33, Cory Benfield wrote:
> >>
> >>> On 29 Aug 2016, at 04:09, M.-A. Lemburg <mal at egenix.com> wrote:
> >>>
> >>> On 28.08.2016 22:40, Christian Heimes wrote:
> >>>> ...
> >>>> I like to reduce the maintenance burden and list of supported OpenSSL
> >>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
> >>>> will reach EOL by the end of this year,
> >>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> >>>> 0.9.8 is still required for some platforms (OSX).
> >>>> ...
> >>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> >>>> 1.0.2 features for 3.7.
> >>>> ...
> >>>
> >>> Hmm, that last part would mean that Python 3.7 will no longer compile
> >>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> >>> Since 14.04 LTS is supported until 2019, I think it would be better
> >>> to only start requiring 1.0.2 in Python 3.8.
> >>
> >> Can someone explain to me why this is a use-case we care about?
> >
> > Ubuntu 14.04 is a widely deployed system and newer Python version
> > should run on such widely deployed systems without having to
> > replace important vendor maintained system libraries such as
> > OpenSSL.
>
> "Widely deployed" is true for a lot of old operating systems including
> Windows XP.
>
> > Python 3.7 starts shipping around June 2018 (assuming the 18 month
> > release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
> > be able to use Python 3.7 on such a system, you'd have to upgrade
> > to a more recent LTS version 10 months before the EOL date (with
> > all the associated issues) or lose vendor maintenance support and
> > run with your own copy of OpenSSL.
>
> Why would you deploy an unsupported Python version on a LTS release? Why
> should compatibility be our concern?
>
> > Sure, but Ubuntu will continue to support OpenSSL 1.0.1
> > until 2019, backporting important security fixes as necessary and
> > that's what's important.
>
> I see an easy solution here: either pay or make Canonical backport all
> required features to OpenSSL 1.0.1. </sarcasm>
>
> > It's unfortunate that Python has to rely on a 3rd party library
> > for security, but we should at least make sure that our users
> > can rely on OS vendor support to keep the lib up to date with
> > security fixes.
>
> No, it is a good thing that we can rely on 3rd party libraries for
> security. Crypto and security is not our domain. It is incredible hard
> to develop and maintain crypto code. Also my proposal enforces OS
> vendors to supply up to date OpenSSL versions.
>
> >
> > On 29.08.2016 10:24, Christian Heimes wrote:
> >> By the way I knew that something like this would come up from you.
> >> Thank you that you satisfied my expectation. :p
> >
> > Sure, I want Python to be used on as many systems as possible,
> > both in terms of architecture and OS. The more the better.
> > If we don't have to drop support early, why should we ?
>
> MAL, I don't like your attitude. It feels like you want me and other
> contributors to waste time on this topic. That is not how this
> discussion is going to end. If *you* want to keep support for outdated
> OpenSSL versions, than it is *your* responsibility and *your* time. You
> cannot and will not put this burden on me.
>

Please keep your dialog civil Christian. That was unwarranted.

Nobody was forcing a burden upon you. 3.6 will remain buildable and usable
on common stable distros so long as we support 1.0.1 which sounds easy to
do.  For 3.7 we can move on and raise the minimum beyond that.

...

> opinion it is more than reasonable to ditch 1.0.1 and earlier.
>

Given that you already said:

"""
For 3.6 I don't require any 1.0.2 feature yet. The 1.1.0 patch keeps
code compatible with 0.9.8zc to 1.1.0. But as soon as I use new
features, the ssl module will no longer be source and build compatible
with < 1.0.2. There is also the point of OpenSSL 1.0.1. It reaches
end-of-lifetime by the end if this year. 1.0.2 will be supported until 2019.

I'm tempted to require 1.0.2 for Python 3.6 but it's technically not
necessary yet.
"""

That to me means we should keep support for 1.0.1 in Python 3.6 unless
there are features in 1.0.2 that you find are an absolute must have within
the next two weeks. We're going to be entering 3.6beta on September 12th
and current stable distros do not ship with a more recent version so lets
not make the lives of our developers and buildbot maintainers hell by
forcing them to install a special version.

Lets make 3.7 require a higher version. The common OSS OS distros of its
time will be better prepared.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160829/4a10ea71/attachment.html>

From christian at python.org  Mon Aug 29 17:20:32 2016
From: christian at python.org (Christian Heimes)
Date: Mon, 29 Aug 2016 23:20:32 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CAGE7PN+58CQUT2ikRhwVSK=31suoRY=ZKXV9_MY4QXgmC1xemA@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <CAGE7PN+58CQUT2ikRhwVSK=31suoRY=ZKXV9_MY4QXgmC1xemA@mail.gmail.com>
Message-ID: <2a7a6fdc-b660-5176-bf98-69c917e05d47@python.org>

On 2016-08-29 23:00, Gregory P. Smith wrote:
> 
> Given that you already said:
> 
> """
> For 3.6 I don't require any 1.0.2 feature yet. The 1.1.0 patch keeps
> code compatible with 0.9.8zc to 1.1.0. But as soon as I use new
> features, the ssl module will no longer be source and build compatible
> with < 1.0.2. There is also the point of OpenSSL 1.0.1. It reaches
> end-of-lifetime by the end if this year. 1.0.2 will be supported until 2019.
> 
> I'm tempted to require 1.0.2 for Python 3.6 but it's technically not
> necessary yet.
> """
> 
> That to me means we should keep support for 1.0.1 in Python 3.6 unless
> there are features in 1.0.2 that you find are an absolute must have
> within the next two weeks. We're going to be entering 3.6beta on
> September 12th and current stable distros do not ship with a more recent
> version so lets not make the lives of our developers and buildbot
> maintainers hell by forcing them to install a special version.
> 
> Lets make 3.7 require a higher version. The common OSS OS distros of its
> time will be better prepared.

My multissl test script allows me to compile and test _ssl.c and
_hashopenssl.c with multiple versions of OpenSSL and LibreSSL in less
than a minute. For 3.6 I have verified source compatibility with 0.9.8zc
(maybe even older) up to 1.1.0.

My argument with MAL is about future features for 3.7. I'm not planning
to require 1.0.2 APIs for 3.6 yet. This may change in case new security
issues are found. I might clean up the ssl module and require 0.9.8zc+
during beta, though.

Christian




From brett at python.org  Mon Aug 29 17:38:19 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Aug 2016 21:38:19 +0000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to code
 objects
Message-ID: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>

For quick background for those that don't remember, part of PEP 523
proposed adding a co_extra field to code objects along with making the
frame evaluation function pluggable (
https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The idea
was that things like JITs and debuggers could use the field as a scratch
space of sorts to store data related to the code object. People who
objected to the new field did either for memory ("it adds another pointer
to the struct that won't be typically used"), or for conceptual reasons
("the code object is immutable and you're proposing a mutable field"). The
latter is addressed by not exposing the field in Python and clearly stating
that code should never expect the field to be filled.

For the former issue of whether the memory was worth it, Dino has been
testing whether the field is necessary for performance from a JIT
perspective. Well, Dino found the time to test Pyjion without the co_extra
field and it isn't pretty. With the field, Pyjion is faster than stock
Python in 15 benchmarks (
https://github.com/Microsoft/Pyjion/tree/master/Perf). Removing the
co_extra field and using an unordered_map from the C++ STL drops that
number to 2. Performance is even worse if we try and use a Python
dictionary instead.

That means we still want to find a solution to attach arbitrary data to
code objects without sacrificing performance. One proposal is what's in PEP
523 for the extra field. Another option is to make the memory allocator for
code objects pluggable and introduce a new flag that signals that the
object was created using a non-default allocator. Obviously we prefer the
former solution due to its simplicity. :)

Anyway, we would like to get this settled this week so that I can get
whatever solution we agree to (if any) in next week in time for Python
3.6b1 feature freeze that would be greatly appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160829/3644d040/attachment.html>

From fijall at gmail.com  Mon Aug 29 18:12:25 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 30 Aug 2016 00:12:25 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
Message-ID: <CAK5idxSpedmkHtuTpKL_z4eCpsKaAN_18Fx84Yrgnio==pRttw@mail.gmail.com>

Hi Brett

For what is worth, vmprof and similar tools would love such field
(there is an open question how can you use vmprof *and* another tool,
but later)

On Mon, Aug 29, 2016 at 11:38 PM, Brett Cannon <brett at python.org> wrote:
> For quick background for those that don't remember, part of PEP 523 proposed
> adding a co_extra field to code objects along with making the frame
> evaluation function pluggable
> (https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The idea
> was that things like JITs and debuggers could use the field as a scratch
> space of sorts to store data related to the code object. People who objected
> to the new field did either for memory ("it adds another pointer to the
> struct that won't be typically used"), or for conceptual reasons ("the code
> object is immutable and you're proposing a mutable field"). The latter is
> addressed by not exposing the field in Python and clearly stating that code
> should never expect the field to be filled.
>
> For the former issue of whether the memory was worth it, Dino has been
> testing whether the field is necessary for performance from a JIT
> perspective. Well, Dino found the time to test Pyjion without the co_extra
> field and it isn't pretty. With the field, Pyjion is faster than stock
> Python in 15 benchmarks
> (https://github.com/Microsoft/Pyjion/tree/master/Perf). Removing the
> co_extra field and using an unordered_map from the C++ STL drops that number
> to 2. Performance is even worse if we try and use a Python dictionary
> instead.
>
> That means we still want to find a solution to attach arbitrary data to code
> objects without sacrificing performance. One proposal is what's in PEP 523
> for the extra field. Another option is to make the memory allocator for code
> objects pluggable and introduce a new flag that signals that the object was
> created using a non-default allocator. Obviously we prefer the former
> solution due to its simplicity. :)
>
> Anyway, we would like to get this settled this week so that I can get
> whatever solution we agree to (if any) in next week in time for Python 3.6b1
> feature freeze that would be greatly appreciated.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>

From christian at python.org  Mon Aug 29 18:28:10 2016
From: christian at python.org (Christian Heimes)
Date: Tue, 30 Aug 2016 00:28:10 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
Message-ID: <723bd0e3-d776-23b4-a9ca-7d4841be521d@python.org>

On 2016-08-29 23:38, Brett Cannon wrote:
> That means we still want to find a solution to attach arbitrary data to
> code objects without sacrificing performance. One proposal is what's in
> PEP 523 for the extra field. Another option is to make the memory
> allocator for code objects pluggable and introduce a new flag that
> signals that the object was created using a non-default allocator.
> Obviously we prefer the former solution due to its simplicity. :)

May I remind you that you can have the field with no extra memory cost?
:) The struct has sub-par alignments.

Christian

From victor.stinner at gmail.com  Mon Aug 29 18:38:31 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Aug 2016 00:38:31 +0200
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
Message-ID: <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>

Hi,

tl; dr: just drop byte support and help developers to use Unicode in
their application!


As you may already know, I dislike your whole project. But first of
all, IMHO you should open a separated thread to discuss changes
related to the Windows console. I consider that they are unrelated,
well defined and makes sense to everyone, no?


It would help the discussion to estimate how much code is going to
break on Windows if Python 3 drops byte support. Maybe try some common
and major applications like Django and Twisted? My expectation is that
very few code explicitly uses bytes path, and that most applicatons
already use Unicode, just because it's much more easier to use Unicode
than bytes on Python 3.

Mercurial does its best to keep filenames as bytes, unchanged. I tried
once to port Mercurial to Python 3, and I recall that it was very
painful to handle filenames as bytes. A simple example:
print(filename) raises a BytesWarning (or displays a quoted string
prefixed with 'b', not really the expected result).


You always use the same example os.listdir(bytes). Ok, this one is a
mess on Windows. But modifying the value of
sys.getfilesystemencoding() has a broad effect: it changes *all*
functions in Python (ok, not really "all", but I'm trying to say that
a lot of functions use it). It's not only filenames: hostnames
(sockets), input/output data from other applications (subprocess
pipes), file content, command line arguments, environment variables,
etc. also use the ANSI code page.

Well, I know well UNIX where everything is stored as bytes. But I know
that hopefully, Windows has cool functions providing data directly as
Unicode, avoiding any mojibake issues, at least for command line
arguments and environment variables. But while your application may be
carefuly with Unicode, in practice "data" is transfered between many
channels and suffers from conversions.

There is no such single channel well defined. For example, "pipes" are
a pain on Windows. There is no "Unicode pipe" to exchange *text*
between two processes. I only know "cmd /u" (which seems specific and
restricted to cmd.exec) or _setmode() which has its own set of issues:
http://bugs.python.org/issue16587

See also this old article
http://archives.miloush.net/michkap/archive/2008/03/18/8306597.html


If Python 3.6 is going to speak UTF-8 only, there is a high risk that
it is going to produce data unreadable by other applications. For
example, write a filename into a file. I expect that the filename will
be stored as bytes, so as UTF-8. If another application tries to
decode the file content from the ANSI code page, it's likely that you
will get a "fatal" Unicode decoding error at the first non-ASCII
character.

The status quo is not better, you cannot store Japanese filenames in a
file if your ANSI code page is french. But if you decide to produce a
UTF-8 file, the effect will be restricted to this file, not to *all*
input and output data! For example, distutils was slowly upgraded
(from "encoding not specified by any spec, but more likely the locale
encoding") to use UTF-8 everywhere ;-)


More generally, I understand that you propose to explicitly break the
interoperability with other applications. Python 2 can be used as an
example of "application" expecting data encoded to the ANSI code page.

Yeah, I expect that storing text as UTF-8 inside the same process will
reduce the number of unicode errors *inside the same process*. It's
likely that you will no more get unicode errors if you control all
data inside the same process.

But for me, interoperability is more important than benefits of your
proposed changes.


Another way to express my concern: I don't understand how you plan to
"transcode" data from/to other applications. Will subprocess guess the
input and output encoding and transcode between the guessed encoding
and UTF-8?

Even today with Python 3.5, it's already easy to get mojibake betwen
the ANSI code page, the OEM code page, and some other encodings used
by an application. In short, you introduce yet another encoding which
is rare on Windows and incompatible with other encodings (well, except
of ASCII which is subset of many encodings, most encoings are
incompatible with all other encodings).


To me, you are building a giant patch to hide the mess, whereas this
problem was already solved in 1991 with the release of Unicode 1.0.
I'm not saying that it's a mistake to use UTF-8 internally. It's more
than I don't think that it's worth it. There is a risk that it adds
extra work to "support UTF-8", whereas this energy would be better
used to support Unicode, no?


I suggest to either fix the problem by really dropping bytes support,
as "announced" in Python 3.3 with the deprecation, or do nothing (and
continue to suffer).

I'm a supporter of the former option :-) Force developers to fix their
application and learn Unicode the hard way :-)

>From what I heard, Unicode is not really a pain of Python 3. The pain
comes from porting Python 2 code to Python 3, because not only you
have to do the boring refactoring work to fix Python 3 syntax issues,
but also keep Python 2 compatibility, *and* fix *all* Unicode issues
*at once*. From what I heard, text handling in Python 3 is really a
pleasure since the "encoding issue" (Unicode?) is "solved".

We only solved the issue by forcing developers to use Unicode
everywhere. I don't think that we can continue to make shy compromises
on this topic anymore...


Ok. Maybe I'm just wrong and using UTF-8 internally will be super cool :-)

Victor

From guido at python.org  Mon Aug 29 18:51:12 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 29 Aug 2016 15:51:12 -0700
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <723bd0e3-d776-23b4-a9ca-7d4841be521d@python.org>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <723bd0e3-d776-23b4-a9ca-7d4841be521d@python.org>
Message-ID: <CAP7+vJ+BroUtkpi4kWiw8yP8VN57mY7YwrRR1Z=scutHbmWhHQ@mail.gmail.com>

Anyway, given the outcome of Dino's tests I have no objections to the
PEP. (Though using Christian's hack would be cool.)

On Mon, Aug 29, 2016 at 3:28 PM, Christian Heimes <christian at python.org> wrote:
> On 2016-08-29 23:38, Brett Cannon wrote:
>> That means we still want to find a solution to attach arbitrary data to
>> code objects without sacrificing performance. One proposal is what's in
>> PEP 523 for the extra field. Another option is to make the memory
>> allocator for code objects pluggable and introduce a new flag that
>> signals that the object was created using a non-default allocator.
>> Obviously we prefer the former solution due to its simplicity. :)
>
> May I remind you that you can have the field with no extra memory cost?
> :) The struct has sub-par alignments.
>
> Christian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From barry at python.org  Mon Aug 29 18:56:00 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 29 Aug 2016 18:56:00 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
Message-ID: <20160829185600.1d27bc02@subdivisions.wooz.org>

On Aug 29, 2016, at 12:33 PM, Cory Benfield wrote:

>Can someone explain to me why this is a use-case we care about?

I do think it would be nice to be able to compile newer versions of Python on
stock LTS releases, especially for people developing software that they want
to support on a wide-range of Python 3 versions.

Sure, you can use your container of choice, and there are ways around this,
but every hurdle just makes it more of a pain to do.  So I'd say if we can
keep the compatibility without too much effort, and without sacrificing the
security or maintainability of the code, we should do it.  If we can't, we
can't.

Cheers,
-Barry

From brett at python.org  Mon Aug 29 19:14:23 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Aug 2016 23:14:23 +0000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAP7+vJ+BroUtkpi4kWiw8yP8VN57mY7YwrRR1Z=scutHbmWhHQ@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <723bd0e3-d776-23b4-a9ca-7d4841be521d@python.org>
 <CAP7+vJ+BroUtkpi4kWiw8yP8VN57mY7YwrRR1Z=scutHbmWhHQ@mail.gmail.com>
Message-ID: <CAP1=2W5gm0V7Coc2269NEpUWV0dQj33DnQV5yp-ju5d5_xd+Cw@mail.gmail.com>

On Mon, 29 Aug 2016 at 15:51 Guido van Rossum <guido at python.org> wrote:

> Anyway, given the outcome of Dino's tests I have no objections to the
> PEP. (Though using Christian's hack would be cool.)
>

Great! I'll mark the PEP as accepted and get the implementation in for 3.6.


>
> On Mon, Aug 29, 2016 at 3:28 PM, Christian Heimes <christian at python.org>
> wrote:
> > On 2016-08-29 23:38, Brett Cannon wrote:
> >> That means we still want to find a solution to attach arbitrary data to
> >> code objects without sacrificing performance. One proposal is what's in
> >> PEP 523 for the extra field. Another option is to make the memory
> >> allocator for code objects pluggable and introduce a new flag that
> >> signals that the object was created using a non-default allocator.
> >> Obviously we prefer the former solution due to its simplicity. :)
> >
> > May I remind you that you can have the field with no extra memory cost?
> > :)


Yes you may. :)


> The struct has sub-par alignments.
>

So the struct in question can be found at
https://github.com/python/cpython/blob/2d264235f6e066611b412f7c2e1603866e0f7f1b/Include/code.h#L10
.
The official docs say the fields can be changed at any time, so
re-arranging them shouldn't break any ABI compatibility promises:
https://docs.python.org/3/c-api/code.html#c.PyCodeObject . Would grouping
all the fields of the same type together, sorting them by individual field
size (i.e. PyObject*, void*, int, unsigned char*), and then adding the
co_extra field at the end of the grouping of PyObject * fields do what
you're suggesting?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160829/1efb8a7b/attachment.html>

From victor.stinner at gmail.com  Mon Aug 29 19:14:55 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Aug 2016 01:14:55 +0200
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
Message-ID: <CAMpsgwYcSD3-K4L1L_==EGTt=abJBqnGSFb604XS=2MHw8ZoNw@mail.gmail.com>

2016-08-20 21:31 GMT+02:00 Nick Coghlan <ncoghlan at gmail.com>:
> Reading your summary meant this finally clicked with something Victor
> has been considering for a while: a "Force UTF-8" switch that told
> Python to ignore the locale encoding on Linux, and instead assume
> UTF-8 everywhere (command line parameter parsing, environment variable
> processing, filesystem encoding, standard streams, etc)
>
> It's essentially the same problem you have on Windows, just with
> slightly different symptoms and consequences.

Yes and no, but more no than yes :-)

On Linux, the issue is quite simple: most major Linux distributions
switched to UTF-8 by default, network shares use UTF-8, filenames are
stored as UTF-8, applications expect UTF-8, etc. I proposed once a "-X
utf8" switch, but more as a convenient workaround for badly configured
system which encode data to UTF-8, but the locale encoding is not
properly configured *in some cases*. The switch does a single thing:
ignore the locale encoding, and force UTF-8 as the locale encoding.

Steve's proposition is specific to Windows, and Windows is a different
world. On Windows, there is one unique distribution: the Microsoft
flavor, and UTF-8 was and is *never* used as the ANSI code page (which
is more and less the same thing that UNIX locale encoding). Using
UTF-8 is something new, not really common in the Windows world. Steve
said that UTF-8 is common in the .NET (but I don't know well Windows
community/universe).

I proposed to Steve to work on an unified "-X utf8" option to
explicitly force UTF-8 on Linux and Windows. But Steve looks to prefer
to force UTF-8 *by default*, but add a new option to revert the old
behaviour.

I proposed the idea, but I'm not sure that we can have a single option
for Linux and Windows. Moreover, I never really worked on trying to
implement "-X utf8" on Linux, because it looks like the "misconfigured
system" are less and less common nowadays. I see very few user
requests in this direction.

By the way, except Steve, did someone complain about the ANSI code
page for bytes on Windows in Python? I recall one or two issues last 5
years about the os.listdir(bytes) issue, but these issues were
specific to Python 2 if I recall correctly?

Victor

From brett at python.org  Mon Aug 29 19:15:14 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Aug 2016 23:15:14 +0000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAK5idxSpedmkHtuTpKL_z4eCpsKaAN_18Fx84Yrgnio==pRttw@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <CAK5idxSpedmkHtuTpKL_z4eCpsKaAN_18Fx84Yrgnio==pRttw@mail.gmail.com>
Message-ID: <CAP1=2W4w-HmXmhZ7+tBFSC0GBWOiXEjXoVi8gwj0AuYXZ4-BOw@mail.gmail.com>

On Mon, 29 Aug 2016 at 15:12 Maciej Fijalkowski <fijall at gmail.com> wrote:

> Hi Brett
>
> For what is worth, vmprof and similar tools would love such field
> (there is an open question how can you use vmprof *and* another tool,
> but later)
>

That's great to hear! Glad the solution has multiple use-cases.

-Brett


>
> On Mon, Aug 29, 2016 at 11:38 PM, Brett Cannon <brett at python.org> wrote:
> > For quick background for those that don't remember, part of PEP 523
> proposed
> > adding a co_extra field to code objects along with making the frame
> > evaluation function pluggable
> > (https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The
> idea
> > was that things like JITs and debuggers could use the field as a scratch
> > space of sorts to store data related to the code object. People who
> objected
> > to the new field did either for memory ("it adds another pointer to the
> > struct that won't be typically used"), or for conceptual reasons ("the
> code
> > object is immutable and you're proposing a mutable field"). The latter is
> > addressed by not exposing the field in Python and clearly stating that
> code
> > should never expect the field to be filled.
> >
> > For the former issue of whether the memory was worth it, Dino has been
> > testing whether the field is necessary for performance from a JIT
> > perspective. Well, Dino found the time to test Pyjion without the
> co_extra
> > field and it isn't pretty. With the field, Pyjion is faster than stock
> > Python in 15 benchmarks
> > (https://github.com/Microsoft/Pyjion/tree/master/Perf). Removing the
> > co_extra field and using an unordered_map from the C++ STL drops that
> number
> > to 2. Performance is even worse if we try and use a Python dictionary
> > instead.
> >
> > That means we still want to find a solution to attach arbitrary data to
> code
> > objects without sacrificing performance. One proposal is what's in PEP
> 523
> > for the extra field. Another option is to make the memory allocator for
> code
> > objects pluggable and introduce a new flag that signals that the object
> was
> > created using a non-default allocator. Obviously we prefer the former
> > solution due to its simplicity. :)
> >
> > Anyway, we would like to get this settled this week so that I can get
> > whatever solution we agree to (if any) in next week in time for Python
> 3.6b1
> > feature freeze that would be greatly appreciated.
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160829/1b0c6c1b/attachment.html>

From victor.stinner at gmail.com  Mon Aug 29 19:22:32 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Aug 2016 01:22:32 +0200
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
 <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
 <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>
 <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>
Message-ID: <CAMpsgwY+jryLrx=8qmjmMQNhAJQrU-5RpaCKJmrbDhyFXMoNYg@mail.gmail.com>

2016-08-24 17:44 GMT+02:00 Steve Dower <steve.dower at python.org>:
> I know Nick and Victor like the idea of a -X flag (or a direct -utf8 flag),
> but I prefer more specific environment variables:
>
> - PYTHONWINDOWSLEGACYSTDIO (for the console changes)
> - PYTHONWINDOWSLEGACYPATHENCODING (assuming getfilesystemencoding() is utf8)
> - PYTHONWINDOWSLEGACYLOCALEENCODING (assuming getpreferredencoding() is
> utf8)

Hum, you propose the opposite behaviour of my proposed "-X utf8"
switch :-) You propose to force UTF-8 by default, but add a new flag
to revert to the old behaviour. Right?

About (...)LEGACYSTDIO: I don't think that we need such option. Using
Unicode is the obvious choice if stdout/stderr is the Windows console
no? I'm not talking about the case where stdout/stderr is redirected
(into a file or a pipe). Are you aware of any case where the "legacy
stdio" is better than the "new" (win_unicode_console-like) stdio?

(...)PATHENCODING and (...)LOCALEENCODING: please don't do that. I
tried once in Python 3.2 on Linux, and it failed badly:
https://mail.python.org/pipermail/python-dev/2010-October/104517.html

I don't think that it's a good idea to add an option to use two
different encodings inside the same process. It's too common to
exchange data between the "path encoding world" and "locale encoding
world".

Hum, currently, is it possible to get two different encodings for
"path encoding" and "locale encoding"? Or both encodings are the ANSI
code page? I only recall that the console uses a different encoding,
the OEM code page.

Victor

From tjreedy at udel.edu  Mon Aug 29 19:48:46 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 29 Aug 2016 19:48:46 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <2a7a6fdc-b660-5176-bf98-69c917e05d47@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <CAGE7PN+58CQUT2ikRhwVSK=31suoRY=ZKXV9_MY4QXgmC1xemA@mail.gmail.com>
 <2a7a6fdc-b660-5176-bf98-69c917e05d47@python.org>
Message-ID: <nq2hkv$at0$1@blaine.gmane.org>

On 8/29/2016 5:20 PM, Christian Heimes wrote:
> On 2016-08-29 23:00, Gregory P. Smith wrote:

>> Lets make 3.7 require a higher version. The common OSS OS distros of its
>> time will be better prepared.

Especially is warned.

> My multissl test script allows me to compile and test _ssl.c and
> _hashopenssl.c with multiple versions of OpenSSL and LibreSSL in less
> than a minute. For 3.6 I have verified source compatibility with 0.9.8zc
> (maybe even older) up to 1.1.0.
>
> My argument with MAL is about future features for 3.7. I'm not planning
> to require 1.0.2 APIs for 3.6 yet. This may change in case new security
> issues are found. I might clean up the ssl module and require 0.9.8zc+
> during beta, though.

So, in 3.6, and maybe even 3.5.3, add a deprecation warning when OSSL < 
1.0.2 is used.


-- 
Terry Jan Reedy


From tjreedy at udel.edu  Mon Aug 29 20:05:40 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 29 Aug 2016 20:05:40 -0400
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
Message-ID: <nq2ikm$bge$1@blaine.gmane.org>

On 8/29/2016 5:38 PM, Brett Cannon wrote:

> who objected to the new field did either for memory ("it adds another
> pointer to the struct that won't be typically used"), or for conceptual
> reasons ("the code object is immutable and you're proposing a mutable
> field"). The latter is addressed by not exposing the field in Python and

Am I correct is thinking that you will also not add the new field as an 
argument to PyCode_New?

 > clearly stating that code should never expect the field to be filled.

I interpret this as "The only code that should access the field should 
be code that put something there."

-- 
Terry Jan Reedy


From brett at python.org  Mon Aug 29 21:00:59 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Aug 2016 01:00:59 +0000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <nq2ikm$bge$1@blaine.gmane.org>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <nq2ikm$bge$1@blaine.gmane.org>
Message-ID: <CAP1=2W6kHFNxXkND9QnzAAiiawnoOYGd76adARs0mBjFhnw5xg@mail.gmail.com>

On Mon, Aug 29, 2016, 17:06 Terry Reedy <tjreedy at udel.edu> wrote:

> On 8/29/2016 5:38 PM, Brett Cannon wrote:
>
> > who objected to the new field did either for memory ("it adds another
> > pointer to the struct that won't be typically used"), or for conceptual
> > reasons ("the code object is immutable and you're proposing a mutable
> > field"). The latter is addressed by not exposing the field in Python and
>
> Am I correct is thinking that you will also not add the new field as an
> argument to PyCode_New?
>

Correct.


>  > clearly stating that code should never expect the field to be filled.
>
> I interpret this as "The only code that should access the field should
> be code that put something there."
>

Yep, seems like a reasonable rule to follow.

-brett


> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/8c6838a0/attachment.html>

From dholth at gmail.com  Mon Aug 29 21:10:21 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 30 Aug 2016 01:10:21 +0000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAMpsgwY+jryLrx=8qmjmMQNhAJQrU-5RpaCKJmrbDhyFXMoNYg@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
 <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
 <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>
 <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>
 <CAMpsgwY+jryLrx=8qmjmMQNhAJQrU-5RpaCKJmrbDhyFXMoNYg@mail.gmail.com>
Message-ID: <CAG8k2+7awGmR2qpKDr9rgGG3pHcquGXFPJGbm76Gu62ZKXSeNQ@mail.gmail.com>

A nice recent blog post about paths encoding in a media player.
http://beets.io/blog/paths.html . It's not merely the porting that makes it
hard.

On Mon, Aug 29, 2016, 19:32 Victor Stinner <victor.stinner at gmail.com> wrote:

> 2016-08-24 17:44 GMT+02:00 Steve Dower <steve.dower at python.org>:
> > I know Nick and Victor like the idea of a -X flag (or a direct -utf8
> flag),
> > but I prefer more specific environment variables:
> >
> > - PYTHONWINDOWSLEGACYSTDIO (for the console changes)
> > - PYTHONWINDOWSLEGACYPATHENCODING (assuming getfilesystemencoding() is
> utf8)
> > - PYTHONWINDOWSLEGACYLOCALEENCODING (assuming getpreferredencoding() is
> > utf8)
>
> Hum, you propose the opposite behaviour of my proposed "-X utf8"
> switch :-) You propose to force UTF-8 by default, but add a new flag
> to revert to the old behaviour. Right?
>
> About (...)LEGACYSTDIO: I don't think that we need such option. Using
> Unicode is the obvious choice if stdout/stderr is the Windows console
> no? I'm not talking about the case where stdout/stderr is redirected
> (into a file or a pipe). Are you aware of any case where the "legacy
> stdio" is better than the "new" (win_unicode_console-like) stdio?
>
> (...)PATHENCODING and (...)LOCALEENCODING: please don't do that. I
> tried once in Python 3.2 on Linux, and it failed badly:
> https://mail.python.org/pipermail/python-dev/2010-October/104517.html
>
> I don't think that it's a good idea to add an option to use two
> different encodings inside the same process. It's too common to
> exchange data between the "path encoding world" and "locale encoding
> world".
>
> Hum, currently, is it possible to get two different encodings for
> "path encoding" and "locale encoding"? Or both encodings are the ANSI
> code page? I only recall that the console uses a different encoding,
> the OEM code page.
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/f09b932d/attachment-0001.html>

From ncoghlan at gmail.com  Mon Aug 29 21:10:48 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 Aug 2016 11:10:48 +1000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
Message-ID: <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>

On 30 August 2016 at 08:38, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> tl; dr: just drop byte support and help developers to use Unicode in
> their application!

My view (and Steve's) is that this approach is likely to result in
Linux-centric projects just dropping even nominal native Windows
support, rather than more Python software that handles Unicode on
Windows (/the CLR/the JVM) correctly.

What Steve is proposing here is essentially a way of providing more
*nix like CPython behaviour on Windows, just as the Windows Subsystem
for Linux is Microsoft's latest iteration on providing such an
environment at the kernel layer.

This is why interoperability with other Windows programs isn't a major
concern for this proposal - projects with Windows developers actually
working on them that are designed to operate with other applications
running on the same device (especially via files) should absolutely be
using strings rather than bytes for interacting with the operating
system.

Instead, the target audience I see for these changes is
network-centric projects where all the primary developers are running
either Linux or Mac OS X on their personal devices, and deploying to
Linux servers. If you tell them "You need to make major changes to
your project to properly support native deployment on Windows", then
they will say "OK, we'll just drop the claim of Windows support, so
we're not giving people false expectations" rather than actually
making the change. That then excludes a lot of potential contributors
from their project (since they need to be able to run Cygwin, WSL,
containers or VMs to get a compatible test environment), and it also
means folks running Windows based infrastructure can't readily run the
service up on an existing system without using like Cygwin, Docker, or
the new WSL.

However, this view is also why I don't agree with being aggressive in
making this behaviour the default on Windows - I think we should make
it readily available as a provisional feature through a single
cross-platform command line switch and environment setting (e.g. "-X
utf8" and "PYTHONASSUMEUTF8") so folks that need it can readily opt in
to it, but we can defer making it the default until 3.7 after folks
have had a full release cycle's worth of experience with it in the
wild.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Aug 29 21:20:47 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 Aug 2016 11:20:47 +1000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAG8k2+7awGmR2qpKDr9rgGG3pHcquGXFPJGbm76Gu62ZKXSeNQ@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CADiSq7f0XHpKyXQgir7Ag9crXhxePzvUdx+z2TZ+tgpFEv9HBQ@mail.gmail.com>
 <57B8BE3F.9090405@python.org>
 <CADiSq7dY7doJu8U9zpfAzyGna1_r390Kxofi-tFkNEFqO_2Z-g@mail.gmail.com>
 <22458.51738.707976.197811@turnbull.sk.tsukuba.ac.jp>
 <36787fec-8202-38aa-a756-e36e59ab9fdd@python.org>
 <22459.53058.684165.404405@turnbull.sk.tsukuba.ac.jp>
 <9bf9f1a0-26bb-2b78-83a1-85a00e8e2e7b@python.org>
 <22461.10110.122755.999441@turnbull.sk.tsukuba.ac.jp>
 <0fc5a216-37c5-cabe-e924-f9cb9235cb1b@python.org>
 <CAMpsgwY+jryLrx=8qmjmMQNhAJQrU-5RpaCKJmrbDhyFXMoNYg@mail.gmail.com>
 <CAG8k2+7awGmR2qpKDr9rgGG3pHcquGXFPJGbm76Gu62ZKXSeNQ@mail.gmail.com>
Message-ID: <CADiSq7fZr6xiY0yhfprt9Fing8GMqqDrGjB81pb_6G9xroCsPw@mail.gmail.com>

On 30 August 2016 at 11:10, Daniel Holth <dholth at gmail.com> wrote:
> A nice recent blog post about paths encoding in a media player.
> http://beets.io/blog/paths.html . It's not merely the porting that makes it
> hard.

Interestingly, with Steve's proposed changes, their approach of using
bytes paths internally should "just work" on Windows in UTF-8 mode,
even without their syspath converter. So I may be being unduly
pessimistic in assuming this will be useful primarily for network
services - cross-platform desktop development may benefit as well.

(The additional info doesn't change my view that we'd benefit from a
release cycle's worth of availability as an opt-in provisional
behaviour though)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Aug 29 22:18:26 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 Aug 2016 12:18:26 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
Message-ID: <CADiSq7f=mSrD6Fipy0HG=jkgtbzBG3q6--vGBDZX38ffgg3=GA@mail.gmail.com>

On 30 August 2016 at 02:33, Cory Benfield <cory at lukasa.co.uk> wrote:
>
>> On 29 Aug 2016, at 04:09, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>> On 28.08.2016 22:40, Christian Heimes wrote:
>>> ...
>>> I like to reduce the maintenance burden and list of supported OpenSSL
>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>>> will reach EOL by the end of this year,
>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>>> 0.9.8 is still required for some platforms (OSX).
>>> ...
>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>> 1.0.2 features for 3.7.
>>> ...
>>
>> Hmm, that last part would mean that Python 3.7 will no longer compile
>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>> Since 14.04 LTS is supported until 2019, I think it would be better
>> to only start requiring 1.0.2 in Python 3.8.
>
> Can someone explain to me why this is a use-case we care about?

I can think of a few key cases where it's potentially going to have an impact:

- availability of new Python versions in public PaaS environments
- availability of new Python versions in free-for-open-source public CI services
- the "maximum supported Python version" for iterations of the manylinux project

Fortunately, public PaaS providers and public CI providers tend to
fall into the "upgrade Linux when new LTS releases are available, not
when the old ones are about to lose support" category, so a policy of
targeting the latest LTS releases (perhaps with a "more than 12 months
old" caveat) rather than the oldest still commercially supported ones
should be sufficient on that front (if the LTS vendors decide they'd
like us to change that policy, they always have the option of funding
it accordingly). Currently, that means:

- Debian 8
- Ubuntu 16.04 (or 14.04 with the "12 months old" caveat)
- RHEL/CentOS 7

And then trusting that supporting those will also cover other Linux
distros that aren't as directly involved in upstream CPython
development.

That means the main potential source of problems I see here is the
manylinux project, but fortunately that concern was accounted for by
explicitly excluding OpenSSL from the defined portable ABI:
https://www.python.org/dev/peps/pep-0513/#security-implications

So it's already a manylinux requirement that anyone wanting to use ssl
in a portable way needs to either use the standard library module, or
else something like PyOpenSSL (which bundles its own copy, independent
of what the system provides)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From cory at lukasa.co.uk  Mon Aug 29 22:28:50 2016
From: cory at lukasa.co.uk (Cory Benfield)
Date: Mon, 29 Aug 2016 22:28:50 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C48D7B.4080200@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
Message-ID: <BBAB0D8B-5572-406A-A40B-F759868321B1@lukasa.co.uk>

> On 29 Aug 2016, at 15:31, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> Ubuntu 14.04 is a widely deployed system and newer Python version
> should run on such widely deployed systems without having to
> replace important vendor maintained system libraries such as
> OpenSSL.

That's quite the non-sequitur. You never answered my question: what does "widely deployed" mean? At what level of deployment do we need to support the system configuration with no changes?

Do we need to support compiling out of box with Windows 10? Because we don't: if they want SSL, we need them to compile and install an OpenSSL. Do we need to support compiling out of the box on macOS 10.12 Sierra? Because we don't: if they want SSL they need to install their own OpenSSL.

At a certain point we need to give up on systems that do not provide up to date copies of important libraries, or say that if you want to use Python on them you need to compile without our support libraries. 

> Python 3.7 starts shipping around June 2018 (assuming the 18 month
> release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
> be able to use Python 3.7 on such a system, you'd have to upgrade
> to a more recent LTS version 10 months before the EOL date (with
> all the associated issues) or lose vendor maintenance support and
> run with your own copy of OpenSSL.

Yes, that's true. But on the other hand, that LTS release is *already out* at this time, and has been for four months. By the time of the release of Python 3.7 it will have been released for two years and two months. The request to upgrade is not unreasonable. 

> Sure, but Ubuntu will continue to support OpenSSL 1.0.1
> until 2019, backporting important security fixes as necessary and
> that's what's important.

Then Ubuntu can ship us an engineer who is willing to support the SSL module with OpenSSL 1.0.1 going forward. The one engineer we have has said he is unwilling to do it.

> This doesn't sound like a feature worth breaking compatibility
> to me.

Does the compatibility guarantee apply to libraries that Python will link against?

Cory

From ncoghlan at gmail.com  Mon Aug 29 22:59:08 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 Aug 2016 12:59:08 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <20160829185600.1d27bc02@subdivisions.wooz.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <20160829185600.1d27bc02@subdivisions.wooz.org>
Message-ID: <CADiSq7fF_yC-EgaaBjgVe93LNGSmRW0adLA0TL1Y4PU5VDZXYQ@mail.gmail.com>

On 30 August 2016 at 08:56, Barry Warsaw <barry at python.org> wrote:
> On Aug 29, 2016, at 12:33 PM, Cory Benfield wrote:
>
>>Can someone explain to me why this is a use-case we care about?
>
> I do think it would be nice to be able to compile newer versions of Python on
> stock LTS releases, especially for people developing software that they want
> to support on a wide-range of Python 3 versions.

Based on this discussion, I've been thinking about how we could
articulate these build support constraints in PEP 11, similar to the
https://www.python.org/dev/peps/pep-0011/#microsoft-windows section we
have for Windows.

For that, I think it makes sense to key off source compatibility with
the most recent releases of long term community or commercially
supported Linux distros that have folks directly involved in CPython
development (specifically Debian stable, Ubuntu LTS, and RHEL/CentOS).

The other thing I've been looking at is how well documented the
process is for building with a custom OpenSSL instead of the system
one, and as near as I can tell, it isn't documented at all - the top
level README doesn't mention it, and because the compilation is
handled by setup.py rather than directly by make, there's no configure
option for it the way there is for "--with-system-expat",
"--with-system-libmpdec" and "--with-system-ffi".

By contrast (and assuming I understand the situation correctly), the
Windows build is already set up around the assumption that you'll need
to build OpenSSL yourself.

So, in addition to updating PEP 11 with guidelines for determining the
OpenSSL constraints for each release, I think we'll also need a README
update to cover building against a non-system OpenSSL.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From nad at python.org  Mon Aug 29 23:24:36 2016
From: nad at python.org (Ned Deily)
Date: Mon, 29 Aug 2016 23:24:36 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CADiSq7fF_yC-EgaaBjgVe93LNGSmRW0adLA0TL1Y4PU5VDZXYQ@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <20160829185600.1d27bc02@subdivisions.wooz.org>
 <CADiSq7fF_yC-EgaaBjgVe93LNGSmRW0adLA0TL1Y4PU5VDZXYQ@mail.gmail.com>
Message-ID: <E06F3ECE-EFDF-48B5-9A34-39792EC21241@python.org>

On Aug 29, 2016, at 22:59, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The other thing I've been looking at is how well documented the
> process is for building with a custom OpenSSL instead of the system
> one, and as near as I can tell, it isn't documented at all - the top
> level README doesn't mention it, and because the compilation is
> handled by setup.py rather than directly by make, there's no configure
> option for it the way there is for "--with-system-expat",
> "--with-system-libmpdec" and "--with-system-ffi".

There is at least one open issue with a supplied patch to provide a --with-ssl configure option:

http://bugs.python.org/issue21541

The patch may need some freshening up.  There are also other open issues on configuring third-party libraries in general.

> By contrast (and assuming I understand the situation correctly), the
> Windows build is already set up around the assumption that you'll need
> to build OpenSSL yourself.
> 
> So, in addition to updating PEP 11 with guidelines for determining the
> OpenSSL constraints for each release, I think we'll also need a README
> update to cover building against a non-system OpenSSL.

For OS X, we can copy the info about building with OpenSSL from Homebrew or MacPorts from the Developer's Guide. There are other things that could be done. The OS X installer build knows how to build OpenSSL as well as other needed libs.  I'd like to refactor that to make those libs usable for non-installer builds.  I've also thought it would be nice to provide simpler configure options to use Homebrew, MaPorts, or build-your-own.  That's probably not going to happen in time for 3.6b1, though.

--
  Ned Deily
  nad at python.org -- []


From steve.dower at python.org  Mon Aug 29 23:29:21 2016
From: steve.dower at python.org (Steve Dower)
Date: Mon, 29 Aug 2016 20:29:21 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
Message-ID: <57C4FD91.5020804@python.org>

On 29Aug2016 1810, Nick Coghlan wrote:
> On 30 August 2016 at 08:38, Victor Stinner <victor.stinner at gmail.com> wrote:
>> Hi,
>>
>> tl; dr: just drop byte support and help developers to use Unicode in
>> their application!
>
> My view (and Steve's) is that this approach is likely to result in
> Linux-centric projects just dropping even nominal native Windows
> support, rather than more Python software that handles Unicode on
> Windows (/the CLR/the JVM) correctly.

Yeah, this basically sums it up. If I could be sure that the Python 
developers who are 99% Linux/1% Windows (i.e. run unit tests once and 
then release) weren't going to see dropping byte support completely as a 
hostile action, I'd much rather go that way.

But let's definitely take note that platform-specific deprecation 
warnings are probably not a good idea for cross-platform functionality.

> What Steve is proposing here is essentially a way of providing more
> *nix like CPython behaviour on Windows

Yep. What actually spurred me into action on this was a Twitter rant 
from one of Twisted's developers about paths on Windows. So I presume 
that Twisted is probably okay *now* (and hopefully because they 
explicitly decode from network traffic into str before accessing the 
file system...)

Using bytes has essentially always been using an arbitrarily-encoded str 
on Windows. The active code page is not an equivalent of "give me the 
path as raw bytes" as it is on POSIX, but my change will make it so that 
it is. There'll be a performance penalty, but otherwise using bytes for 
paths will become reliable.

Unfortunately, any implicitly-encoded cross-version interoperability 
will have to be broken by such a change. There's just no way around it. 
But I've seen no evidence that it's common, and there are two 
workarounds available (set the environment variable, or change your code 
to specify the encoding used).

> However, this view is also why I don't agree with being aggressive in
> making this behaviour the default on Windows - I think we should make
> it readily available as a provisional feature through a single
> cross-platform command line switch and environment setting (e.g. "-X
> utf8" and "PYTHONASSUMEUTF8") so folks that need it can readily opt in
> to it, but we can defer making it the default until 3.7 after folks
> have had a full release cycle's worth of experience with it in the
> wild.

Given the people who would need to opt-in to the behaviour are merely 
the recipients of a library written by someone else, I don't think this 
is the right approach. Stephen Turnbull in an earlier post referred to 
organisations that fully control their systems in order to ensure that 
the implicit encodings all match. These are also the people who can 
apply an environment variable to avoid a behaviour change.

However, someone who just installed an HTTP library that was developed 
on POSIX and perhaps not even tested on Windows should not have to flick 
the switch themselves. In contrast, if it is known that 3.6 *definitely* 
changed something here, we will certainly see more effort applied to 
making sure libraries are updated. (Compare these two bug reports: "your 
library breaks on Python 3.6" vs "your library breaks on Python 3.6 when 
I set this environment variable". The fix for the latter is quite 
reasonably going to be "don't do that".)

The other discussion about OpenSSL and LTS systems is also interesting. 
Do we really expect users to take their fully functioning systems and 
blindly upgrade to a new major version of Python expecting everything to 
just work? That seems very unlikely to me, and also doesn't match my 
experience (but I can't quantify that in any useful way, so take it as 
you wish).

Cheers,
Steve


From ncoghlan at gmail.com  Mon Aug 29 23:58:05 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 Aug 2016 13:58:05 +1000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <57C4FD91.5020804@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <57C4FD91.5020804@python.org>
Message-ID: <CADiSq7fxEysuAN_NOfDj9h6CX=sx=056AmoAmW9-xdAPR8fDAw@mail.gmail.com>

On 30 August 2016 at 13:29, Steve Dower <steve.dower at python.org> wrote:
> The other discussion about OpenSSL and LTS systems is also interesting. Do
> we really expect users to take their fully functioning systems and blindly
> upgrade to a new major version of Python expecting everything to just work?
> That seems very unlikely to me, and also doesn't match my experience (but I
> can't quantify that in any useful way, so take it as you wish).

As far as I'm aware, new developers running new Windows and Mac OS X
releases from python.org and new versions of non-LTS Linux distros are
the folks most impacted by the 3-6 month post-release window where the
rest of the Python ecosystem plays catch-up to any breaking changes in
the new release - while a lot of major libraries and frameworks do
compatibility testing during the beta period, it's far from being a
universal practice.

Existing developers are able to stick with the versions they already
know work, and upgrade at a time that suits them, rather than getting
the newest version just because it's the newest version.

Cheers,
Nick.

P.S. My working assumption is that these two observations are not
unrelated phenomena ;)

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From benjamin at python.org  Tue Aug 30 01:13:54 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 29 Aug 2016 22:13:54 -0700
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <1981f645-eb17-5088-cb65-f91d0c4d9fd8@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <1472425597.2349179.708635705.4A77A4A6@webmail.messagingengine.com>
 <2EFA23B2-B60C-415A-84CA-405F601FEC62@python.org>
 <1981f645-eb17-5088-cb65-f91d0c4d9fd8@python.org>
Message-ID: <1472534034.617650.709950905.720F5089@webmail.messagingengine.com>



On Sun, Aug 28, 2016, at 22:42, Christian Heimes wrote:
> On 2016-08-29 04:38, Ned Deily wrote:
> > On Aug 28, 2016, at 19:06, Benjamin Peterson <benjamin at python.org> wrote:
> >> On Sun, Aug 28, 2016, at 13:40, Christian Heimes wrote:
> >>> Here is the deal for 2.7 to 3.5:
> >>>
> >>> 1) All versions older than 0.9.8 are completely out-of-scope and no
> >>> longer supported.
> >> +1
> >>> 2) 0.9.8 is semi-support. Python will still compile and work with 0.9.8.
> >>> However we do NOT promise that is secure to run 0.9.8. We also require a
> >>> recent version. Patch level 0.9.8zc from October 2014 is reasonable
> >>> because it comes with SCSV fallback (CVE-2014-3566).
> >> I think we should support 0.9.8 for 2.7 and drop it for 3.6.
> > 
> > Sounds good to me, too.  I think we should also not change things for 3.5.x at this point, e.g. continue to support 0.9.8 there.
> 
> 
> In my proto-PEP I'm talking about different levels of support: full,
> build-only and unsupported. Full support means that the combination of
> Python and OpenSSL versions is reasonable secure and recommended.
> 
> On the other hand build-only support doesn't come with any security
> promise. The ssl and hashlib module are source compatible with OpenSSL
> 0.9.8. You can still compile Python, do https connections but they might
> not be secure. It's "Warranty void" mode.

I'm not sure having such "support" is a good idea. If we're not able to
support a security module securely, it's probably better if it doesn't
compile at all.

From ncoghlan at gmail.com  Tue Aug 30 02:29:07 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 Aug 2016 16:29:07 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <1472534034.617650.709950905.720F5089@webmail.messagingengine.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <1472425597.2349179.708635705.4A77A4A6@webmail.messagingengine.com>
 <2EFA23B2-B60C-415A-84CA-405F601FEC62@python.org>
 <1981f645-eb17-5088-cb65-f91d0c4d9fd8@python.org>
 <1472534034.617650.709950905.720F5089@webmail.messagingengine.com>
Message-ID: <CADiSq7eyn25O-LCiaGOf6JSb0yh7HYtS6yMChCRQdLfzV_PgjQ@mail.gmail.com>

On 30 August 2016 at 15:13, Benjamin Peterson <benjamin at python.org> wrote:
> On Sun, Aug 28, 2016, at 22:42, Christian Heimes wrote:
>> In my proto-PEP I'm talking about different levels of support: full,
>> build-only and unsupported. Full support means that the combination of
>> Python and OpenSSL versions is reasonable secure and recommended.
>>
>> On the other hand build-only support doesn't come with any security
>> promise. The ssl and hashlib module are source compatible with OpenSSL
>> 0.9.8. You can still compile Python, do https connections but they might
>> not be secure. It's "Warranty void" mode.
>
> I'm not sure having such "support" is a good idea. If we're not able to
> support a security module securely, it's probably better if it doesn't
> compile at all.

We may not be able to practically support these variations directly
upstream, but that doesn't mean the particular downstream
redistributor or end user building against the older version can't -
they get to narrow their support matrix in a different way from us by
only caring about their particular platform or deployment environment.

So I don't think it makes sense to fight this particular battle right
now - it may be something we want to explore in the future as a
deliberate "You must have the ability to maintain patches against
CPython and hence presumably OpenSSL if you want to use older OpenSSL
versions" barrier to entry, but just the notion of imposing minimum
OpenSSL versions on *nix and hence potentially deprecating upstream
support for older LTS Linux releases before their distributors do is
already going to be a significant change.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From fijall at gmail.com  Tue Aug 30 04:20:28 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 30 Aug 2016 10:20:28 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAP1=2W6kHFNxXkND9QnzAAiiawnoOYGd76adARs0mBjFhnw5xg@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <nq2ikm$bge$1@blaine.gmane.org>
 <CAP1=2W6kHFNxXkND9QnzAAiiawnoOYGd76adARs0mBjFhnw5xg@mail.gmail.com>
Message-ID: <CAK5idxTEfHy3TA61+Q3whGSnrSaSxqH==M=g94yzb0E9gc3=cQ@mail.gmail.com>

On Tue, Aug 30, 2016 at 3:00 AM, Brett Cannon <brett at python.org> wrote:
>
>
> On Mon, Aug 29, 2016, 17:06 Terry Reedy <tjreedy at udel.edu> wrote:
>>
>> On 8/29/2016 5:38 PM, Brett Cannon wrote:
>>
>> > who objected to the new field did either for memory ("it adds another
>> > pointer to the struct that won't be typically used"), or for conceptual
>> > reasons ("the code object is immutable and you're proposing a mutable
>> > field"). The latter is addressed by not exposing the field in Python and
>>
>> Am I correct is thinking that you will also not add the new field as an
>> argument to PyCode_New?
>
>
> Correct.
>
>>
>>  > clearly stating that code should never expect the field to be filled.
>>
>> I interpret this as "The only code that should access the field should
>> be code that put something there."
>
>
> Yep, seems like a reasonable rule to follow.
>
> -brett

How do we make sure that multuple tools don't stomp on each other?

From victor.stinner at gmail.com  Tue Aug 30 04:21:09 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Aug 2016 10:21:09 +0200
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
Message-ID: <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>

Le 30 ao?t 2016 03:10, "Nick Coghlan" <ncoghlan at gmail.com> a ?crit :
> However, this view is also why I don't agree with being aggressive in
> making this behaviour the default on Windows - I think we should make
> it readily available as a provisional feature through a single
> cross-platform command line switch and environment setting (e.g. "-X
> utf8" and "PYTHONASSUMEUTF8") so folks that need it can readily opt in
> to it,

I'm sorry, but I should have start from this point.

Modifyng the default or adding an option are completely different. I like
the idea of adding a -X utf8 option on Windows. If it's an opt-in option,
the developer takes the responsabilty of any possible backward incompatible
change and plays the Unicode dance when handling input/output data with
other applications.

My long email tries to explain why I consider that modifying the default in
3.6 is a strong NO for me.

> but we can defer making it the default until 3.7 after folks
> have had a full release cycle's worth of experience with it in the
> wild.

If Steve changes its project to add an option but don't change the default,
I will help to make it happen before 3.6 and help to implement the UNIX
part. It would even make it stronger if the option is "portable", even if
the exact semantic is differnent between UNIX and Windows.

If the default doesn't change, I don't think that a PEP is required.

Later when we will get enough feedback, we will be able to decide to drop
the option (if it was a very bad idea because of very bad feedback), or
even make it the default on a platform (Windows).

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/13a06b74/attachment.html>

From tjreedy at udel.edu  Tue Aug 30 05:21:40 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 30 Aug 2016 05:21:40 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CADiSq7fF_yC-EgaaBjgVe93LNGSmRW0adLA0TL1Y4PU5VDZXYQ@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <20160829185600.1d27bc02@subdivisions.wooz.org>
 <CADiSq7fF_yC-EgaaBjgVe93LNGSmRW0adLA0TL1Y4PU5VDZXYQ@mail.gmail.com>
Message-ID: <nq3j75$e5h$1@blaine.gmane.org>

On 8/29/2016 10:59 PM, Nick Coghlan wrote:

> By contrast (and assuming I understand the situation correctly), the
> Windows build is already set up around the assumption that you'll need
> to build OpenSSL yourself.

If one installs a minimal svn client and passes -e to Zack's wonderful 
built.bat, current versions of external dependencies are automatically 
downloaded and compiled with the result placed where needed.  So I did 
nothing more to have OpenSSL updated to 1.0.2h last June. This is too 
easy to say I 'built it myself'.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Tue Aug 30 05:24:27 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 30 Aug 2016 05:24:27 -0400
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAK5idxTEfHy3TA61+Q3whGSnrSaSxqH==M=g94yzb0E9gc3=cQ@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <nq2ikm$bge$1@blaine.gmane.org>
 <CAP1=2W6kHFNxXkND9QnzAAiiawnoOYGd76adARs0mBjFhnw5xg@mail.gmail.com>
 <CAK5idxTEfHy3TA61+Q3whGSnrSaSxqH==M=g94yzb0E9gc3=cQ@mail.gmail.com>
Message-ID: <nq3jcc$e5h$2@blaine.gmane.org>

On 8/30/2016 4:20 AM, Maciej Fijalkowski wrote:
> On Tue, Aug 30, 2016 at 3:00 AM, Brett Cannon <brett at python.org> wrote:
>>
>>
>> On Mon, Aug 29, 2016, 17:06 Terry Reedy <tjreedy at udel.edu> wrote:
>>>
>>> On 8/29/2016 5:38 PM, Brett Cannon wrote:
>>>
>>>> who objected to the new field did either for memory ("it adds another
>>>> pointer to the struct that won't be typically used"), or for conceptual
>>>> reasons ("the code object is immutable and you're proposing a mutable
>>>> field"). The latter is addressed by not exposing the field in Python and
>>>
>>> Am I correct is thinking that you will also not add the new field as an
>>> argument to PyCode_New?
>>
>>
>> Correct.
>>
>>>
>>>  > clearly stating that code should never expect the field to be filled.
>>>
>>> I interpret this as "The only code that should access the field should
>>> be code that put something there."
>>
>>
>> Yep, seems like a reasonable rule to follow.
>>
>> -brett
>
> How do we make sure that multuple tools don't stomp on each other?
>
AFAIK, we can't.  The multiple tool people will have to work that out, 
or document incompatibilities between tools.

-- 
Terry Jan Reedy


From christian at python.org  Tue Aug 30 06:28:11 2016
From: christian at python.org (Christian Heimes)
Date: Tue, 30 Aug 2016 12:28:11 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAP1=2W5gm0V7Coc2269NEpUWV0dQj33DnQV5yp-ju5d5_xd+Cw@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <723bd0e3-d776-23b4-a9ca-7d4841be521d@python.org>
 <CAP7+vJ+BroUtkpi4kWiw8yP8VN57mY7YwrRR1Z=scutHbmWhHQ@mail.gmail.com>
 <CAP1=2W5gm0V7Coc2269NEpUWV0dQj33DnQV5yp-ju5d5_xd+Cw@mail.gmail.com>
Message-ID: <nq3n3q$9jn$1@blaine.gmane.org>

On 2016-08-30 01:14, Brett Cannon wrote:
> So the struct in question can be found at
> https://github.com/python/cpython/blob/2d264235f6e066611b412f7c2e1603866e0f7f1b/Include/code.h#L10 .
> The official docs say the fields can be changed at any time, so
> re-arranging them shouldn't break any ABI compatibility promises:
> https://docs.python.org/3/c-api/code.html#c.PyCodeObject . Would
> grouping all the fields of the same type together, sorting them by
> individual field size (i.e. PyObject*, void*, int, unsigned char*), and
> then adding the co_extra field at the end of the grouping of PyObject *
> fields do what you're suggesting?

You don't have to resort them all, just move co_firstlineno after
co_flags, so all int fields are together. Pointers are typically
alignment to multiple of 64 on a 64bit machine. In its current shape
PyCodeObject is padded with two unused areas of 32bit: 5 * int32 + 32
bits of padding, 9 * pointers (64 bits each), 1 * int32 + another 32
bits of padding, 3 * pointers. When you move co_firstlineno, you fill in
the gap.

Christian



From storchaka at gmail.com  Tue Aug 30 08:13:50 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 30 Aug 2016 15:13:50 +0300
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <npt414$mdm$1@blaine.gmane.org>
References: <npt414$mdm$1@blaine.gmane.org>
Message-ID: <nq3t9u$9cp$1@blaine.gmane.org>

On 28.08.16 01:25, Terry Reedy wrote:
> 0. Do nothing.

The problem is not in pathological __index__. The problem is in 
executing Python code and releasing GIL. In multithread production code 
one thread can read a slice when other thread modifies a collection. In 
very very rare case it causes a crash (or worse, a corruption of data). 
We shouldn't left it as is.

> 1. Detect length change and raise.

It would be simpler solution. But I afraid that this can break 
third-party code that "just works" now. For example slicing a list "just 
works" if step is 1. It can return not what the author expected if a 
list grows, but it never crashes, and existing code can depends on 
current behavior. This solution is not applicable in maintained versions.

> 2. Retrieve length after any possible changes and proceed as normal.

This behavior looks the most expectable to me, but needs more work.

> B. Add PySlice_GetIndicesEx2 (or ExEx?), which would receive *collection
> instead of length, so the length could be retrieved after the __index__
> calls.  Change calls. Deprecate PySlice_GetIndicesEx.

This is not always possible. The limit for a slice is not always the 
length of the collection (see multidimensional arrays). And how to 
determine the length? Call __len__? It can be overridden in Python, this 
causes releasing GIL, switching to other thread and modifying the 
collection. The original problem is returned.

> And what versions should be patched?

Since this is heisenbug that can cause a crash, I think we should apply 
some solutions for all versions. But in develop version we a free to 
introduce small incompatibility.

I prefer 2A in maintained versions (may be including 3.3 and 3.4) and 2A 
or 1A in 3.6.



From dimaqq at gmail.com  Tue Aug 30 08:31:41 2016
From: dimaqq at gmail.com (Dima Tisnek)
Date: Tue, 30 Aug 2016 14:31:41 +0200
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <nq3t9u$9cp$1@blaine.gmane.org>
References: <npt414$mdm$1@blaine.gmane.org> <nq3t9u$9cp$1@blaine.gmane.org>
Message-ID: <CAGGBzXKkfRdgaxvBEXQyXTZe1e2jZVAhySz1a3q1_mUonmM24w@mail.gmail.com>

On 30 August 2016 at 14:13, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 1. Detect length change and raise.
>
>
> It would be simpler solution. But I afraid that this can break third-party
> code that "just works" now. For example slicing a list "just works" if step
> is 1. It can return not what the author expected if a list grows, but it
> never crashes, and existing code can depends on current behavior. This
> solution is not applicable in maintained versions.

Serhiy,

If dictionary is iterated in thread1 while thread2 changes the
dictionary, thread1 currently raises RuntimeError.

Would cloning current dict behaviour to slice with overridden
__index__ make sense?


I'd argue 3rd party code depends on slicing not to raise an exception,
is same as 3rd party code depending on dict iteration not to raise and
exception; If same container may be concurrently used in another
thread, then 3rd party code is actually buggy. It's OK to break such
code.


Just my 2c.

From fijall at gmail.com  Tue Aug 30 09:11:25 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 30 Aug 2016 15:11:25 +0200
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <CAGGBzXKkfRdgaxvBEXQyXTZe1e2jZVAhySz1a3q1_mUonmM24w@mail.gmail.com>
References: <npt414$mdm$1@blaine.gmane.org> <nq3t9u$9cp$1@blaine.gmane.org>
 <CAGGBzXKkfRdgaxvBEXQyXTZe1e2jZVAhySz1a3q1_mUonmM24w@mail.gmail.com>
Message-ID: <CAK5idxTqgSv9_4+PZxuyuxoCt-GW4hu81s3HvhQU8gZZ_fkP2w@mail.gmail.com>

On Tue, Aug 30, 2016 at 2:31 PM, Dima Tisnek <dimaqq at gmail.com> wrote:
> On 30 August 2016 at 14:13, Serhiy Storchaka <storchaka at gmail.com> wrote:
>>> 1. Detect length change and raise.
>>
>>
>> It would be simpler solution. But I afraid that this can break third-party
>> code that "just works" now. For example slicing a list "just works" if step
>> is 1. It can return not what the author expected if a list grows, but it
>> never crashes, and existing code can depends on current behavior. This
>> solution is not applicable in maintained versions.
>
> Serhiy,
>
> If dictionary is iterated in thread1 while thread2 changes the
> dictionary, thread1 currently raises RuntimeError.
>
> Would cloning current dict behaviour to slice with overridden
> __index__ make sense?
>
>
> I'd argue 3rd party code depends on slicing not to raise an exception,
> is same as 3rd party code depending on dict iteration not to raise and
> exception; If same container may be concurrently used in another
> thread, then 3rd party code is actually buggy. It's OK to break such
> code.
>
>
> Just my 2c.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

I'm with Dima here.

It's more complicated - if the third party rely on the code working
when one thread slices while the other thread modifies that gives
implicit atomicity requirements. Those specific requirements are very
hard to maintain across the python versions and python
implementations. Replicating the exact CPython behavior (for each
CPython version too!) is a major nightmare for such specific
scenarios.

I propose the following:

* we raise an error if detected

-or-

* we define the exact behavior what it means to modify the collection
in one thread while the other is slicing it (what do you get? what are
the guarantees? does it also apply if the list is resized?)

From stefan at bytereef.org  Tue Aug 30 09:17:01 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 30 Aug 2016 15:17:01 +0200
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <CAK5idxTqgSv9_4+PZxuyuxoCt-GW4hu81s3HvhQU8gZZ_fkP2w@mail.gmail.com>
References: <npt414$mdm$1@blaine.gmane.org> <nq3t9u$9cp$1@blaine.gmane.org>
 <CAGGBzXKkfRdgaxvBEXQyXTZe1e2jZVAhySz1a3q1_mUonmM24w@mail.gmail.com>
 <CAK5idxTqgSv9_4+PZxuyuxoCt-GW4hu81s3HvhQU8gZZ_fkP2w@mail.gmail.com>
Message-ID: <20160830131700.GA5075@bytereef.org>

On Tue, Aug 30, 2016 at 03:11:25PM +0200, Maciej Fijalkowski wrote:
> It's more complicated - if the third party rely on the code working
> when one thread slices while the other thread modifies that gives
> implicit atomicity requirements. Those specific requirements are very
> hard to maintain across the python versions and python
> implementations. Replicating the exact CPython behavior (for each
> CPython version too!) is a major nightmare for such specific
> scenarios.
> 
> I propose the following:
> 
> * we raise an error if detected

+1


Stefan Krah


From steve.dower at python.org  Tue Aug 30 10:31:26 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 30 Aug 2016 07:31:26 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
Message-ID: <E1bek4v-0005pO-5D@se2-syd.hostedmail.net.au>

As I've said before, on Windows this is a compatibility hack to make code written for other platforms work. Making the user opt in is not fair, and does not help improve the libraries that need it, because few people will change their library to work with a non default option.

The "developer" I'm concerned about doesn't need to turn this on - bytes work just about fine on POSIX (if you don't inspect the contents). It's the random user on Windows who installed their library that has the problem. They don't know the fix, and may not know how to apply it (e.g. if it's their Jupyter notebook that won't find one of their files - no obvious command line options here).

Any system that requires communication between two different versions of Python must have install instructions (if it's public) or someone who maintains it. It won't magically break without an upgrade, and it should not get an upgrade without testing. The environment variable is available for this kind of scenario, though I'd hope the testing occurs during beta and it gets fixed by the time we release.

Changing the locale encoding is something I'm quite happy to back out of. It's already easy enough for the developer to specify the encoding when opening a file, or to wrap open() and change their own default. But developers cannot change the encoding used by the os module, which is why we need to do it.

Cheers,
Steve

Top-posted from my Windows Phone

-----Original Message-----
From: "Victor Stinner" <victor.stinner at gmail.com>
Sent: ?8/?30/?2016 1:21
To: "Nick Coghlan" <ncoghlan at gmail.com>
Cc: "Steve Dower" <steve.dower at python.org>; "Python Dev" <python-dev at python.org>
Subject: Re: [Python-Dev] File system path encoding on Windows

Le 30 ao?t 2016 03:10, "Nick Coghlan" <ncoghlan at gmail.com> a ?crit :
> However, this view is also why I don't agree with being aggressive in
> making this behaviour the default on Windows - I think we should make
> it readily available as a provisional feature through a single
> cross-platform command line switch and environment setting (e.g. "-X
> utf8" and "PYTHONASSUMEUTF8") so folks that need it can readily opt in
> to it,
I'm sorry, but I should have start from this point.
Modifyng the default or adding an option are completely different. I like the idea of adding a -X utf8 option on Windows. If it's an opt-in option, the developer takes the responsabilty of any possible backward incompatible change and plays the Unicode dance when handling input/output data with other applications.
My long email tries to explain why I consider that modifying the default in 3.6 is a strong NO for me.
> but we can defer making it the default until 3.7 after folks
> have had a full release cycle's worth of experience with it in the
> wild.
If Steve changes its project to add an option but don't change the default, I will help to make it happen before 3.6 and help to implement the UNIX part. It would even make it stronger if the option is "portable", even if the exact semantic is differnent between UNIX and Windows.
If the default doesn't change, I don't think that a PEP is required.
Later when we will get enough feedback, we will be able to decide to drop the option (if it was a very bad idea because of very bad feedback), or even make it the default on a platform (Windows).
Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/864dca85/attachment-0001.html>

From storchaka at gmail.com  Tue Aug 30 10:41:29 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 30 Aug 2016 17:41:29 +0300
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <CAGGBzXKkfRdgaxvBEXQyXTZe1e2jZVAhySz1a3q1_mUonmM24w@mail.gmail.com>
References: <npt414$mdm$1@blaine.gmane.org> <nq3t9u$9cp$1@blaine.gmane.org>
 <CAGGBzXKkfRdgaxvBEXQyXTZe1e2jZVAhySz1a3q1_mUonmM24w@mail.gmail.com>
Message-ID: <nq45uo$f7c$1@blaine.gmane.org>

On 30.08.16 15:31, Dima Tisnek wrote:
> On 30 August 2016 at 14:13, Serhiy Storchaka <storchaka at gmail.com> wrote:
>>> 1. Detect length change and raise.
>>
>>
>> It would be simpler solution. But I afraid that this can break third-party
>> code that "just works" now. For example slicing a list "just works" if step
>> is 1. It can return not what the author expected if a list grows, but it
>> never crashes, and existing code can depends on current behavior. This
>> solution is not applicable in maintained versions.
>
> Serhiy,
>
> If dictionary is iterated in thread1 while thread2 changes the
> dictionary, thread1 currently raises RuntimeError.
>
> Would cloning current dict behaviour to slice with overridden
> __index__ make sense?

No, these are different things. The problem with dict iterating is 
unavoidable, but slicing can be defined consistently (as described by 
Terry in option 2). Changing a dict can change the order and invalidates 
iterators (except trivial cases of just created or finished iterators). 
But slicing can be atomic (and it is atomic de facto in many cases), we 
just should call all __index__-es before looking on a sequence.

> I'd argue 3rd party code depends on slicing not to raise an exception,
> is same as 3rd party code depending on dict iteration not to raise and
> exception; If same container may be concurrently used in another
> thread, then 3rd party code is actually buggy. It's OK to break such
> code.

We shouldn't break third-party code in maintained releases. De facto 
slicing is atomic now in many cases, and it is nowhere documented that 
it is not atomic. The problem only with using broken by design 
PySlice_GetIndicesEx() in CPython. If CPython would implemented without 
PySlice_GetIndicesEx() (with more cumbersome code), it is likely this 
issue wouldn't be raised.



From victor.stinner at gmail.com  Tue Aug 30 11:06:06 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Aug 2016 17:06:06 +0200
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
Message-ID: <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>

2016-08-30 16:31 GMT+02:00 Steve Dower <steve.dower at python.org>:
> It's the
> random user on Windows who installed their library that has the problem.
> They don't know the fix, and may not know how to apply it (e.g. if it's
> their Jupyter notebook that won't find one of their files - no obvious
> command line options here).

There is already a DeprecationWarning. Sadly, it's hidden by default:
you need a debug build of Python or more simply to pass -Wd command
line option.

Maybe we should make this warning (Deprecation warning on bytes paths)
visible by default, or add a new warning suggesting to enable -X utf8
the first time a Python function gets a byte string (like a filename)?


> Any system that requires communication between two different versions of
> Python must have install instructions (if it's public) or someone who
> maintains it. It won't magically break without an upgrade, and it should not
> get an upgrade without testing. The environment variable is available for
> this kind of scenario, though I'd hope the testing occurs during beta and it
> gets fixed by the time we release.

I disagree that breaking backward compatibility is worth it. Most
users don't care of Unicode since their application already "just
works well" for their use case.

Having to set an env var to "repair" their app to be able to upgrade
Python is not really convenient.

Victor

From ivo.bellinsalarin at gmail.com  Tue Aug 30 09:41:43 2016
From: ivo.bellinsalarin at gmail.com (Ivo Bellin Salarin)
Date: Tue, 30 Aug 2016 13:41:43 +0000
Subject: [Python-Dev] Lib/http/client.py: could it return an OSError with
 the current response?
Message-ID: <CAPc4eF8da8eYrzWcZ7wxDn1NDAFW8VgPtZDm07+iYwvgfyULCw@mail.gmail.com>

Hi everybody,

Sorry for bothering you, this is my first post to the python-dev ML.

While using requests to tunnel a request via a proxy requiring user
authentication, I have seen that httplib (
https://hg.python.org/cpython/file/3.5/Lib/http/client.py#l831) raises the
message returned by the proxy, along with its status code (407) without
including the proxy response. This one could be very interesting to the
consumer, since it could contain some useful headers (like the supported
authentication schemes).

Would it be possible to change the http/client.py behavior in order to
raise an exception including the whole response?

If you don't see any problem with my proposal, how can I propose a pull
request? :-)

Thanks in advance,
Ivo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/8a284073/attachment.html>

From guido at python.org  Tue Aug 30 11:24:12 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Aug 2016 08:24:12 -0700
Subject: [Python-Dev] Lib/http/client.py: could it return an OSError
 with the current response?
In-Reply-To: <CAPc4eF8da8eYrzWcZ7wxDn1NDAFW8VgPtZDm07+iYwvgfyULCw@mail.gmail.com>
References: <CAPc4eF8da8eYrzWcZ7wxDn1NDAFW8VgPtZDm07+iYwvgfyULCw@mail.gmail.com>
Message-ID: <CAP7+vJ+0oSCqBSnLg3cRUM2ir5zvOxdaza0fQnqbeZT+NyCFdQ@mail.gmail.com>

If you can do it without breaking existing code that doesn't expect
the extra information, please go ahead! For things like this it is
typically best to go straight to the bug tracker (bugs.python.org)
rather than asking the list first -- if the issue turns out to be
controversial or mysterious it's always possible to go to the list
later at the advice of the person doing triage in the tracker.

On Tue, Aug 30, 2016 at 6:41 AM, Ivo Bellin Salarin
<ivo.bellinsalarin at gmail.com> wrote:
> Hi everybody,
>
> Sorry for bothering you, this is my first post to the python-dev ML.
>
> While using requests to tunnel a request via a proxy requiring user
> authentication, I have seen that httplib
> (https://hg.python.org/cpython/file/3.5/Lib/http/client.py#l831) raises the
> message returned by the proxy, along with its status code (407) without
> including the proxy response. This one could be very interesting to the
> consumer, since it could contain some useful headers (like the supported
> authentication schemes).
>
> Would it be possible to change the http/client.py behavior in order to raise
> an exception including the whole response?
>
> If you don't see any problem with my proposal, how can I propose a pull
> request? :-)
>
> Thanks in advance,
> Ivo
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From solipsis at pitrou.net  Tue Aug 30 12:00:35 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Aug 2016 18:00:35 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
Message-ID: <20160830180035.190c3383@fsol>

On Sun, 28 Aug 2016 22:40:11 +0200
Christian Heimes <christian at python.org> wrote:
> 
> Here is the deal for 2.7 to 3.5:
> 
> 1) All versions older than 0.9.8 are completely out-of-scope and no
> longer supported.
> 
> 2) 0.9.8 is semi-support. Python will still compile and work with 0.9.8.
> However we do NOT promise that is secure to run 0.9.8. We also require a
> recent version. Patch level 0.9.8zc from October 2014 is reasonable
> because it comes with SCSV fallback (CVE-2014-3566).
> 
> 3) 1.0.0 is irrelevant. Users are either stuck on 0.9.8 or are able to
> upgrade to 1.0.1+. Let's not support it.
> 
> 4) 1.0.1 is discouraged but still supported until its EOL.
> 
> 5) 1.0.2 is the recommend version.
> 
> 6) 1.1 support will be added by #26470 soon.
> 
> 7) LibreSSL 2.3 is supported but with a slightly limited feature set.

Can you expand briefly how "limited" the feature set is?  Does it only
disable some arcane features, so that e.g. asyncio + TLS supports works
fine?

Other than that, it all sounds good to me.

Regards

Antoine.



From solipsis at pitrou.net  Tue Aug 30 12:07:29 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Aug 2016 18:07:29 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
Message-ID: <20160830180729.63540a92@fsol>

On Mon, 29 Aug 2016 21:38:19 +0000
Brett Cannon <brett at python.org> wrote:
> For quick background for those that don't remember, part of PEP 523
> proposed adding a co_extra field to code objects along with making the
> frame evaluation function pluggable (
> https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The idea
> was that things like JITs and debuggers could use the field as a scratch
> space of sorts to store data related to the code object. People who
> objected to the new field did either for memory ("it adds another pointer
> to the struct that won't be typically used"), or for conceptual reasons
> ("the code object is immutable and you're proposing a mutable field"). The
> latter is addressed by not exposing the field in Python and clearly stating
> that code should never expect the field to be filled.

Just a question: Maciej mentioned the field may be useful for vmprof.
That's already two potential users (vmprof and Pyjion) for a single
field.  OTOH, the PEP says:

"""It is not recommended that multiple users attempt to use the co_extra
simultaneously. While a dictionary could theoretically be set to the
field and various users could use a key specific to the project, there
is still the issue of key collisions as well as performance degradation
from using a dictionary lookup on every frame evaluation. Users are
expected to do a type check to make sure that the field has not been
previously set by someone else."""

Does it mean that, in the event vmprof comes in and changes the field,
Pyjion will have to recompile the function the next time it comes to
execute it?

And, conversely, if Pyjion compiles a function while vmprof is enabled,
vmprof will lose timing information (or whatever else, because I'm not
sure what vmprof plans to store thre) for that code object?

Regards

Antoine.



From brett at python.org  Tue Aug 30 13:14:31 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Aug 2016 17:14:31 +0000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <20160830180729.63540a92@fsol>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
Message-ID: <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>

On Tue, 30 Aug 2016 at 09:08 Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 29 Aug 2016 21:38:19 +0000
> Brett Cannon <brett at python.org> wrote:
> > For quick background for those that don't remember, part of PEP 523
> > proposed adding a co_extra field to code objects along with making the
> > frame evaluation function pluggable (
> > https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The
> idea
> > was that things like JITs and debuggers could use the field as a scratch
> > space of sorts to store data related to the code object. People who
> > objected to the new field did either for memory ("it adds another pointer
> > to the struct that won't be typically used"), or for conceptual reasons
> > ("the code object is immutable and you're proposing a mutable field").
> The
> > latter is addressed by not exposing the field in Python and clearly
> stating
> > that code should never expect the field to be filled.
>
> Just a question: Maciej mentioned the field may be useful for vmprof.
> That's already two potential users (vmprof and Pyjion) for a single
> field.


PTVS has also said they would find it useful for debugging.


>   OTOH, the PEP says:
>
> """It is not recommended that multiple users attempt to use the co_extra
> simultaneously. While a dictionary could theoretically be set to the
> field and various users could use a key specific to the project, there
> is still the issue of key collisions as well as performance degradation
> from using a dictionary lookup on every frame evaluation. Users are
> expected to do a type check to make sure that the field has not been
> previously set by someone else."""
>
> Does it mean that, in the event vmprof comes in and changes the field,
> Pyjion will have to recompile the function the next time it comes to
> execute it?
>

As of right now Pyjion simply doesn't JIT the function.


>
> And, conversely, if Pyjion compiles a function while vmprof is enabled,
> vmprof will lose timing information (or whatever else, because I'm not
> sure what vmprof plans to store there) for that code object?
>

Depends on what vmprof chooses to do. Since the data is designed to be
disposable it could decide it should always take precedence and overwrite
the data if someone beat it to using the field. Basically I don't think we
want co_extra1, co_extra2, etc. But we don't want to require a dict either
as that kills performance. Using a list where users could push on objects
might work, but I have no clue what that would do to perf as you would have
to still needlessly search the list when only one piece of code uses the
field.

Basically I don't see a good way to make a general solution for people who
want to use the field simultaneously, so tools that use the field will need
to be clear on how they choose to handle the situation, such "as we use it
if it isn't set" or "we always use it no matter what". This isn't a perfect
solution in all cases and I think that's just going to have to be the way
it is.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/377aa98d/attachment.html>

From brett at python.org  Tue Aug 30 13:20:03 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Aug 2016 17:20:03 +0000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAK5idxTEfHy3TA61+Q3whGSnrSaSxqH==M=g94yzb0E9gc3=cQ@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <nq2ikm$bge$1@blaine.gmane.org>
 <CAP1=2W6kHFNxXkND9QnzAAiiawnoOYGd76adARs0mBjFhnw5xg@mail.gmail.com>
 <CAK5idxTEfHy3TA61+Q3whGSnrSaSxqH==M=g94yzb0E9gc3=cQ@mail.gmail.com>
Message-ID: <CAP1=2W7nTpP9A9NvgCFX+AgMPNDme0JvtyaeeTMBou-BttKFNQ@mail.gmail.com>

On Tue, 30 Aug 2016 at 01:20 Maciej Fijalkowski <fijall at gmail.com> wrote:

> On Tue, Aug 30, 2016 at 3:00 AM, Brett Cannon <brett at python.org> wrote:
> >
> >
> > On Mon, Aug 29, 2016, 17:06 Terry Reedy <tjreedy at udel.edu> wrote:
> >>
> >> On 8/29/2016 5:38 PM, Brett Cannon wrote:
> >>
> >> > who objected to the new field did either for memory ("it adds another
> >> > pointer to the struct that won't be typically used"), or for
> conceptual
> >> > reasons ("the code object is immutable and you're proposing a mutable
> >> > field"). The latter is addressed by not exposing the field in Python
> and
> >>
> >> Am I correct is thinking that you will also not add the new field as an
> >> argument to PyCode_New?
> >
> >
> > Correct.
> >
> >>
> >>  > clearly stating that code should never expect the field to be filled.
> >>
> >> I interpret this as "The only code that should access the field should
> >> be code that put something there."
> >
> >
> > Yep, seems like a reasonable rule to follow.
> >
> > -brett
>
> How do we make sure that multuple tools don't stomp on each other?
>

It will be up to the tool. For Pyjion we just don't use the field if
someone else is using it, so if vmprof chooses to take precedence then it
can. Otherwise we can work out some common practice like if the field is
set and it isn't an object you put there then create a list, push on what
was already there, push on what you want to add, and set the field to the
list. That lets us do a type-check for the common case of only one object
being set, and in the odd case of the list things don't fail as you can
search the list for your object while acknowledging performance will suffer
(or we use a dict, I don't care really as long as we don't require a
storage data structure for the field in the single user case).

My point is that we can figure this out among Pyjion, PTVS, and vmprof if
we are the first users and update the PEP accordingly as guidance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/d9dfa820/attachment.html>

From solipsis at pitrou.net  Tue Aug 30 13:30:51 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Aug 2016 19:30:51 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
Message-ID: <20160830193051.532e8d9a@fsol>

On Tue, 30 Aug 2016 17:14:31 +0000
Brett Cannon <brett at python.org> wrote:
> 
> Depends on what vmprof chooses to do. Since the data is designed to be
> disposable it could decide it should always take precedence and overwrite
> the data if someone beat it to using the field. Basically I don't think we
> want co_extra1, co_extra2, etc. But we don't want to require a dict either
> as that kills performance. Using a list where users could push on objects
> might work, but I have no clue what that would do to perf as you would have
> to still needlessly search the list when only one piece of code uses the
> field.

Perhaps a list would work indeed.  Realistically, if there are at most
2-3 users of the field at any given time (and most probably only one or
zero), a simple type check (by pointer equality) on each list item may
be sufficient.

Speaking about Numba, we don't have any planned use for the field, so I
can't really give any further suggestion.

Regards

Antoine.

From brett at python.org  Tue Aug 30 13:35:35 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Aug 2016 17:35:35 +0000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <20160830193051.532e8d9a@fsol>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
Message-ID: <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>

On Tue, 30 Aug 2016 at 10:32 Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Tue, 30 Aug 2016 17:14:31 +0000
> Brett Cannon <brett at python.org> wrote:
> >
> > Depends on what vmprof chooses to do. Since the data is designed to be
> > disposable it could decide it should always take precedence and overwrite
> > the data if someone beat it to using the field. Basically I don't think
> we
> > want co_extra1, co_extra2, etc. But we don't want to require a dict
> either
> > as that kills performance. Using a list where users could push on objects
> > might work, but I have no clue what that would do to perf as you would
> have
> > to still needlessly search the list when only one piece of code uses the
> > field.
>
> Perhaps a list would work indeed.  Realistically, if there are at most
> 2-3 users of the field at any given time (and most probably only one or
> zero), a simple type check (by pointer equality) on each list item may
> be sufficient.
>

Let's see what Maciej says, but we could standardize on switching the field
to a list when a conflict of usage is detected so the common case in the
frame eval function is checking for your own type, and if that fails then
doing a PyList_CheckExact() and look for your object, otherwise make a list
and move over to that for everyone to use. A little bit more code, but it's
simple code and takes care of conflicts only when it calls for it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/965e23f9/attachment.html>

From ncoghlan at gmail.com  Tue Aug 30 13:42:49 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 03:42:49 +1000
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <npt414$mdm$1@blaine.gmane.org>
References: <npt414$mdm$1@blaine.gmane.org>
Message-ID: <CADiSq7ehQy-VJdx+t8KjjgUd2WYY3oM+6WWJ90DYJr7hdkVweQ@mail.gmail.com>

On 28 August 2016 at 08:25, Terry Reedy <tjreedy at udel.edu> wrote:
> Slicing can be made to malfunction and even crash with an 'evil' __index__
> method. https://bugs.python.org/issue27867
>
> The crux of the problem is this: PySlice_GetIndicesEx
> receives a slice object and a sequence length.  Calling __index__ on the
> start, stop, and step components can mutate the sequence and invalidate the
> length.  Adjusting the int values of start and stop according to an invalid
> length (in particular, one that is too long) will result in invalid results
> or a crash.
>
> Possible actions -- very briefly.  For more see end of
> https://bugs.python.org/issue27867?@ok_message=msg 273801
> 0. Do nothing.
> 1. Detect length change and raise.
> 2. Retrieve length after any possible changes and proceed as normal.
>
> Possible implementation strategies for 1. and 2.
> A. Change all functions that call PySlice_GetIndicesEx.
> B. Add PySlice_GetIndicesEx2 (or ExEx?), which would receive *collection
> instead of length, so the length could be retrieved after the __index__
> calls.  Change calls. Deprecate PySlice_GetIndicesEx.

Given Serhiy's clarification that this is primarily a thread safety
problem, I'm more supportive of the "PySlice_GetIndicesForObject"
approach (since that can call all the __index__ methods first, leaving
the final __len__ call as the only problematic case).

However, given the observation that __len__ can also release the GIL,
I'm not clear on how 2A is supposed to work - a poorly timed thread
switch means there's always going to be a risk of len(obj) returning
outdated information if a container implemented in Python is being
mutated concurrently from different threads, so what can be done
differently in the calling functions that couldn't be done in a new
API that accepted the container reference?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Tue Aug 30 13:48:13 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Aug 2016 19:48:13 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
 <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
Message-ID: <20160830194813.0cbb261b@fsol>

On Tue, 30 Aug 2016 17:35:35 +0000
Brett Cannon <brett at python.org> wrote:
> >
> > Perhaps a list would work indeed.  Realistically, if there are at most
> > 2-3 users of the field at any given time (and most probably only one or
> > zero), a simple type check (by pointer equality) on each list item may
> > be sufficient.
> >  
> 
> Let's see what Maciej says, but we could standardize on switching the field
> to a list when a conflict of usage is detected so the common case in the
> frame eval function is checking for your own type, and if that fails then
> doing a PyList_CheckExact() and look for your object, otherwise make a list
> and move over to that for everyone to use. A little bit more code, but it's
> simple code and takes care of conflicts only when it calls for it.

That's a bit obscure and confusing, though (I *think* the weakref module
uses a similar kludge in some place).  If you want to iterate on it you
have to write some bizarre macro to share the loop body between the two
different code-paths (list / non-list), or some equally tedious
function-pointer-based code.

Why not make it always a list?  List objects are reasonably cheap in
memory and access time... (unlike dicts)

Regards

Antoine.

From eric at trueblade.com  Tue Aug 30 13:55:07 2016
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 30 Aug 2016 13:55:07 -0400
Subject: [Python-Dev] Changes to PEP 498 (f-strings)
Message-ID: <119e0950-845f-2fc8-f1b4-30b9f4c00801@trueblade.com>

After a long discussion on python-ideas (starting at
https://mail.python.org/pipermail/python-ideas/2016-August/041727.html)
I'm proposing the following change to PEP 498: backslashes inside
brackets will be disallowed. The point of this is to disallow convoluted
code like:

>>> d = {'a': 4}
>>> f'{d[\'a\']}'
'4'

In addition, I'll disallow escapes to be used for brackets, as in:

>>> f'\x7bd["a"]}'
'4'

(where chr(0x7b) ==  "{").

Because we're so close to 3.6 beta 1, my plan is to:

1. Modify the PEP to reflect these restrictions.
2. Modify the code to prevent _any_ backslashes inside f-strings.

This is a more restrictive change than the PEP will describe, but it's
much easier to implement. After beta 1, and hopefully before beta 2, I
will implement the restrictions as I've outlined above (and as they will
be documented in the PEP). The net effects are:

a. Some code that works in the alphas won't work in beta 1. I'll
document this.
b. All code that's valid in beta 1 will work in beta 2, and some
f-strings that are syntax errors in beta 1 will work in beta 2.

I've discussed this issue with Ned and Guido, who are okay with these
changes.

The python-ideas thread I referenced above has some discussion about
further changes to f-strings. Those proposals are outside the scope of
3.6, but the changes I'm putting forth here will allow for those
additional changes, should we decide to make them. That's a discussion
for 3.7, however.

I'm sending this email out just to notify people of this upcoming
change. I hope this won't generate much discussion. If you feel the need
to discuss this issue further, please use the python-ideas thread (where
some people are already ignoring it!).

Eric.

From steve.dower at python.org  Tue Aug 30 14:04:45 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 30 Aug 2016 11:04:45 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
Message-ID: <57C5CABD.2090401@python.org>

On 30Aug2016 0806, Victor Stinner wrote:
> 2016-08-30 16:31 GMT+02:00 Steve Dower <steve.dower at python.org>:
>> It's the
>> random user on Windows who installed their library that has the problem.
>> They don't know the fix, and may not know how to apply it (e.g. if it's
>> their Jupyter notebook that won't find one of their files - no obvious
>> command line options here).
>
> There is already a DeprecationWarning. Sadly, it's hidden by default:
> you need a debug build of Python or more simply to pass -Wd command
> line option.

It also only appears on Windows, so developers who do the right thing on 
POSIX never find out about it. Your average user isn't going to see it - 
they'll just see the OSError when their file is not found due to the 
lossy encoding.

> Maybe we should make this warning (Deprecation warning on bytes paths)
> visible by default, or add a new warning suggesting to enable -X utf8
> the first time a Python function gets a byte string (like a filename)?

The more important thing in my opinion is to make it visible on all 
platforms, regardless of whether bytes paths are suitable or not. But 
this will probably be seen as hostile by the majority of open-source 
Python developers, which is why I'd rather just quietly fix the 
incompatibility.

>> Any system that requires communication between two different versions of
>> Python must have install instructions (if it's public) or someone who
>> maintains it. It won't magically break without an upgrade, and it should not
>> get an upgrade without testing. The environment variable is available for
>> this kind of scenario, though I'd hope the testing occurs during beta and it
>> gets fixed by the time we release.
>
> I disagree that breaking backward compatibility is worth it. Most
> users don't care of Unicode since their application already "just
> works well" for their use case.

Again, the problem is libraries (code written by someone else that you 
want to reuse), not applications (code written by you to solve your 
business problem in your environment). Code that assumes the default 
encodings are sufficient is already broken in the general case, and 
libraries nearly always need to cover the general case while 
applications do not. The stdlib needs to cover the general case, which 
is why I keep using open(os.listdir(b'.')[-1]) as an example of 
something that should never fail because of encoding issues.

In theory, we should encourage library developers to support Windows 
properly by using str for paths, probably by disabling bytes paths 
everywhere. Alternatively, we make it so that bytes paths work fine 
everywhere and stop telling people that their code is wrong for a 
platform they're already not hugely concerned about.

> Having to set an env var to "repair" their app to be able to upgrade
> Python is not really convenient.

Upgrading Python in an already running system isn't going to be really 
convenient anyway. Going from x.y.z to x.y.z+1 should be convenient, but 
from x.y to x.y+1 deserves testing and possibly code or environment 
changes. I don't understand why changing Python at the same time we 
change the version number is suddenly controversial.

Cheers,
Steve


From guido at python.org  Tue Aug 30 14:08:48 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Aug 2016 11:08:48 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <57C5CABD.2090401@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
 <57C5CABD.2090401@python.org>
Message-ID: <CAP7+vJK4BGyii0f+U+p_CD0tRv4aCBZ-=Ovaz3cCXrpd15-UJw@mail.gmail.com>

Is this thread something I need to follow closely?

-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Tue Aug 30 14:05:46 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 04:05:46 +1000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
Message-ID: <CADiSq7d_oJ8_nv6fKTVxVX=RrF-iUouay3F6BWJrrLD_+vrKpw@mail.gmail.com>

On 31 August 2016 at 01:06, Victor Stinner <victor.stinner at gmail.com> wrote:
> 2016-08-30 16:31 GMT+02:00 Steve Dower <steve.dower at python.org>:
>> Any system that requires communication between two different versions of
>> Python must have install instructions (if it's public) or someone who
>> maintains it. It won't magically break without an upgrade, and it should not
>> get an upgrade without testing. The environment variable is available for
>> this kind of scenario, though I'd hope the testing occurs during beta and it
>> gets fixed by the time we release.
>
> I disagree that breaking backward compatibility is worth it. Most
> users don't care of Unicode since their application already "just
> works well" for their use case.
>
> Having to set an env var to "repair" their app to be able to upgrade
> Python is not really convenient.

This seems to be the crux of the disagreement: our perceptions of the
relative risks to native Windows Python applications that currently
work properly on Python 3.5 vs the potential compatibility benefits to
primarily *nix applications that currently *don't* work on Windows
under Python 3.5.

If I'm understanding Steve's position correctly, his view is that
native Python applications that are working well on Windows under
Python 3.5 *must already be using strings to interact with the OS*.
This means that they will be unaffected by the proposed changes, as
the proposed changes only impact attempts to pass bytes to the OS, not
attempts to pass strings.

In uncontrolled environments, using bytes to interact with the OS on
Windows just *plain doesn't work properly* under the current model, so
the proposed change is a matter of changing precisely how those
applications fail, rather than moving them from a previously working
state to a newly broken state.

For the proposed default behaviour change to cause problems then,
there must be large bodies of software that exist in sufficiently
controlled environments that they can get bytes-on-WIndows to work in
the first place by carefully managing the code page settings, but then
*also* permit uncontrolled upgrades to Python 3.6 without first
learning that they need to add a new environment variable setting to
preserve the Python 3.5 (and earlier) bytes handling behaviour.
Steve's assertion is that this intersection of "managed code page
settings" and "unmanaged Python upgrades" results in the null set.

A new opt-in config option eliminates any risk of breaking anything,
but means Steve will have to wait until 3.7 to try out the idea of
having more *nix centric software "just work" on Windows. In the grand
scheme of things, I still think it's worth taking that additional
time, especially if things are designed so that embedding applications
can easily flip the default behaviour.

Yes, there will be environments on Windows where the command line
option won't help, just as there are environments on Linux where it
won't help. I think that's OK, as we can use the 3.6 cycle to thrash
out the details of the new behaviour in the environments where it
*does* help (e.g. developers running their test suites on Windows
systems), get people used to the idea that the behaviour of binary
paths on Windows is going to change, and then actually make the switch
in Python 3.7.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From brett at python.org  Tue Aug 30 14:12:01 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Aug 2016 18:12:01 +0000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <20160830194813.0cbb261b@fsol>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
 <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
 <20160830194813.0cbb261b@fsol>
Message-ID: <CAP1=2W7sB_f-TvLajsCPMsfxR4rWf5r1T=nQVemgsVgDjDbx=g@mail.gmail.com>

On Tue, 30 Aug 2016 at 10:49 Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Tue, 30 Aug 2016 17:35:35 +0000
> Brett Cannon <brett at python.org> wrote:
> > >
> > > Perhaps a list would work indeed.  Realistically, if there are at most
> > > 2-3 users of the field at any given time (and most probably only one or
> > > zero), a simple type check (by pointer equality) on each list item may
> > > be sufficient.
> > >
> >
> > Let's see what Maciej says, but we could standardize on switching the
> field
> > to a list when a conflict of usage is detected so the common case in the
> > frame eval function is checking for your own type, and if that fails then
> > doing a PyList_CheckExact() and look for your object, otherwise make a
> list
> > and move over to that for everyone to use. A little bit more code, but
> it's
> > simple code and takes care of conflicts only when it calls for it.
>
> That's a bit obscure and confusing, though (I *think* the weakref module
> uses a similar kludge in some place).  If you want to iterate on it you
> have to write some bizarre macro to share the loop body between the two
> different code-paths (list / non-list), or some equally tedious
> function-pointer-based code.


I don't quite follow where the complexity you're suggesting comes from. The
frame evaluation function in Pyjion would just do:

  if (co_extra == NULL) {
      // No one using the field.
      co_extra = pyjion_cache = PyPyjion_New();
  }
  else if (!is_pyjion_object(co_extra)) {
    // Someone other than us is using the field.
    if (PyList_CheckExact(co_extra)) {
      // Field is already a list.
      ... look for object ...
      if (ob_found != NULL) {
        // We're in the list.
        pyjion_cache = ob_found;
      }
      else {
        // Not in the list, so add ourselves.
        pyjion_cache = PyPyjion_New();
        co_extra.append(pyjion_cache);
      }
    }
    else {
      // Someone else in the field, not a list (yet).
      other_ob = co_extra;
      co_extra = PyList_New();
      co_extra.append(other_ob);
      pyjion_cache = PyPyjion_New();
      co_extra.append(pyjion_cache);
    }
  }
  else {
    // We're in the field.
    pyjion_cache = co_extra;
  }


>
> Why not make it always a list?  List objects are reasonably cheap in
> memory and access time... (unlike dicts)
>

Because I would prefer to avoid any form of unnecessary performance
overhead for the common case.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/68ce54ef/attachment.html>

From steve.dower at python.org  Tue Aug 30 14:17:15 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 30 Aug 2016 11:17:15 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAP7+vJK4BGyii0f+U+p_CD0tRv4aCBZ-=Ovaz3cCXrpd15-UJw@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
 <57C5CABD.2090401@python.org>
 <CAP7+vJK4BGyii0f+U+p_CD0tRv4aCBZ-=Ovaz3cCXrpd15-UJw@mail.gmail.com>
Message-ID: <57C5CDAB.6050705@python.org>

On 30Aug2016 1108, Guido van Rossum wrote:
> Is this thread something I need to follow closely?

I have PEPs coming, and I'll distil the technical parts of the 
discussion into those.

We may need you to impose an opinion on whether 3.6 is an appropriate 
time for the change or it should wait for 3.7. I think the technical 
implications are fairly clear, it's just the risk of 
surprising/upsetting users that is not.

Cheers,
Steve

From solipsis at pitrou.net  Tue Aug 30 14:20:37 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Aug 2016 20:20:37 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAP1=2W7sB_f-TvLajsCPMsfxR4rWf5r1T=nQVemgsVgDjDbx=g@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
 <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
 <20160830194813.0cbb261b@fsol>
 <CAP1=2W7sB_f-TvLajsCPMsfxR4rWf5r1T=nQVemgsVgDjDbx=g@mail.gmail.com>
Message-ID: <20160830202037.28943cec@fsol>

On Tue, 30 Aug 2016 18:12:01 +0000
Brett Cannon <brett at python.org> wrote:
> > Why not make it always a list?  List objects are reasonably cheap in
> > memory and access time... (unlike dicts)
> 
> Because I would prefer to avoid any form of unnecessary performance
> overhead for the common case.

But the performance overhead of iterating over a 1-element list
is small enough (it's just an array access after a pointer dereference)
that it may not be larger than the overhead of the multiple tests and
conditional branches your example shows.

Regards

Antoine.

From storchaka at gmail.com  Tue Aug 30 14:55:16 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 30 Aug 2016 21:55:16 +0300
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <20160830202037.28943cec@fsol>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
 <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
 <20160830194813.0cbb261b@fsol>
 <CAP1=2W7sB_f-TvLajsCPMsfxR4rWf5r1T=nQVemgsVgDjDbx=g@mail.gmail.com>
 <20160830202037.28943cec@fsol>
Message-ID: <nq4kqk$gig$1@blaine.gmane.org>

On 30.08.16 21:20, Antoine Pitrou wrote:
> On Tue, 30 Aug 2016 18:12:01 +0000
> Brett Cannon <brett at python.org> wrote:
>>> Why not make it always a list?  List objects are reasonably cheap in
>>> memory and access time... (unlike dicts)
>>
>> Because I would prefer to avoid any form of unnecessary performance
>> overhead for the common case.
>
> But the performance overhead of iterating over a 1-element list
> is small enough (it's just an array access after a pointer dereference)
> that it may not be larger than the overhead of the multiple tests and
> conditional branches your example shows.

Iterating over a tuple is even faster. It needs one pointer dereference 
less.

And for memory efficiency we can use just a raw array of pointers.



From storchaka at gmail.com  Tue Aug 30 15:06:47 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 30 Aug 2016 22:06:47 +0300
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <CADiSq7ehQy-VJdx+t8KjjgUd2WYY3oM+6WWJ90DYJr7hdkVweQ@mail.gmail.com>
References: <npt414$mdm$1@blaine.gmane.org>
 <CADiSq7ehQy-VJdx+t8KjjgUd2WYY3oM+6WWJ90DYJr7hdkVweQ@mail.gmail.com>
Message-ID: <nq4lg6$pi3$1@blaine.gmane.org>

On 30.08.16 20:42, Nick Coghlan wrote:
> On 28 August 2016 at 08:25, Terry Reedy <tjreedy at udel.edu> wrote:
>> Slicing can be made to malfunction and even crash with an 'evil' __index__
>> method. https://bugs.python.org/issue27867
>>
>> The crux of the problem is this: PySlice_GetIndicesEx
>> receives a slice object and a sequence length.  Calling __index__ on the
>> start, stop, and step components can mutate the sequence and invalidate the
>> length.  Adjusting the int values of start and stop according to an invalid
>> length (in particular, one that is too long) will result in invalid results
>> or a crash.
>>
>> Possible actions -- very briefly.  For more see end of
>> https://bugs.python.org/issue27867?@ok_message=msg 273801
>> 0. Do nothing.
>> 1. Detect length change and raise.
>> 2. Retrieve length after any possible changes and proceed as normal.
>>
>> Possible implementation strategies for 1. and 2.
>> A. Change all functions that call PySlice_GetIndicesEx.
>> B. Add PySlice_GetIndicesEx2 (or ExEx?), which would receive *collection
>> instead of length, so the length could be retrieved after the __index__
>> calls.  Change calls. Deprecate PySlice_GetIndicesEx.
>
> Given Serhiy's clarification that this is primarily a thread safety
> problem, I'm more supportive of the "PySlice_GetIndicesForObject"
> approach (since that can call all the __index__ methods first, leaving
> the final __len__ call as the only problematic case).

This doesn't work with multidimensional slicing (like 
_testbuffer.ndarray or NumPy arrays).

> However, given the observation that __len__ can also release the GIL,
> I'm not clear on how 2A is supposed to work - a poorly timed thread
> switch means there's always going to be a risk of len(obj) returning
> outdated information if a container implemented in Python is being
> mutated concurrently from different threads, so what can be done
> differently in the calling functions that couldn't be done in a new
> API that accepted the container reference?

Current code doesn't use __len__. It uses something like 
PyUnicode_GET_LENGTH().

The solution was found easier than I afraid. See my patch to issue27867.



From mal at egenix.com  Tue Aug 30 16:07:55 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 30 Aug 2016 22:07:55 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
Message-ID: <57C5E79B.9010302@egenix.com>

On 29.08.2016 22:16, Christian Heimes wrote:
> On 2016-08-29 21:31, M.-A. Lemburg wrote:
>> On 29.08.2016 18:33, Cory Benfield wrote:
>>>
>>>> On 29 Aug 2016, at 04:09, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>
>>>> On 28.08.2016 22:40, Christian Heimes wrote:
>>>>> ...
>>>>> I like to reduce the maintenance burden and list of supported OpenSSL
>>>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1
>>>>> will reach EOL by the end of this year,
>>>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
>>>>> 0.9.8 is still required for some platforms (OSX).
>>>>> ...
>>>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
>>>>> 1.0.2 features for 3.7.
>>>>> ...
>>>>
>>>> Hmm, that last part would mean that Python 3.7 will no longer compile
>>>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
>>>> Since 14.04 LTS is supported until 2019, I think it would be better
>>>> to only start requiring 1.0.2 in Python 3.8.
>>>
>>> Can someone explain to me why this is a use-case we care about?
>>
>> Ubuntu 14.04 is a widely deployed system and newer Python version
>> should run on such widely deployed systems without having to
>> replace important vendor maintained system libraries such as
>> OpenSSL.
> 
> "Widely deployed" is true for a lot of old operating systems including
> Windows XP.
> 
>> Python 3.7 starts shipping around June 2018 (assuming the 18 month
>> release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
>> be able to use Python 3.7 on such a system, you'd have to upgrade
>> to a more recent LTS version 10 months before the EOL date (with
>> all the associated issues) or lose vendor maintenance support and
>> run with your own copy of OpenSSL.
> 
> Why would you deploy an unsupported Python version on a LTS release? Why
> should compatibility be our concern?
> 
>> Sure, but Ubuntu will continue to support OpenSSL 1.0.1
>> until 2019, backporting important security fixes as necessary and
>> that's what's important.
> 
> I see an easy solution here: either pay or make Canonical backport all
> required features to OpenSSL 1.0.1. </sarcasm>
> 
>> It's unfortunate that Python has to rely on a 3rd party library
>> for security, but we should at least make sure that our users
>> can rely on OS vendor support to keep the lib up to date with
>> security fixes.
> 
> No, it is a good thing that we can rely on 3rd party libraries for
> security. Crypto and security is not our domain. It is incredible hard
> to develop and maintain crypto code. Also my proposal enforces OS
> vendors to supply up to date OpenSSL versions.

That was not my point. It's unfortunate that Python depends on
a library which is inevitably going to need updates frequently,
and which then may have the implication that Python won't compile on
systems which don't ship with more recent OpenSSL libs - even
if your application doesn't even need ssl at all.

>> On 29.08.2016 10:24, Christian Heimes wrote:
>>> By the way I knew that something like this would come up from you.
>>> Thank you that you satisfied my expectation. :p
>>
>> Sure, I want Python to be used on as many systems as possible,
>> both in terms of architecture and OS. The more the better.
>> If we don't have to drop support early, why should we ?
> 
> MAL, I don't like your attitude. It feels like you want me and other
> contributors to waste time on this topic. That is not how this
> discussion is going to end. If *you* want to keep support for outdated
> OpenSSL versions, than it is *your* responsibility and *your* time. You
> cannot and will not put this burden on me.

Please reread what I suggested: to postpone the switch to require
OpenSSL 1.0.2 by one Python release version. And in my reply I then
put this into more context, saying that your schedule will likely
work out.

Postponing this should not introduce more work for anyone; if you'd
like to add support for 1.0.2 feature early this can also easily be
done by making such support optional depending on which OpenSSL
lib Python is compiled against. This takes a few #ifdefs, nothing
more.

> Python is running out of developers with OpenSSL expertise. It's Alex,
> Antoine, Benjamin, Victor and me. Antoine and me haven't been active for
> a while. Victor and Benjamin are mostly working on other topics. As far
> as I can judge Alex, he rather works on PyCA than CPython stdlib.
> 
> I'm both interested and willing to improve Python's ssl stack, and I'm
> going to do this in my own free time. Yes, I'm working for Red Hat's
> security engineering, but I'm not getting paid to work on Python (except
> for a couple of hours now and then when a bug is relevant for my daily
> work). I will only contribute improvements and fixes on my own terms,
> that means I'm not going to waste my time with outdated versions. In my
> opinion it is more than reasonable to ditch 1.0.1 and earlier.

I want you to consider the consequences of doing this carefully.

Crypto is important to have, but at the same time it's not
essentially for everything you do in Python, e.g. you can
easily run data analysis scripts or applications without ever
touching the ssl module.

Yet, a move to require OpenSSL 1.0.2 for Python 3.7 will make
it impossible to run such apps on systems that still use OpenSSL
1.0.1, e.g. Ubuntu 14.04 or CentOS 7.

>> This doesn't sound like a feature worth breaking compatibility
>> to me.
> 
> It does.

Why not make the 1.0.2 and 1.1.0 support optional as we do
in so many other situations for various systems and libs ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 30 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From brett at python.org  Tue Aug 30 17:06:53 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Aug 2016 21:06:53 +0000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <nq4kqk$gig$1@blaine.gmane.org>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
 <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
 <20160830194813.0cbb261b@fsol>
 <CAP1=2W7sB_f-TvLajsCPMsfxR4rWf5r1T=nQVemgsVgDjDbx=g@mail.gmail.com>
 <20160830202037.28943cec@fsol> <nq4kqk$gig$1@blaine.gmane.org>
Message-ID: <CAP1=2W5xcfpacx+Ce+Kh0jA7-X4V0BQ4XpEjAxK5vAuqV5exZw@mail.gmail.com>

On Tue, 30 Aug 2016 at 11:56 Serhiy Storchaka <storchaka at gmail.com> wrote:

> On 30.08.16 21:20, Antoine Pitrou wrote:
> > On Tue, 30 Aug 2016 18:12:01 +0000
> > Brett Cannon <brett at python.org> wrote:
> >>> Why not make it always a list?  List objects are reasonably cheap in
> >>> memory and access time... (unlike dicts)
> >>
> >> Because I would prefer to avoid any form of unnecessary performance
> >> overhead for the common case.
> >
> > But the performance overhead of iterating over a 1-element list
> > is small enough (it's just an array access after a pointer dereference)
> > that it may not be larger than the overhead of the multiple tests and
> > conditional branches your example shows.
>
> Iterating over a tuple is even faster. It needs one pointer dereference
> less.
>

I'll talk it over with Dino and see what he thinks.


>
> And for memory efficiency we can use just a raw array of pointers.
>

I would rather not do that as that leads to having to track the end of the
array, special memory cleanup, etc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/252c4a66/attachment.html>

From rosuav at gmail.com  Tue Aug 30 17:11:23 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 31 Aug 2016 07:11:23 +1000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <nq4kqk$gig$1@blaine.gmane.org>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
 <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
 <20160830194813.0cbb261b@fsol>
 <CAP1=2W7sB_f-TvLajsCPMsfxR4rWf5r1T=nQVemgsVgDjDbx=g@mail.gmail.com>
 <20160830202037.28943cec@fsol> <nq4kqk$gig$1@blaine.gmane.org>
Message-ID: <CAPTjJmoBYjJZ7=q05qn5mzfr01pxXLCu8c1wnx8=WvjRewjWeA@mail.gmail.com>

On Wed, Aug 31, 2016 at 4:55 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 30.08.16 21:20, Antoine Pitrou wrote:
>>
>> On Tue, 30 Aug 2016 18:12:01 +0000
>> Brett Cannon <brett at python.org> wrote:
>>>>
>>>> Why not make it always a list?  List objects are reasonably cheap in
>>>> memory and access time... (unlike dicts)
>>>
>>>
>>> Because I would prefer to avoid any form of unnecessary performance
>>> overhead for the common case.
>>
>>
>> But the performance overhead of iterating over a 1-element list
>> is small enough (it's just an array access after a pointer dereference)
>> that it may not be larger than the overhead of the multiple tests and
>> conditional branches your example shows.
>
>
> Iterating over a tuple is even faster. It needs one pointer dereference
> less.
>
> And for memory efficiency we can use just a raw array of pointers.

Didn't all this kind of thing come up when function annotations were
discussed? Insane schemes like dictionaries with UUID keys and so on.
The decision then was YAGNI. The decision now, IMO, should be the
same. Keep things simple.

ChrisA

From guido at python.org  Tue Aug 30 17:20:26 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Aug 2016 14:20:26 -0700
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
Message-ID: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>

I'm happy to present PEP 526 for your collective review:
https://www.python.org/dev/peps/pep-0526/ (HTML)
https://github.com/python/peps/blob/master/pep-0526.txt (source)

There's also an implementation ready:
https://github.com/ilevkivskyi/cpython/tree/pep-526

I don't want to post the full text here but I encourage feedback on
the high-order ideas, including but not limited to

- Whether (given PEP 484's relative success) it's worth adding syntax
for variable/attribute annotations.

- Whether the keyword-free syntax idea proposed here is best:
  NAME: TYPE
  TARGET: TYPE = VALUE

Note that there's an extensive list of rejected ideas in the PEP;
please be so kind to read it before posting here:
https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-left-out-for-now


-- 
--Guido van Rossum (python.org/~guido)

From victor.stinner at gmail.com  Tue Aug 30 17:51:36 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 Aug 2016 23:51:36 +0200
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CADiSq7d_oJ8_nv6fKTVxVX=RrF-iUouay3F6BWJrrLD_+vrKpw@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
 <CADiSq7d_oJ8_nv6fKTVxVX=RrF-iUouay3F6BWJrrLD_+vrKpw@mail.gmail.com>
Message-ID: <CAMpsgwZOoAiGZhSQzq9NdzZrE=AMJYVYnZ+f17yqNKb480Nr3g@mail.gmail.com>

Le 30 ao?t 2016 8:05 PM, "Nick Coghlan" <ncoghlan at gmail.com> a ?crit :
> This seems to be the crux of the disagreement: our perceptions of the
> relative risks to native Windows Python applications that currently
> work properly on Python 3.5 vs the potential compatibility benefits to
> primarily *nix applications that currently *don't* work on Windows
> under Python 3.5.

As I already wrote once, my problem is also tjat I simply have no idea how
much Python 3 code uses bytes filename. For example, does it concern more
than 25% of py3 modules on PyPi, or less than 5%?

Having an idea of the ratio would help to move the discussion forward.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/60331ee2/attachment.html>

From cory at lukasa.co.uk  Tue Aug 30 18:03:11 2016
From: cory at lukasa.co.uk (Cory Benfield)
Date: Tue, 30 Aug 2016 18:03:11 -0400
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C5E79B.9010302@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
Message-ID: <CF6C0469-3116-4EAD-9C54-E9FF46CA2E4A@lukasa.co.uk>


> On 30 Aug 2016, at 16:07, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> That was not my point. It's unfortunate that Python depends on
> a library which is inevitably going to need updates frequently,
> and which then may have the implication that Python won't compile on
> systems which don't ship with more recent OpenSSL libs - even
> if your application doesn't even need ssl at all.
> 
> Crypto is important to have, but at the same time it's not
> essentially for everything you do in Python, e.g. you can
> easily run data analysis scripts or applications without ever
> touching the ssl module.
> 
> Yet, a move to require OpenSSL 1.0.2 for Python 3.7 will make
> it impossible to run such apps on systems that still use OpenSSL
> 1.0.1, e.g. Ubuntu 14.04 or CentOS 7.

If your application doesn?t need SSL, then you can compile without OpenSSL. I just downloaded and compiled the current tip of the CPython repository on a system with no OpenSSL, and the world didn?t explode, it just printed this:

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel      
_dbm                  _gdbm                 _lzma              
_sqlite3              _ssl                  _tkinter           
readline              zlib                                     
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

So this user you have considered, who needs Python but not the ssl module, is still well served. The ssl module is not mandatory in CPython, and no-one is proposing that it should be.

But the real question is this: who *is* this hypothetical user? This user apparently needs the latest CPython, but is entirely unwilling to update literally anything else, including moving to a more recent release of their operating system. They are equipped to compile Python from source, but are apparently unwilling or unable to install a more recent OpenSSL from source. I?m not entirely certain that python-dev should be supporting that user: that user should be contacting their LTS supplier.

Cory

From srkunze at mail.de  Tue Aug 30 18:05:48 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 31 Aug 2016 00:05:48 +0200
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
Message-ID: <2eb58e17-45fd-1a05-844d-7d535b55d55f@mail.de>

Thanks Guido, also to the rest of the PEP team (4 people) :)


On 30.08.2016 23:20, Guido van Rossum wrote:
> I'm happy to present PEP 526 for your collective review:
> https://www.python.org/dev/peps/pep-0526/ (HTML)
> https://github.com/python/peps/blob/master/pep-0526.txt (source)
>
> There's also an implementation ready:
> https://github.com/ilevkivskyi/cpython/tree/pep-526
>
> I don't want to post the full text here but I encourage feedback on
> the high-order ideas, including but not limited to
>
> - Whether (given PEP 484's relative success) it's worth adding syntax
> for variable/attribute annotations.

I'd say no, especially because of the negative feedback by not a few 
thread participants.

> - Whether the keyword-free syntax idea proposed here is best:
>    NAME: TYPE
>    TARGET: TYPE = VALUE

If it will come, it's the best because of its similarity with parameter 
annotations and IIRC there are languages that already do it like this.

> Note that there's an extensive list of rejected ideas in the PEP;
> please be so kind to read it before posting here:
> https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-left-out-for-now

I find everything else well covered in the PEP especially corner-cases 
like variables without initialization, scopes etc.


Sven

From brett at python.org  Tue Aug 30 18:48:46 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Aug 2016 22:48:46 +0000
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
Message-ID: <CAP1=2W7aq62L0oOqb8Dr3xLXqZSaXaAmGDYMMRa+cmGoB3iuVA@mail.gmail.com>

On Tue, 30 Aug 2016 at 14:21 Guido van Rossum <guido at python.org> wrote:

> I'm happy to present PEP 526 for your collective review:
> https://www.python.org/dev/peps/pep-0526/ (HTML)
> https://github.com/python/peps/blob/master/pep-0526.txt (source)
>
> There's also an implementation ready:
> https://github.com/ilevkivskyi/cpython/tree/pep-526
>
> I don't want to post the full text here but I encourage feedback on
> the high-order ideas, including but not limited to
>
> - Whether (given PEP 484's relative success) it's worth adding syntax
> for variable/attribute annotations.
>

I think so, otherwise type hints are in this weird "half in, half out"
situation in terms of support that only non-OO code can fully utilize.
Either we're going to have type hints for those that want it and properly
support it for full use, or we shouldn't have type hints at all and this
syntax fills in a nice gaps that was a bit awkward to use before.


>
> - Whether the keyword-free syntax idea proposed here is best:
>   NAME: TYPE
>   TARGET: TYPE = VALUE
>

I personally like it. I've been learning Rust lately and it matches up with
their syntax (sans `let`) and I have been happy with it (same goes for
TypeScript's use of the same syntax that Rust uses).

-Brett


>
> Note that there's an extensive list of rejected ideas in the PEP;
> please be so kind to read it before posting here:
>
> https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-left-out-for-now
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/65caaf5b/attachment.html>

From victor.stinner at gmail.com  Tue Aug 30 19:11:37 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 31 Aug 2016 01:11:37 +0200
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAMpsgwZOoAiGZhSQzq9NdzZrE=AMJYVYnZ+f17yqNKb480Nr3g@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
 <CADiSq7d_oJ8_nv6fKTVxVX=RrF-iUouay3F6BWJrrLD_+vrKpw@mail.gmail.com>
 <CAMpsgwZOoAiGZhSQzq9NdzZrE=AMJYVYnZ+f17yqNKb480Nr3g@mail.gmail.com>
Message-ID: <CAMpsgwbq_aWNVVP982aDHkx7bYMVU+1bkNhOaQmSSdZDq_nALA@mail.gmail.com>

2016-08-30 23:51 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:
> As I already wrote once, my problem is also tjat I simply have no idea how
> much Python 3 code uses bytes filename. For example, does it concern more
> than 25% of py3 modules on PyPi, or less than 5%?

I made a very quick test on Windows using a modified Python raising an
exception on bytes path.

First of all, setuptools fails. It's a kind of blocker issue :-) I
quickly fixed it (only one line needs to be modified).

I tried to run Twisted unit tests (python -m twisted.trial twisted) of
Twisted 16.4. I got a lot of exceptions on bytes path from the
twisted/python/filepath.py module, but also from
twisted/trial/util.py. It looks like these modules are doing their
best to convert all paths to... bytes. I had to modify more than 5
methods just to be able to start running unit tests.

Quick result: setuptools and Twisted rely on bytes path. Dropping
bytes path support on Windows breaks these modules.

It also means that these modules don't support the full Unicode range
on Windows on Python 3.5.

Victor

From steve.dower at python.org  Tue Aug 30 19:27:23 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 30 Aug 2016 16:27:23 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAMpsgwbq_aWNVVP982aDHkx7bYMVU+1bkNhOaQmSSdZDq_nALA@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
 <CADiSq7d_oJ8_nv6fKTVxVX=RrF-iUouay3F6BWJrrLD_+vrKpw@mail.gmail.com>
 <CAMpsgwZOoAiGZhSQzq9NdzZrE=AMJYVYnZ+f17yqNKb480Nr3g@mail.gmail.com>
 <CAMpsgwbq_aWNVVP982aDHkx7bYMVU+1bkNhOaQmSSdZDq_nALA@mail.gmail.com>
Message-ID: <57C6165B.4090207@python.org>

On 30Aug2016 1611, Victor Stinner wrote:
> 2016-08-30 23:51 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:
>> As I already wrote once, my problem is also tjat I simply have no idea how
>> much Python 3 code uses bytes filename. For example, does it concern more
>> than 25% of py3 modules on PyPi, or less than 5%?
>
> I made a very quick test on Windows using a modified Python raising an
> exception on bytes path.
>
> First of all, setuptools fails. It's a kind of blocker issue :-) I
> quickly fixed it (only one line needs to be modified).
>
> I tried to run Twisted unit tests (python -m twisted.trial twisted) of
> Twisted 16.4. I got a lot of exceptions on bytes path from the
> twisted/python/filepath.py module, but also from
> twisted/trial/util.py. It looks like these modules are doing their
> best to convert all paths to... bytes. I had to modify more than 5
> methods just to be able to start running unit tests.
>
> Quick result: setuptools and Twisted rely on bytes path. Dropping
> bytes path support on Windows breaks these modules.
>
> It also means that these modules don't support the full Unicode range
> on Windows on Python 3.5.

Thanks. That's a good idea (certainly better than mine, which was to go 
reading code...)

I haven't looked into setuptools, but Twisted appears to be correctly 
using sys.getfilesystemencoding() when they coerce to bytes, which means 
the proposed change will simply allow the full Unicode range when paths 
are encoded.

However, if there are places where bytes are not transcoded when they 
should be *then* there will be new issues. I wonder if we can quickly 
test whether that happens (e.g. use the file system encoding to "taint" 
the path somehow - special prefix? - so we can raise if bytes that 
haven't been correctly encoded at some point are passed in).

Some of my other searching revealed occasional correct use of 
sys.getfilesystemencoding(), a decent number of uses as a fallback when 
other encodings are not available, and it's very hard to search for code 
that uses the os module with bytes not checked to be the right encoding. 
This is why I argue that the beta period is the best opportunity to 
check, and why we're better to flip the switch now and flip it back if 
it all goes horribly wrong - the alternative is a *very* labour 
intensive exercise that I doubt we can muster.



From greg at krypto.org  Tue Aug 30 19:55:59 2016
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 30 Aug 2016 23:55:59 +0000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C5E79B.9010302@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
Message-ID: <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>

On Tue, Aug 30, 2016 at 1:08 PM M.-A. Lemburg <mal at egenix.com> wrote:

> On 29.08.2016 22:16, Christian Heimes wrote:
> > On 2016-08-29 21:31, M.-A. Lemburg wrote:
> >> On 29.08.2016 18:33, Cory Benfield wrote:
> >>>
> >>>> On 29 Aug 2016, at 04:09, M.-A. Lemburg <mal at egenix.com> wrote:
> >>>>
> >>>> On 28.08.2016 22:40, Christian Heimes wrote:
> >>>>> ...
> >>>>> I like to reduce the maintenance burden and list of supported OpenSSL
> >>>>> versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year.
> 1.0.1
> >>>>> will reach EOL by the end of this year,
> >>>>> https://www.openssl.org/policies/releasestrat.html . However OpenSSL
> >>>>> 0.9.8 is still required for some platforms (OSX).
> >>>>> ...
> >>>>> For upcoming 3.6 I would like to limit support to 1.0.2+ and require
> >>>>> 1.0.2 features for 3.7.
> >>>>> ...
> >>>>
> >>>> Hmm, that last part would mean that Python 3.7 will no longer compile
> >>>> on e.g. Ubuntu 14.04 LTS which uses OpenSSL 1.0.1 as default version.
> >>>> Since 14.04 LTS is supported until 2019, I think it would be better
> >>>> to only start requiring 1.0.2 in Python 3.8.
> >>>
> >>> Can someone explain to me why this is a use-case we care about?
> >>
> >> Ubuntu 14.04 is a widely deployed system and newer Python version
> >> should run on such widely deployed systems without having to
> >> replace important vendor maintained system libraries such as
> >> OpenSSL.
> >
> > "Widely deployed" is true for a lot of old operating systems including
> > Windows XP.
> >
> >> Python 3.7 starts shipping around June 2018 (assuming the 18 month
> >> release cycle). Ubuntu 14.04 EOL is April 2019, so in order to
> >> be able to use Python 3.7 on such a system, you'd have to upgrade
> >> to a more recent LTS version 10 months before the EOL date (with
> >> all the associated issues) or lose vendor maintenance support and
> >> run with your own copy of OpenSSL.
> >
> > Why would you deploy an unsupported Python version on a LTS release? Why
> > should compatibility be our concern?
> >
> >> Sure, but Ubuntu will continue to support OpenSSL 1.0.1
> >> until 2019, backporting important security fixes as necessary and
> >> that's what's important.
> >
> > I see an easy solution here: either pay or make Canonical backport all
> > required features to OpenSSL 1.0.1. </sarcasm>
> >
> >> It's unfortunate that Python has to rely on a 3rd party library
> >> for security, but we should at least make sure that our users
> >> can rely on OS vendor support to keep the lib up to date with
> >> security fixes.
> >
> > No, it is a good thing that we can rely on 3rd party libraries for
> > security. Crypto and security is not our domain. It is incredible hard
> > to develop and maintain crypto code. Also my proposal enforces OS
> > vendors to supply up to date OpenSSL versions.
>
> That was not my point. It's unfortunate that Python depends on
> a library which is inevitably going to need updates frequently,
> and which then may have the implication that Python won't compile on
> systems which don't ship with more recent OpenSSL libs - even
> if your application doesn't even need ssl at all.
>
> >> On 29.08.2016 10:24, Christian Heimes wrote:
> >>> By the way I knew that something like this would come up from you.
> >>> Thank you that you satisfied my expectation. :p
> >>
> >> Sure, I want Python to be used on as many systems as possible,
> >> both in terms of architecture and OS. The more the better.
> >> If we don't have to drop support early, why should we ?
> >
> > MAL, I don't like your attitude. It feels like you want me and other
> > contributors to waste time on this topic. That is not how this
> > discussion is going to end. If *you* want to keep support for outdated
> > OpenSSL versions, than it is *your* responsibility and *your* time. You
> > cannot and will not put this burden on me.
>
> Please reread what I suggested: to postpone the switch to require
> OpenSSL 1.0.2 by one Python release version. And in my reply I then
> put this into more context, saying that your schedule will likely
> work out.
>
> Postponing this should not introduce more work for anyone; if you'd
> like to add support for 1.0.2 feature early this can also easily be
> done by making such support optional depending on which OpenSSL
> lib Python is compiled against. This takes a few #ifdefs, nothing
> more.
>
> > Python is running out of developers with OpenSSL expertise. It's Alex,
> > Antoine, Benjamin, Victor and me. Antoine and me haven't been active for
> > a while. Victor and Benjamin are mostly working on other topics. As far
> > as I can judge Alex, he rather works on PyCA than CPython stdlib.
> >
> > I'm both interested and willing to improve Python's ssl stack, and I'm
> > going to do this in my own free time. Yes, I'm working for Red Hat's
> > security engineering, but I'm not getting paid to work on Python (except
> > for a couple of hours now and then when a bug is relevant for my daily
> > work). I will only contribute improvements and fixes on my own terms,
> > that means I'm not going to waste my time with outdated versions. In my
> > opinion it is more than reasonable to ditch 1.0.1 and earlier.
>
> I want you to consider the consequences of doing this carefully.
>

We have. The proposal now stands at: We're free to ditch 1.0.1 and earlier
support in 3.7 as soon as we have a need for a 1.0.2 specific API.

There don't seem to be any important negative consequences of doing so.

Crypto is important to have, but at the same time it's not
> essentially for everything you do in Python, e.g. you can
> easily run data analysis scripts or applications without ever
> touching the ssl module.
>

While technically true, the ssl module is required to fetch and install
software via pip which for casual users means it is essential. People can
always operate without it if they are willing to download and build the
libraries they need manually.

Yet, a move to require OpenSSL 1.0.2 for Python 3.7 will make
> it impossible to run such apps on systems that still use OpenSSL
> 1.0.1, e.g. Ubuntu 14.04 or CentOS 7.
>

Not important. That isn't something we need to worry about. Compiling a new
libssl is easy.  People using systems that are 4+ years old by the time 3.7
comes out who expect new software to compile and just work are expecting
too much.

I find that users of such systems either use only what their distro itself
supplies (ie: ancient versions at that point) or are fully comfortable
building any dependencies their own software needs. If they are comfortable
building a CPython runtime in the first place, they should be comfortable
building required libraries. Nothing new there.


> Why not make the 1.0.2 and 1.1.0 support optional as we do
> in so many other situations for various systems and libs ?
>

In general, conditional compilation complicates code and adds a maintenance
burden. The sooner we can ditch older versions, the cleaner and easier to
maintain our code is.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/b6541825/attachment.html>

From victor.stinner at gmail.com  Tue Aug 30 20:02:25 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 31 Aug 2016 02:02:25 +0200
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAMpsgwbq_aWNVVP982aDHkx7bYMVU+1bkNhOaQmSSdZDq_nALA@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
 <CADiSq7d_oJ8_nv6fKTVxVX=RrF-iUouay3F6BWJrrLD_+vrKpw@mail.gmail.com>
 <CAMpsgwZOoAiGZhSQzq9NdzZrE=AMJYVYnZ+f17yqNKb480Nr3g@mail.gmail.com>
 <CAMpsgwbq_aWNVVP982aDHkx7bYMVU+1bkNhOaQmSSdZDq_nALA@mail.gmail.com>
Message-ID: <CAMpsgwbCPnee=Rezft1StvTrRpetEmBvYPjBVsxjDdF2Ek+DNg@mail.gmail.com>

I made another quick&dirty test on Django 1.10 (I ran Django test
suite on my modified Python raising exception on bytes path): I didn't
notice any exception related to bytes path.

Django seems to only use Unicode for paths.

I can try to run more tests if you know some other major Python
applications (modules?) working on Windows/Python 3.

Note: About Twisted, I forgot to mention that I'm not really surprised
that Twisted uses bytes. Twisted was created something like 10 years
ago, when bytes was the defacto choice. Using Unicode in Python 2 was
painful when you imagine a module as large as Twisted. Twisted has to
support Python 2 and Python 3, so it's not surprising that it still
uses bytes in some places, instead of Unicode. Moreover, as many
Python applications/modules, Linux is a first citizen, whereas Windows
is more supported as "best effort".

Victor

From steve.dower at python.org  Tue Aug 30 20:27:12 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 30 Aug 2016 17:27:12 -0700
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <CAMpsgwbCPnee=Rezft1StvTrRpetEmBvYPjBVsxjDdF2Ek+DNg@mail.gmail.com>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
 <CADiSq7d_oJ8_nv6fKTVxVX=RrF-iUouay3F6BWJrrLD_+vrKpw@mail.gmail.com>
 <CAMpsgwZOoAiGZhSQzq9NdzZrE=AMJYVYnZ+f17yqNKb480Nr3g@mail.gmail.com>
 <CAMpsgwbq_aWNVVP982aDHkx7bYMVU+1bkNhOaQmSSdZDq_nALA@mail.gmail.com>
 <CAMpsgwbCPnee=Rezft1StvTrRpetEmBvYPjBVsxjDdF2Ek+DNg@mail.gmail.com>
Message-ID: <57C62460.5060003@python.org>

On 30Aug2016 1702, Victor Stinner wrote:
> I made another quick&dirty test on Django 1.10 (I ran Django test
> suite on my modified Python raising exception on bytes path): I didn't
> notice any exception related to bytes path.
>
> Django seems to only use Unicode for paths.
>
> I can try to run more tests if you know some other major Python
> applications (modules?) working on Windows/Python 3.

The major ones aren't really the concern. I'd be interested to see where 
numpy and pandas are at, but I suspect they've already encountered and 
fixed many of these issues due to the size of the user base. (Though 
skim-reading numpy I see lots of code that would be affected - for 
better or worse - if the default encoding for open() changed...)

I'm more concerned about the long-tail of more focused libraries. Feel 
free to grab a random selection of Django extensions and try them out, 
but I don't really think it's worth the effort. I'm certainly not 
demanding you do it.

> Note: About Twisted, I forgot to mention that I'm not really surprised
> that Twisted uses bytes. Twisted was created something like 10 years
> ago, when bytes was the defacto choice. Using Unicode in Python 2 was
> painful when you imagine a module as large as Twisted. Twisted has to
> support Python 2 and Python 3, so it's not surprising that it still
> uses bytes in some places, instead of Unicode.

Yeah, I don't think they're doing anything wrong and wouldn't want to 
call them out on it. Especially since they already correctly handle it 
by asking Python what encoding should be used for the bytes.

> Moreover, as many
> Python applications/modules, Linux is a first citizen, whereas Windows
> is more supported as "best effort".

That last point is exactly why I think this is important. Any arguments 
against making Windows behave more like Linux (i.e. bytes paths are 
reliable) need to be clear as to why this doesn't matter or is less 
important than other concerns.

Cheers,
Steve


From steve at pearwood.info  Tue Aug 30 21:00:41 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 31 Aug 2016 11:00:41 +1000
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
Message-ID: <20160831010040.GM26300@ando.pearwood.info>

On Tue, Aug 30, 2016 at 02:20:26PM -0700, Guido van Rossum wrote:
> I'm happy to present PEP 526 for your collective review:

Are you hoping to get this in before 3.6 beta? Because I'm not sure I 
can give this much attention before then, but I really want to.


-- 
Steve

From vadmium+py at gmail.com  Tue Aug 30 21:15:18 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Wed, 31 Aug 2016 01:15:18 +0000
Subject: [Python-Dev] Lib/http/client.py: could it return an OSError
 with the current response?
In-Reply-To: <CAPc4eF8da8eYrzWcZ7wxDn1NDAFW8VgPtZDm07+iYwvgfyULCw@mail.gmail.com>
References: <CAPc4eF8da8eYrzWcZ7wxDn1NDAFW8VgPtZDm07+iYwvgfyULCw@mail.gmail.com>
Message-ID: <CA+eR4cFqrNBsBqet294i4TDALZ2+-LkV9xGyLtRH1RLAyO+HXg@mail.gmail.com>

On 30 August 2016 at 13:41, Ivo Bellin Salarin
<ivo.bellinsalarin at gmail.com> wrote:
> While using requests to tunnel a request via a proxy requiring user
> authentication, I have seen that httplib
> (https://hg.python.org/cpython/file/3.5/Lib/http/client.py#l831) raises the
> message returned by the proxy, along with its status code (407) without
> including the proxy response. This one could be very interesting to the
> consumer, since it could contain some useful headers (like the supported
> authentication schemes).

Here are some existing bug threads which may be relevant:

https://bugs.python.org/issue7291 (urllib.request support for handling
tunnel authentication)
https://bugs.python.org/issue24964 (get tunnel response header fields
in http.client)

> Would it be possible to change the http/client.py behavior in order to raise
> an exception including the whole response?

That would be one way, and might be good enough for getting a
Proxy-Authenticate value. Although there might be other ways that
also:

* Allow reading the body (e.g. HTML page) of the error response. IMO
an exception instance is the wrong place for this;
urllib.error.HTTPError is a bad example.
* Allow the tunnel response fields to be obtained even when the
request was successful

> If you don't see any problem with my proposal, how can I propose a pull
> request? :-)

Perhaps you can use one of the patches at one of the above bug reports
as a starting point. What you want seems to be a prerequisite for
Issue 7291 (urllib.request), so maybe we can discuss it there. Or open
a new bug to focus on the http.client-only aspect.

From guido at python.org  Tue Aug 30 22:15:55 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Aug 2016 19:15:55 -0700
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <20160831010040.GM26300@ando.pearwood.info>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
 <20160831010040.GM26300@ando.pearwood.info>
Message-ID: <CAP7+vJLuV_34PtCN5AXueEOoRQFHLLnyDVgFZPfUcbDnZueJNA@mail.gmail.com>

On Tue, Aug 30, 2016 at 6:00 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Tue, Aug 30, 2016 at 02:20:26PM -0700, Guido van Rossum wrote:
>> I'm happy to present PEP 526 for your collective review:
>
> Are you hoping to get this in before 3.6 beta? Because I'm not sure I
> can give this much attention before then, but I really want to.

Yes I am hoping for that. Unlike PEP 484, this PEP is forward-looking
(more like PEP 492, async/await), and the sooner we can get it in the
sooner people who want to use it won't have to worry about supporting
older Python versions. (And am I ever looking forward to the day when
Python 3.5 counts as "older". :-)

While some of the details are better, this is substantially the same
proposal that we discussed at length in python-ideas, starting at
https://mail.python.org/pipermail/python-ideas/2016-August/041294.html
(and you participated vigorously in that thread, so very little in the
PEP should be news to you).

-- 
--Guido van Rossum (python.org/~guido)

From jackdied at gmail.com  Tue Aug 30 22:44:22 2016
From: jackdied at gmail.com (Jack Diederich)
Date: Tue, 30 Aug 2016 22:44:22 -0400
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <20160831010040.GM26300@ando.pearwood.info>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
 <20160831010040.GM26300@ando.pearwood.info>
Message-ID: <CACLn2+2LHn-FmMKGnqxgP-Rd92sBOjwuACqzEsdhkXkhqyeXAA@mail.gmail.com>

+0. We should try and be consistent even if this is a thing I don't
want. And trust me, I don't!

That said, as long as pro-mypy people are willing to make everyone else pay
a mypy reading tax for code let's try and reduce the cognitive burden.

* Duplicate type annotations should be a syntax error.
  Duplicate annotations aren't possible in functions so that wasn't an
issue in 484. 526 makes some things syntax errors and some things runtime
errors (for good reason -- function bodies aren't evaluated right away).
Double-annotating a variable is something we can figure out at compile time
and doing the double annotating is non-sensical so we should error on it
because we can.

*  Dissallowing annotations on global and nonlocal
  Agreed, allowing it would be confusing because it would either be a
re-definition or a hard to read annotation-at-a-distance.

* Where __annotations__ live
  It is strange to allow modules.__annotations__ and
MyClass.__annotations__ but not myfunc.__annotations__ (or more in line
with existing function implementations a myfunc.__code__.co_annotations).
If we know enough from the syntax parse to have func.__code__.co_varnames
be known then we should try to do that with annotations.  Let's raise a
SyntaxError for function body annotations that conflict with same-named
variables that are annotated in the function signature as well.

I did C++ for years before I did Python and wrote C++ in many languages
(including Python). So ideally I'm -1000 on all this stuff for cultural
reasons -- if you let a C++ person add types they will for false comfort.
But again, I'm +0 on this specific proposal because we have already gone
down the garden path.

-Jack


On Tue, Aug 30, 2016 at 9:00 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Tue, Aug 30, 2016 at 02:20:26PM -0700, Guido van Rossum wrote:
> > I'm happy to present PEP 526 for your collective review:
>
> Are you hoping to get this in before 3.6 beta? Because I'm not sure I
> can give this much attention before then, but I really want to.
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> jackdied%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/6df8d85f/attachment-0001.html>

From guido at python.org  Tue Aug 30 23:03:00 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Aug 2016 20:03:00 -0700
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CACLn2+2LHn-FmMKGnqxgP-Rd92sBOjwuACqzEsdhkXkhqyeXAA@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
 <20160831010040.GM26300@ando.pearwood.info>
 <CACLn2+2LHn-FmMKGnqxgP-Rd92sBOjwuACqzEsdhkXkhqyeXAA@mail.gmail.com>
Message-ID: <CAP7+vJJ1o-Kjsj-6gwk0r22s6u=j=5qKz8tAjOqmiLR7Mh+GPQ@mail.gmail.com>

On Tue, Aug 30, 2016 at 7:44 PM, Jack Diederich <jackdied at gmail.com> wrote:
> +0. We should try and be consistent even if this is a thing I don't want.
> And trust me, I don't!

No problem. You won't have to!

> That said, as long as pro-mypy people are willing to make everyone else pay
> a mypy reading tax for code let's try and reduce the cognitive burden.
>
> * Duplicate type annotations should be a syntax error.
>   Duplicate annotations aren't possible in functions so that wasn't an issue
> in 484. 526 makes some things syntax errors and some things runtime errors
> (for good reason -- function bodies aren't evaluated right away).
> Double-annotating a variable is something we can figure out at compile time
> and doing the double annotating is non-sensical so we should error on it
> because we can.

Actually I'm not so sure that double-annotating is always nonsensical.
In the mypy tracker we're seeing some requests for type *inference*
that allows a variable to be given another type later, e.g.

    x = 'abc'
    test_func(x)
    x = 42
    another_test_func(x)

Maybe there's a use for explicit annotations too. I would rather not
get in the way of letting type checkers decide such semantics.

> *  Dissallowing annotations on global and nonlocal
>   Agreed, allowing it would be confusing because it would either be a
> re-definition or a hard to read annotation-at-a-distance.
>
> * Where __annotations__ live
>   It is strange to allow modules.__annotations__ and MyClass.__annotations__
> but not myfunc.__annotations__ (or more in line with existing function
> implementations a myfunc.__code__.co_annotations). If we know enough from
> the syntax parse to have func.__code__.co_varnames be known then we should
> try to do that with annotations.  Let's raise a SyntaxError for function
> body annotations that conflict with same-named variables that are annotated
> in the function signature as well.

But myfunc.__annotations__ already exists -- PEP 3107 puts the
signature annotations there. The problem with co_annotations is that
annotations are evaluated (they can be quite complex expressions, e.g.
Optional[Tuple[int, int, some_mod.SomeClass]]), while co_varnames is
just a list of strings. And code objects must be immutable. The issue
with rejecting duplicate annotations so sternly is the same as for the
previous bullet.

> I did C++ for years before I did Python and wrote C++ in many languages
> (including Python). So ideally I'm -1000 on all this stuff for cultural
> reasons -- if you let a C++ person add types they will for false comfort.
> But again, I'm +0 on this specific proposal because we have already gone
> down the garden path.

As long as you run mypy the comfort shouldn't be false. (But your
starting with C++ before Python explains a lot. :-)

> -Jack

-- 
--Guido van Rossum (python.org/~guido)

From jackdied at gmail.com  Tue Aug 30 23:37:41 2016
From: jackdied at gmail.com (Jack Diederich)
Date: Tue, 30 Aug 2016 23:37:41 -0400
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CAP7+vJJ1o-Kjsj-6gwk0r22s6u=j=5qKz8tAjOqmiLR7Mh+GPQ@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
 <20160831010040.GM26300@ando.pearwood.info>
 <CACLn2+2LHn-FmMKGnqxgP-Rd92sBOjwuACqzEsdhkXkhqyeXAA@mail.gmail.com>
 <CAP7+vJJ1o-Kjsj-6gwk0r22s6u=j=5qKz8tAjOqmiLR7Mh+GPQ@mail.gmail.com>
Message-ID: <CACLn2+0BXNue_R8jB5dvm1jTU_Amn85aoADkhBUS7jDKMs14hA@mail.gmail.com>

On Tue, Aug 30, 2016 at 11:03 PM, Guido van Rossum <guido at python.org> wrote:

> On Tue, Aug 30, 2016 at 7:44 PM, Jack Diederich <jackdied at gmail.com>
> wrote:
> > +0. We should try and be consistent even if this is a thing I don't want.
> > And trust me, I don't!
>
> No problem. You won't have to!
>
>
Yes! I don't have to want it, it is here!


> > That said, as long as pro-mypy people are willing to make everyone else
> pay
> > a mypy reading tax for code let's try and reduce the cognitive burden.
> >
> > * Duplicate type annotations should be a syntax error.
> >   Duplicate annotations aren't possible in functions so that wasn't an
> issue
> > in 484. 526 makes some things syntax errors and some things runtime
> errors
> > (for good reason -- function bodies aren't evaluated right away).
> > Double-annotating a variable is something we can figure out at compile
> time
> > and doing the double annotating is non-sensical so we should error on it
> > because we can.
>
> Actually I'm not so sure that double-annotating is always nonsensical.
> In the mypy tracker we're seeing some requests for type *inference*
> that allows a variable to be given another type later, e.g.
>
>     x = 'abc'
>     test_func(x)
>     x = 42
>     another_test_func(x)
>
> Maybe there's a use for explicit annotations too. I would rather not
> get in the way of letting type checkers decide such semantics.
>
>
Other languages (including rpython) don't allow rebinding types (or
sometimes even re-assignment to same type). We are going for clarity [and
bondage, and discipline]. If we are doing types let's do types like other
people do. I think *disallowing* redefining the type is general to
enforcing types. +1 on being consistent with other langs. If plain
redoubling of types is allowed I'm OK "i: int = 0" doesn't summon horrors
when said three times into a mirror. But we can't always know what "int"
evaluates to so I'd just disallow it.


> > *  Dissallowing annotations on global and nonlocal
> >   Agreed, allowing it would be confusing because it would either be a
> > re-definition or a hard to read annotation-at-a-distance.
> >
> > * Where __annotations__ live
> >   It is strange to allow modules.__annotations__ and
> MyClass.__annotations__
> > but not myfunc.__annotations__ (or more in line with existing function
> > implementations a myfunc.__code__.co_annotations). If we know enough
> from
> > the syntax parse to have func.__code__.co_varnames be known then we
> should
> > try to do that with annotations.  Let's raise a SyntaxError for function
> > body annotations that conflict with same-named variables that are
> annotated
> > in the function signature as well.
>
> But myfunc.__annotations__ already exists -- PEP 3107 puts the
> signature annotations there. The problem with co_annotations is that
> annotations are evaluated (they can be quite complex expressions, e.g.
> Optional[Tuple[int, int, some_mod.SomeClass]]), while co_varnames is
> just a list of strings. And code objects must be immutable. The issue
> with rejecting duplicate annotations so sternly is the same as for the
> previous bullet.
>
>
If we disallow re-assignment of types as a syntax error then the conflict
with myfunc.__annotations__ goes away for vars that share a name with the
function arguments. The fact that variables with types can't be known until
the function body executes a particular line is .. I'm not sure how to deal
with that. For modules and classes you can assert that the body at the top
indent level has been executed. For functions you can only assert that it
has been parsed. So myfunc.__annotations__ could say that the type has a
definition but only later know what the definition is.

> I did C++ for years before I did Python and wrote C++ in many languages
> > (including Python). So ideally I'm -1000 on all this stuff for cultural
> > reasons -- if you let a C++ person add types they will for false comfort.
> > But again, I'm +0 on this specific proposal because we have already gone
> > down the garden path.
>
> As long as you run mypy the comfort shouldn't be false. (But your
> starting with C++ before Python explains a lot. :-)
>

We've talked about this and we have different relationships with tools. I'm
a monk who thinks using a debugger is an admission of failure; you think
linters are a fine method of dissuading others of sin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/c9e7abfb/attachment.html>

From ncoghlan at gmail.com  Wed Aug 31 00:10:05 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 14:10:05 +1000
Subject: [Python-Dev] What do we do about bad slicing and possible
 crashes (issue 27867)
In-Reply-To: <nq4lg6$pi3$1@blaine.gmane.org>
References: <npt414$mdm$1@blaine.gmane.org>
 <CADiSq7ehQy-VJdx+t8KjjgUd2WYY3oM+6WWJ90DYJr7hdkVweQ@mail.gmail.com>
 <nq4lg6$pi3$1@blaine.gmane.org>
Message-ID: <CADiSq7eNqj__EWxuhbGE7mJY50TjFqpR7X3Yya7YNgQ8-a3A5A@mail.gmail.com>

On 31 August 2016 at 05:06, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 30.08.16 20:42, Nick Coghlan wrote:
>> Given Serhiy's clarification that this is primarily a thread safety
>> problem, I'm more supportive of the "PySlice_GetIndicesForObject"
>> approach (since that can call all the __index__ methods first, leaving
>> the final __len__ call as the only problematic case).
>
> This doesn't work with multidimensional slicing (like _testbuffer.ndarray or
> NumPy arrays).

Thanks, that makes sense.

>> However, given the observation that __len__ can also release the GIL,
>> I'm not clear on how 2A is supposed to work - a poorly timed thread
>> switch means there's always going to be a risk of len(obj) returning
>> outdated information if a container implemented in Python is being
>> mutated concurrently from different threads, so what can be done
>> differently in the calling functions that couldn't be done in a new
>> API that accepted the container reference?
>
> Current code doesn't use __len__. It uses something like
> PyUnicode_GET_LENGTH().

Oh, I see - it's the usual rule that C code can be made implicitly
atomic if it avoids calling hooks potentially written in Python, but
pure Python containers need explicit locks to allow concurrent access
from multiple threads.

> The solution was found easier than I afraid. See my patch to issue27867.

+1 from me. Would it make sense to make these public and new additions
to the stable ABI for 3.6+?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug 31 00:14:49 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 14:14:49 +1000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CAPTjJmoBYjJZ7=q05qn5mzfr01pxXLCu8c1wnx8=WvjRewjWeA@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
 <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
 <20160830194813.0cbb261b@fsol>
 <CAP1=2W7sB_f-TvLajsCPMsfxR4rWf5r1T=nQVemgsVgDjDbx=g@mail.gmail.com>
 <20160830202037.28943cec@fsol> <nq4kqk$gig$1@blaine.gmane.org>
 <CAPTjJmoBYjJZ7=q05qn5mzfr01pxXLCu8c1wnx8=WvjRewjWeA@mail.gmail.com>
Message-ID: <CADiSq7fBaxMRXuP8SyebQkHSNEHcbYny3YOHrJJoJpLwW=QWtg@mail.gmail.com>

On 31 August 2016 at 07:11, Chris Angelico <rosuav at gmail.com> wrote:
> Didn't all this kind of thing come up when function annotations were
> discussed? Insane schemes like dictionaries with UUID keys and so on.
> The decision then was YAGNI. The decision now, IMO, should be the
> same. Keep things simple.

Different use case - for annotations, the *reader* of the code is one
of the intended audiences, so as the author of the code, you decide
what you want to tell them, and that then constrains the tools you can
use (or vice-versa - you pick the kinds of tools you want to use, and
that constrains what you can tell your readers).

This case is different - there are no human readers involved, only
automated tools, so adding a mandatory redirection through a sequence
is just a small performance hit rather than a readability problem.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug 31 00:30:00 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 14:30:00 +1000
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <nq4kqk$gig$1@blaine.gmane.org>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
 <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
 <20160830194813.0cbb261b@fsol>
 <CAP1=2W7sB_f-TvLajsCPMsfxR4rWf5r1T=nQVemgsVgDjDbx=g@mail.gmail.com>
 <20160830202037.28943cec@fsol> <nq4kqk$gig$1@blaine.gmane.org>
Message-ID: <CADiSq7dcMeJ=rbGvwdA=rH43y6xb0kqVgeBvH4i64k2b-vHuPg@mail.gmail.com>

On 31 August 2016 at 04:55, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 30.08.16 21:20, Antoine Pitrou wrote:
>> But the performance overhead of iterating over a 1-element list
>> is small enough (it's just an array access after a pointer dereference)
>> that it may not be larger than the overhead of the multiple tests and
>> conditional branches your example shows.
>
> Iterating over a tuple is even faster. It needs one pointer dereference
> less.

That comes at the cost of making metadata additions a bit more
complicated though - you'd have to replace the existing tuple with a
new one that adds your own metadata, rather than just appending to a
list.

I do think there are enough subtleties here (going from no metadata ->
some metadata, and some metadata -> more metadata) that it makes sense
to provide a standard API for it (excluded from the stable ABI),
rather than expecting plugin developers to roll their own.

Strawman:

    PyObject * PyCode_GetExtra(PyCodeObject *code, PyTypeObject *extra_type);
    int PyCode_SetExtra(PyCodeObject *code, PyObject *extra);
    int PyCode_DelExtra(PyCodeObject *code, PyTypeObject *extra_type);

Then Brett's example code would become:

    pyjion_cache = PyCode_GetExtra(code_obj, &PyPyjion_Type);
    if (pyjion_cache == NULL) {
        pyjion_cache = PyPyjion_New();
        if (PyCode_SetExtra(code_obj, pyjion_cache) < 0) {
            /* Something went wrong, report that somehow */        }
    }
    /* pyjion_cache is valid here */

Making those APIs fast (for an assumed small number of simultaneously
active interpreter plugins) and thread-safe is then an internal
CPython implementation detail, rather than being something plugin
writers need to concern themselves with.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug 31 00:53:05 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 14:53:05 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
Message-ID: <CADiSq7dcmoS=2ZXho-aWh7qM0of28EaJhPJU+VF_FOh5ePi4xg@mail.gmail.com>

On 31 August 2016 at 09:55, Gregory P. Smith <greg at krypto.org> wrote:
> On Tue, Aug 30, 2016 at 1:08 PM M.-A. Lemburg <mal at egenix.com> wrote:
>> Yet, a move to require OpenSSL 1.0.2 for Python 3.7 will make
>> it impossible to run such apps on systems that still use OpenSSL
>> 1.0.1, e.g. Ubuntu 14.04 or CentOS 7.
>
> Not important. That isn't something we need to worry about. Compiling a new
> libssl is easy.  People using systems that are 4+ years old by the time 3.7
> comes out who expect new software to compile and just work are expecting too
> much.
>
> I find that users of such systems either use only what their distro itself
> supplies (ie: ancient versions at that point) or are fully comfortable
> building any dependencies their own software needs. If they are comfortable
> building a CPython runtime in the first place, they should be comfortable
> building required libraries. Nothing new there.

There's a 3rd variant, which is to raise support tickets with their
LTS vendors to request compatibility backports. I strongly encourage
that behaviour by end user organisations when wearing both my upstream
volunteer contributor hat, since it means they're not bothering
community volunteers with their institutional support requests, and my
downstream redistributor employee hat, since the more Python related
customer support requests Red Hat receives, the easier it gets for
folks internally (including me) to put together business cases arguing
for increased direct investment in the upstream Python ecosystem :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug 31 00:58:09 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 14:58:09 +1000
Subject: [Python-Dev] File system path encoding on Windows
In-Reply-To: <57C62460.5060003@python.org>
References: <f6dfca05-244b-b695-de3d-5a45c549fb14@python.org>
 <CAMpsgwaT7POtbVvK12uf34F=G0AgS05cRhdGYwbZ_9u8YwZxPQ@mail.gmail.com>
 <CADiSq7czQKcRXK9Fi+=KnJFPJ+0TH87BhdHziQQbs0a337047w@mail.gmail.com>
 <CAMpsgwbFMoi6hzZkgi5Ofa9w1bH9oNJ41GCb0_5vF7PzLgGWDg@mail.gmail.com>
 <E1bek4u-0005pP-JW@se2-syd.hostedmail.net.au>
 <CAMpsgwYyGxZNA6obk-x=aLSuHGx3r-xhraYWA5t5t_sGYpFRog@mail.gmail.com>
 <CADiSq7d_oJ8_nv6fKTVxVX=RrF-iUouay3F6BWJrrLD_+vrKpw@mail.gmail.com>
 <CAMpsgwZOoAiGZhSQzq9NdzZrE=AMJYVYnZ+f17yqNKb480Nr3g@mail.gmail.com>
 <CAMpsgwbq_aWNVVP982aDHkx7bYMVU+1bkNhOaQmSSdZDq_nALA@mail.gmail.com>
 <CAMpsgwbCPnee=Rezft1StvTrRpetEmBvYPjBVsxjDdF2Ek+DNg@mail.gmail.com>
 <57C62460.5060003@python.org>
Message-ID: <CADiSq7cN0fS2p5YuC=nNtuHWAaA77b6w8u0i27ciNmq_MC7yAQ@mail.gmail.com>

On 31 August 2016 at 10:27, Steve Dower <steve.dower at python.org> wrote:
> On 30Aug2016 1702, Victor Stinner wrote:
>> I can try to run more tests if you know some other major Python
>> applications (modules?) working on Windows/Python 3.
>
> The major ones aren't really the concern. I'd be interested to see where
> numpy and pandas are at, but I suspect they've already encountered and fixed
> many of these issues due to the size of the user base. (Though skim-reading
> numpy I see lots of code that would be affected - for better or worse - if
> the default encoding for open() changed...)

For a case of "Don't break software already trying to do things
right", the https://github.com/beetbox/beets example that Daniel
linked earlier would be a good one to test.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug 31 01:11:23 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 15:11:23 +1000
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CACLn2+0BXNue_R8jB5dvm1jTU_Amn85aoADkhBUS7jDKMs14hA@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
 <20160831010040.GM26300@ando.pearwood.info>
 <CACLn2+2LHn-FmMKGnqxgP-Rd92sBOjwuACqzEsdhkXkhqyeXAA@mail.gmail.com>
 <CAP7+vJJ1o-Kjsj-6gwk0r22s6u=j=5qKz8tAjOqmiLR7Mh+GPQ@mail.gmail.com>
 <CACLn2+0BXNue_R8jB5dvm1jTU_Amn85aoADkhBUS7jDKMs14hA@mail.gmail.com>
Message-ID: <CADiSq7dCncrJLHWLC259-oxemjgkRsZ2R9C2q5geYnzO-Mk=Vg@mail.gmail.com>

On 31 August 2016 at 13:37, Jack Diederich <jackdied at gmail.com> wrote:
> On Tue, Aug 30, 2016 at 11:03 PM, Guido van Rossum <guido at python.org> wrote:
>> But myfunc.__annotations__ already exists -- PEP 3107 puts the
>> signature annotations there. The problem with co_annotations is that
>> annotations are evaluated (they can be quite complex expressions, e.g.
>> Optional[Tuple[int, int, some_mod.SomeClass]]), while co_varnames is
>> just a list of strings. And code objects must be immutable. The issue
>> with rejecting duplicate annotations so sternly is the same as for the
>> previous bullet.
>>
>
> If we disallow re-assignment of types as a syntax error then the conflict
> with myfunc.__annotations__ goes away for vars that share a name with the
> function arguments. The fact that variables with types can't be known until
> the function body executes a particular line is .. I'm not sure how to deal
> with that. For modules and classes you can assert that the body at the top
> indent level has been executed. For functions you can only assert that it
> has been parsed. So myfunc.__annotations__ could say that the type has a
> definition but only later know what the definition is.

What if we included local variable annotations in func.__annotations__
as cells, like the entries in func.__closure__?

We could also use that as a micro-optimisation technique: once the
type annotation cell is populated, CPython would just use it, rather
than re-evaluating the local variable type annotation expression every
time the function is called.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Wed Aug 31 01:40:57 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Aug 2016 22:40:57 -0700
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CADiSq7dCncrJLHWLC259-oxemjgkRsZ2R9C2q5geYnzO-Mk=Vg@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
 <20160831010040.GM26300@ando.pearwood.info>
 <CACLn2+2LHn-FmMKGnqxgP-Rd92sBOjwuACqzEsdhkXkhqyeXAA@mail.gmail.com>
 <CAP7+vJJ1o-Kjsj-6gwk0r22s6u=j=5qKz8tAjOqmiLR7Mh+GPQ@mail.gmail.com>
 <CACLn2+0BXNue_R8jB5dvm1jTU_Amn85aoADkhBUS7jDKMs14hA@mail.gmail.com>
 <CADiSq7dCncrJLHWLC259-oxemjgkRsZ2R9C2q5geYnzO-Mk=Vg@mail.gmail.com>
Message-ID: <CAP7+vJJMvT9CQQfNsEmAV1YeqX99O=0EeeZeORHYOjfS9400Sg@mail.gmail.com>

On Tuesday, August 30, 2016, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 31 August 2016 at 13:37, Jack Diederich <jackdied at gmail.com
> <javascript:;>> wrote:
> > On Tue, Aug 30, 2016 at 11:03 PM, Guido van Rossum <guido at python.org
> <javascript:;>> wrote:
> >> But myfunc.__annotations__ already exists -- PEP 3107 puts the
> >> signature annotations there. The problem with co_annotations is that
> >> annotations are evaluated (they can be quite complex expressions, e.g.
> >> Optional[Tuple[int, int, some_mod.SomeClass]]), while co_varnames is
> >> just a list of strings. And code objects must be immutable. The issue
> >> with rejecting duplicate annotations so sternly is the same as for the
> >> previous bullet.
> >>
> >
> > If we disallow re-assignment of types as a syntax error then the conflict
> > with myfunc.__annotations__ goes away for vars that share a name with the
> > function arguments. The fact that variables with types can't be known
> until
> > the function body executes a particular line is .. I'm not sure how to
> deal
> > with that. For modules and classes you can assert that the body at the
> top
> > indent level has been executed. For functions you can only assert that it
> > has been parsed. So myfunc.__annotations__ could say that the type has a
> > definition but only later know what the definition is.
>
> What if we included local variable annotations in func.__annotations__
> as cells, like the entries in func.__closure__?
>
> We could also use that as a micro-optimisation technique: once the
> type annotation cell is populated, CPython would just use it, rather
> than re-evaluating the local variable type annotation expression every
> time the function is called.
>

But what runtime use have the annotations on locals? They are not part of
any inspectable interface. I don't want to spend any effort on them at
runtime. (Just the bit that they are treated as locals.)

--Guido


-- 
--Guido (mobile)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160830/f957883b/attachment.html>

From victor.stinner at gmail.com  Wed Aug 31 02:18:12 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 31 Aug 2016 08:18:12 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <20160830180729.63540a92@fsol>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
Message-ID: <CAMpsgwb4f=c4QqDSD3zSQ6sG4rz4SndSvRcZpB+cyPX3TN_Hww@mail.gmail.com>

The PEP 445, C API for malloc, allows to plug multiple wrappers and each
wrapper has its own "void* context" data. When you register a new wrapper,
you store the current context and function to later chain it.

See the hooks example:
https://www.python.org/dev/peps/pep-0445/#use-case-3-setup-hooks-on-memory-block-allocators

Since the PEP 523 also adds a function, would it be possible to somehow
design a mecanism to "chain wrappers"?

I know that the PEP 523 has a different design, so maybe it's not possible.

For example, the context can be passed to PyFrameEvalFunction. In this
case, each project would have to register its own eval function, including
vmprof. I don't know if it makes sense for vmprof to modify the behaviour
at runtime (add a C frame per Python eval frame).

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160831/b8edab90/attachment.html>

From p.f.moore at gmail.com  Wed Aug 31 02:40:47 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 Aug 2016 07:40:47 +0100
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
Message-ID: <CACac1F-wfq=R8-Nf3HF_Mpt-FmNi9YHPNx=i7Q0cF8jerkvdYw@mail.gmail.com>

On 31 August 2016 at 00:55, Gregory P. Smith <greg at krypto.org> wrote:
> I find that users of such systems either use only what their distro itself
> supplies (ie: ancient versions at that point) or are fully comfortable
> building any dependencies their own software needs. If they are comfortable
> building a CPython runtime in the first place, they should be comfortable
> building required libraries. Nothing new there

In our environment (corporate systems locked to older OS releases,
with Python *not* a strategic solution but used for ad-hoc automation)
it's quite common to find only an ancient version of Python available,
but want to build a new version without any ability to influence
corporate IT to allow new versions of the necessary libraries.

But I strongly agree, this is *my* problem, and Python policy should
not be based on the idea that what I want to do "should" be supported.

So +1 on the proposed change here.

Paul

From stefan_ml at behnel.de  Wed Aug 31 02:59:00 2016
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 31 Aug 2016 08:59:00 +0200
Subject: [Python-Dev] Update on PEP 523 and adding a co_extra field to
 code objects
In-Reply-To: <CADiSq7dcMeJ=rbGvwdA=rH43y6xb0kqVgeBvH4i64k2b-vHuPg@mail.gmail.com>
References: <CAP1=2W6ZA58V+UD4cMSNrX43oW=MXrO12D4e6tFMUaTpS2dabA@mail.gmail.com>
 <20160830180729.63540a92@fsol>
 <CAP1=2W5fz7KH4s5dwhk4i7o7i3jQ375qL13WXj=wkzVHYr=yLg@mail.gmail.com>
 <20160830193051.532e8d9a@fsol>
 <CAP1=2W5CrSk3rw_S220aj9B5Eo8fbYuUj0OUfQ0oWcz_d+mKZA@mail.gmail.com>
 <20160830194813.0cbb261b@fsol>
 <CAP1=2W7sB_f-TvLajsCPMsfxR4rWf5r1T=nQVemgsVgDjDbx=g@mail.gmail.com>
 <20160830202037.28943cec@fsol> <nq4kqk$gig$1@blaine.gmane.org>
 <CADiSq7dcMeJ=rbGvwdA=rH43y6xb0kqVgeBvH4i64k2b-vHuPg@mail.gmail.com>
Message-ID: <nq5v7j$aae$1@blaine.gmane.org>

Nick Coghlan schrieb am 31.08.2016 um 06:30:
> On 31 August 2016 at 04:55, Serhiy Storchaka wrote:
>> On 30.08.16 21:20, Antoine Pitrou wrote:
>>> But the performance overhead of iterating over a 1-element list
>>> is small enough (it's just an array access after a pointer dereference)
>>> that it may not be larger than the overhead of the multiple tests and
>>> conditional branches your example shows.
>>
>> Iterating over a tuple is even faster. It needs one pointer dereference
>> less.
> 
> That comes at the cost of making metadata additions a bit more
> complicated though - you'd have to replace the existing tuple with a
> new one that adds your own metadata, rather than just appending to a
> list.
> 
> I do think there are enough subtleties here (going from no metadata ->
> some metadata, and some metadata -> more metadata) that it makes sense
> to provide a standard API for it (excluded from the stable ABI),
> rather than expecting plugin developers to roll their own.
> 
> Strawman:
> 
>     PyObject * PyCode_GetExtra(PyCodeObject *code, PyTypeObject *extra_type);
>     int PyCode_SetExtra(PyCodeObject *code, PyObject *extra);
>     int PyCode_DelExtra(PyCodeObject *code, PyTypeObject *extra_type);
> 
> Then Brett's example code would become:
> 
>     pyjion_cache = PyCode_GetExtra(code_obj, &PyPyjion_Type);
>     if (pyjion_cache == NULL) {
>         pyjion_cache = PyPyjion_New();
>         if (PyCode_SetExtra(code_obj, pyjion_cache) < 0) {
>             /* Something went wrong, report that somehow */        }
>     }
>     /* pyjion_cache is valid here */
> 
> Making those APIs fast (for an assumed small number of simultaneously
> active interpreter plugins) and thread-safe is then an internal
> CPython implementation detail, rather than being something plugin
> writers need to concern themselves with.

Looks like a good idea. New non-trivial field, new API.

GetExtra() can be a macro that implements the "only one entry and type
pointer matches" case for speed, then call back into the list lookup for
the less common cases.

Stefan



From mal at egenix.com  Wed Aug 31 04:31:12 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 31 Aug 2016 10:31:12 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
Message-ID: <57C695D0.4020406@egenix.com>

On 31.08.2016 01:55, Gregory P. Smith wrote:
> On Tue, Aug 30, 2016 at 1:08 PM M.-A. Lemburg <mal at egenix.com> wrote:
>>> On 29.08.2016 22:16, Christian Heimes wrote:
>>> In my
>>> opinion it is more than reasonable to ditch 1.0.1 and earlier.
>>
>> I want you to consider the consequences of doing this carefully.
>>
> 
> We have. The proposal now stands at: We're free to ditch 1.0.1 and earlier
> support in 3.7 as soon as we have a need for a 1.0.2 specific API.
> 
> There don't seem to be any important negative consequences of doing so.

Uhm, what about not being able to run Python unless you are willing
to stop taking benefit of OS vendor supplied OpenSSL security
fixes ?

In all this discussion I have yet to find a compelling security
relevant argument for using an 1.0.2 API which is so important
that we cannot make this optional at runtime.

The only argument Christian reported was this one:

"""
> BTW: Are there any features in 1.0.2 that we need and would warrant
> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?

Yes, there are features I want to use, e.g. proper hostname
verification. Python's post-handshake verification is a hack and leads
to information disclosure.
"""

Regarding that argument: hostname validation can be done
in 1.0.1 by providing a verification hook handler. That's
intended and by design, not a hack. 1.0.2 comes with
support for hostname validation making this a little easier
(you still have to set this up, though).

So essentially, the only argument so far has been that
we can remove some support code and let 1.0.2 take care of
this.

That's a maintenance argument, not a security one. And the
code in question has been rather stable, so it doesn't add
much maintenance overhead:

https://hg.python.org/cpython/annotate/default/Lib/ssl.py#l195

For dropping Python support on older platforms, I'd expect to
at least see an argument such as: 1.0.1 has a security flaw
which cannot be addressed by design (e.g. a missing crypto
feature which is essential for working with modern SSL
servers).

>> Crypto is important to have, but at the same time it's not
>> essentially for everything you do in Python, e.g. you can
>> easily run data analysis scripts or applications without ever
>> touching the ssl module.
> 
> While technically true, the ssl module is required to fetch and install
> software via pip which for casual users means it is essential. People can
> always operate without it if they are willing to download and build the
> libraries they need manually.
>
>> Yet, a move to require OpenSSL 1.0.2 for Python 3.7 will make
>> it impossible to run such apps on systems that still use OpenSSL
>> 1.0.1, e.g. Ubuntu 14.04 or CentOS 7.
>>
> 
> Not important. That isn't something we need to worry about. Compiling a new
> libssl is easy.  People using systems that are 4+ years old by the time 3.7
> comes out who expect new software to compile and just work are expecting
> too much.

Perhaps, but industry OSes are not upgraded that often, so it's
not unusual to run into a system where you'd like to get Python
working on an LTS OS version which is still maintained but doesn't
use the latest and greatest software versions.

> I find that users of such systems either use only what their distro itself
> supplies (ie: ancient versions at that point) or are fully comfortable
> building any dependencies their own software needs. If they are comfortable
> building a CPython runtime in the first place, they should be comfortable
> building required libraries. Nothing new there.

Why do you assume that people will compile their own CPython
on such systems ? It's well possible, and probably more likely,
that they want to run a binary version of the application on
such a platform, only to find that it doesn't run because of
a missing OpenSSL 1.0.2 API (the libssl.so.1.0 will still be
available, but not expose the requested API - the lib version
of OpenSSL did not change between 1.0.1 and 1.0.2).

>> Why not make the 1.0.2 and 1.1.0 support optional as we do
>> in so many other situations for various systems and libs ?
>>
> 
> In general, conditional compilation complicates code and adds a maintenance
> burden. The sooner we can ditch older versions, the cleaner and easier to
> maintain our code is.

I think we need to find the right balance here. A few lines of
conditional code vs. Python not running on a system at all.

And just to be clear, since some people appear to think that
I want to make them work for me for free (something I find a bit
bizarre given how much time I have invested in Python development -
you are benefiting from the code I've written just as much as
I am from the code you have written):

We are using our own egenix-pyopenssl package which comes with
OpenSSL 1.0.2 and we do take on the challenge to support this
on multiple platforms, including older ones. So the argument
is somewhat pointless.

I am thinking of Python users out there who are running on LTS
OS releases simply because their IT doesn't let them run anything
else. If we want to cut them off, we should have compelling arguments
to do so. IMO, the arguments brought up so far are not good
enough.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From christian at python.org  Wed Aug 31 04:37:02 2016
From: christian at python.org (Christian Heimes)
Date: Wed, 31 Aug 2016 10:37:02 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C5E79B.9010302@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
Message-ID: <9e9c07ef-b799-3536-4747-1330752de51f@python.org>

On 2016-08-30 22:07, M.-A. Lemburg wrote:
> That was not my point. It's unfortunate that Python depends on
> a library which is inevitably going to need updates frequently,
> and which then may have the implication that Python won't compile on
> systems which don't ship with more recent OpenSSL libs - even
> if your application doesn't even need ssl at all.

That's wrong. ssl support is optional. hashlib builds without OpenSSL, too.

> Please reread what I suggested: to postpone the switch to require
> OpenSSL 1.0.2 by one Python release version. And in my reply I then
> put this into more context, saying that your schedule will likely
> work out.

OpenSSL 1.0.2 requirement is already postponed to 3.7.

> Postponing this should not introduce more work for anyone; if you'd
> like to add support for 1.0.2 feature early this can also easily be
> done by making such support optional depending on which OpenSSL
> lib Python is compiled against. This takes a few #ifdefs, nothing
> more.

No, the SSL module will require features that are OpenSSL 1.0.2 only.

>>> This doesn't sound like a feature worth breaking compatibility
>>> to me.
>>
>> It does.
> 
> Why not make the 1.0.2 and 1.1.0 support optional as we do
> in so many other situations for various systems and libs ?

Please read my mails. I gave you two reasons. First it's going to make
my work harder and I'm not willing to invest extra work to supported
deprecated, unsupported and insecure versions. Second I'm going to
require features that are 1.0.2 only.

Christian


From christian at python.org  Wed Aug 31 04:37:02 2016
From: christian at python.org (Christian Heimes)
Date: Wed, 31 Aug 2016 10:37:02 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C5E79B.9010302@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
Message-ID: <9e9c07ef-b799-3536-4747-1330752de51f@python.org>

On 2016-08-30 22:07, M.-A. Lemburg wrote:
> That was not my point. It's unfortunate that Python depends on
> a library which is inevitably going to need updates frequently,
> and which then may have the implication that Python won't compile on
> systems which don't ship with more recent OpenSSL libs - even
> if your application doesn't even need ssl at all.

That's wrong. ssl support is optional. hashlib builds without OpenSSL, too.

> Please reread what I suggested: to postpone the switch to require
> OpenSSL 1.0.2 by one Python release version. And in my reply I then
> put this into more context, saying that your schedule will likely
> work out.

OpenSSL 1.0.2 requirement is already postponed to 3.7.

> Postponing this should not introduce more work for anyone; if you'd
> like to add support for 1.0.2 feature early this can also easily be
> done by making such support optional depending on which OpenSSL
> lib Python is compiled against. This takes a few #ifdefs, nothing
> more.

No, the SSL module will require features that are OpenSSL 1.0.2 only.

>>> This doesn't sound like a feature worth breaking compatibility
>>> to me.
>>
>> It does.
> 
> Why not make the 1.0.2 and 1.1.0 support optional as we do
> in so many other situations for various systems and libs ?

Please read my mails. I gave you two reasons. First it's going to make
my work harder and I'm not willing to invest extra work to supported
deprecated, unsupported and insecure versions. Second I'm going to
require features that are 1.0.2 only.

Christian



From solipsis at pitrou.net  Wed Aug 31 04:43:01 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 31 Aug 2016 10:43:01 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com>
Message-ID: <20160831104301.4e61c428@fsol>

On Wed, 31 Aug 2016 10:31:12 +0200
"M.-A. Lemburg" <mal at egenix.com> wrote:
> 
> I am thinking of Python users out there who are running on LTS
> OS releases simply because their IT doesn't let them run anything
> else.

There is a solution nowadays, which is to use Anaconda (or Miniconda).

Regards

Antoine.



From christian at python.org  Wed Aug 31 04:50:19 2016
From: christian at python.org (Christian Heimes)
Date: Wed, 31 Aug 2016 10:50:19 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C695D0.4020406@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com>
Message-ID: <7f12976b-aae0-4a42-e591-86b87cdc47dd@python.org>

On 2016-08-31 10:31, M.-A. Lemburg wrote:
> In all this discussion I have yet to find a compelling security
> relevant argument for using an 1.0.2 API which is so important
> that we cannot make this optional at runtime.
> 
> The only argument Christian reported was this one:
> 
> """
>> BTW: Are there any features in 1.0.2 that we need and would warrant
>> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?
> 
> Yes, there are features I want to use, e.g. proper hostname
> verification. Python's post-handshake verification is a hack and leads
> to information disclosure.
> """
> 
> Regarding that argument: hostname validation can be done
> in 1.0.1 by providing a verification hook handler. That's
> intended and by design, not a hack. 1.0.2 comes with
> support for hostname validation making this a little easier
> (you still have to set this up, though).

Are you willing to do implement and maintain this callback? Are you
willing to do all work?

Are you aware how many security bugs we had in our own verification
code? I'm aware of at least two critical bugs.

Christian

From mal at egenix.com  Wed Aug 31 05:33:49 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 31 Aug 2016 11:33:49 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <7f12976b-aae0-4a42-e591-86b87cdc47dd@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com>
 <7f12976b-aae0-4a42-e591-86b87cdc47dd@python.org>
Message-ID: <57C6A47D.4050107@egenix.com>

On 31.08.2016 10:50, Christian Heimes wrote:
> On 2016-08-31 10:31, M.-A. Lemburg wrote:
>> In all this discussion I have yet to find a compelling security
>> relevant argument for using an 1.0.2 API which is so important
>> that we cannot make this optional at runtime.
>>
>> The only argument Christian reported was this one:
>>
>> """
>>> BTW: Are there any features in 1.0.2 that we need and would warrant
>>> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?
>>
>> Yes, there are features I want to use, e.g. proper hostname
>> verification. Python's post-handshake verification is a hack and leads
>> to information disclosure.
>> """
>>
>> Regarding that argument: hostname validation can be done
>> in 1.0.1 by providing a verification hook handler. That's
>> intended and by design, not a hack. 1.0.2 comes with
>> support for hostname validation making this a little easier
>> (you still have to set this up, though).
> 
> Are you willing to do implement and maintain this callback? Are you
> willing to do all work?

Maintain: yes, if needed.

It is already implemented, so that part isn't hard :-)

> Are you aware how many security bugs we had in our own verification
> code? I'm aware of at least two critical bugs.

Not that many, given that the host name validation is more
a best practices art rather than one where all participants
implement the standards:

http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=match_hostname&submit=search&status=-1%2C1%2C2%2C3

The only critical bug I could find was this one (NUL bytes
in subjectAltName):

http://bugs.python.org/issue18709

but as I understand, the true origin of the bug was an OpenSSL
function, not the host name matching code in Python.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From mal at egenix.com  Wed Aug 31 05:33:59 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 31 Aug 2016 11:33:59 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <20160831104301.4e61c428@fsol>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com> <20160831104301.4e61c428@fsol>
Message-ID: <57C6A487.2090301@egenix.com>

On 31.08.2016 10:43, Antoine Pitrou wrote:
> On Wed, 31 Aug 2016 10:31:12 +0200
> "M.-A. Lemburg" <mal at egenix.com> wrote:
>>
>> I am thinking of Python users out there who are running on LTS
>> OS releases simply because their IT doesn't let them run anything
>> else.
> 
> There is a solution nowadays, which is to use Anaconda (or Miniconda).

Sure, or use ActivePython or eGenix PyRun :-)

But is that really what we want to tell people ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From antoine at python.org  Wed Aug 31 05:43:52 2016
From: antoine at python.org (Antoine Pitrou)
Date: Wed, 31 Aug 2016 11:43:52 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C6A487.2090301@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com> <20160831104301.4e61c428@fsol>
 <57C6A487.2090301@egenix.com>
Message-ID: <9d0145ab-7fa8-ae6d-b709-53a9815667d9@python.org>


Le 31/08/2016 ? 11:33, M.-A. Lemburg a ?crit :
> On 31.08.2016 10:43, Antoine Pitrou wrote:
>> On Wed, 31 Aug 2016 10:31:12 +0200
>> "M.-A. Lemburg" <mal at egenix.com> wrote:
>>>
>>> I am thinking of Python users out there who are running on LTS
>>> OS releases simply because their IT doesn't let them run anything
>>> else.
>>
>> There is a solution nowadays, which is to use Anaconda (or Miniconda).
> 
> Sure, or use ActivePython or eGenix PyRun :-)

Uh, right, I was being employer-biased here, sorry.

> But is that really what we want to tell people ?

Why not?  python.org does not provide official binaries for Linux or
Unix systems (except OS X), and most people don't like compile their
infrastructure themselves.

People who want an up-to-date Python can either use:
- use the python.org binaries on OS X and Windows
- use a recent OS providing a recent Python version (for Linux and Unix
variants)
- use a vendor-supported backport on old OSes (if so provided, for
example on RedHat with Software Collections?)
- use a third party-supported backport on old OSes (ActivePython, eGenix
PyRun, etc.)
- as a last resort, hand-compile their Python, in which case they have
to be careful to gather the required dependencies

Regards

Antoine.

From christian at python.org  Wed Aug 31 06:05:05 2016
From: christian at python.org (Christian Heimes)
Date: Wed, 31 Aug 2016 12:05:05 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C6A47D.4050107@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com>
 <7f12976b-aae0-4a42-e591-86b87cdc47dd@python.org>
 <57C6A47D.4050107@egenix.com>
Message-ID: <91e10464-9c0c-3cb4-499c-a5ee23907f79@python.org>

On 2016-08-31 11:33, M.-A. Lemburg wrote:
> On 31.08.2016 10:50, Christian Heimes wrote:
>> On 2016-08-31 10:31, M.-A. Lemburg wrote:
>>> In all this discussion I have yet to find a compelling security
>>> relevant argument for using an 1.0.2 API which is so important
>>> that we cannot make this optional at runtime.
>>>
>>> The only argument Christian reported was this one:
>>>
>>> """
>>>> BTW: Are there any features in 1.0.2 that we need and would warrant
>>>> dropping support for 1.0.1 earlier than Ubuntu 14.04 LTS ?
>>>
>>> Yes, there are features I want to use, e.g. proper hostname
>>> verification. Python's post-handshake verification is a hack and leads
>>> to information disclosure.
>>> """
>>>
>>> Regarding that argument: hostname validation can be done
>>> in 1.0.1 by providing a verification hook handler. That's
>>> intended and by design, not a hack. 1.0.2 comes with
>>> support for hostname validation making this a little easier
>>> (you still have to set this up, though).
>>
>> Are you willing to do implement and maintain this callback? Are you
>> willing to do all work?
> 
> Maintain: yes, if needed.
> 
> It is already implemented, so that part isn't hard :-)

No, it is not implemented as callback. It is implemented as post
verification step, which is wrong.

>> Are you aware how many security bugs we had in our own verification
>> code? I'm aware of at least two critical bugs.
> 
> Not that many, given that the host name validation is more
> a best practices art rather than one where all participants
> implement the standards:
> 
> http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=match_hostname&submit=search&status=-1%2C1%2C2%2C3
> 
> The only critical bug I could find was this one (NUL bytes
> in subjectAltName):
> 
> http://bugs.python.org/issue18709
> 
> but as I understand, the true origin of the bug was an OpenSSL
> function, not the host name matching code in Python.

Ah, I forgot about the NULL bytes issue. The bug is not caused by a
problem in OpenSSL. We just the wrong function to convert General Name
ASN.1 strings to unicode.

Then there are four critical bugs:

* NULL bytes in SAN
* wrong, insecure RFC for wildcard matching
* DoS caused excessive regular expression matching for wildcards
* invalid handling of IDNA for wildcard matching

IP address verification is still wrong, too.


This was my last reply to your mails on this topic. It's clear to me
that you are not open to Cory's, Nick's or my arguments and that you
won't change your position. More replies are just a waste of my limited
time.

Instead I'm going to focus on a PEP to define OpenSSL support and to
auto-deprecate unsupported OpenSSL versions. The PEP is a very high
chance to get accepted. Everybody except you support the plan.

Christian

From ncoghlan at gmail.com  Wed Aug 31 06:09:21 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 20:09:21 +1000
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CAP7+vJJMvT9CQQfNsEmAV1YeqX99O=0EeeZeORHYOjfS9400Sg@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
 <20160831010040.GM26300@ando.pearwood.info>
 <CACLn2+2LHn-FmMKGnqxgP-Rd92sBOjwuACqzEsdhkXkhqyeXAA@mail.gmail.com>
 <CAP7+vJJ1o-Kjsj-6gwk0r22s6u=j=5qKz8tAjOqmiLR7Mh+GPQ@mail.gmail.com>
 <CACLn2+0BXNue_R8jB5dvm1jTU_Amn85aoADkhBUS7jDKMs14hA@mail.gmail.com>
 <CADiSq7dCncrJLHWLC259-oxemjgkRsZ2R9C2q5geYnzO-Mk=Vg@mail.gmail.com>
 <CAP7+vJJMvT9CQQfNsEmAV1YeqX99O=0EeeZeORHYOjfS9400Sg@mail.gmail.com>
Message-ID: <CADiSq7dbBzQ0ZmxrLCvyb3WTz3eByKYQ4kb+yXi=D2=FF_L8oA@mail.gmail.com>

On 31 August 2016 at 15:40, Guido van Rossum <guido at python.org> wrote:
> On Tuesday, August 30, 2016, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> What if we included local variable annotations in func.__annotations__
>> as cells, like the entries in func.__closure__?
>>
>> We could also use that as a micro-optimisation technique: once the
>> type annotation cell is populated, CPython would just use it, rather
>> than re-evaluating the local variable type annotation expression every
>> time the function is called.
>
> But what runtime use have the annotations on locals? They are not part of
> any inspectable interface. I don't want to spend any effort on them at
> runtime. (Just the bit that they are treated as locals.)

I guess as long as they're included somewhere in the AST for the
function body, I don't mind if the translation to bytecode throws them
away - that's essentially saying that a function level type annotation
is effectively interpreted as if it was:

    if False:
        __annotations__[<varname>] = <annotation>

So the code generator will pick up syntax errors during normal
execution, but not runtime errors (since the expression never actually
gets evaluated).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From christian at python.org  Wed Aug 31 06:17:34 2016
From: christian at python.org (Christian Heimes)
Date: Wed, 31 Aug 2016 12:17:34 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <20160830180035.190c3383@fsol>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <20160830180035.190c3383@fsol>
Message-ID: <f90bd08e-37d9-7e81-b313-7a1f6c48ecfa@python.org>

On 2016-08-30 18:00, Antoine Pitrou wrote:
> On Sun, 28 Aug 2016 22:40:11 +0200
> Christian Heimes <christian at python.org> wrote:
>>
>> Here is the deal for 2.7 to 3.5:
>>
>> 1) All versions older than 0.9.8 are completely out-of-scope and no
>> longer supported.
>>
>> 2) 0.9.8 is semi-support. Python will still compile and work with 0.9.8.
>> However we do NOT promise that is secure to run 0.9.8. We also require a
>> recent version. Patch level 0.9.8zc from October 2014 is reasonable
>> because it comes with SCSV fallback (CVE-2014-3566).
>>
>> 3) 1.0.0 is irrelevant. Users are either stuck on 0.9.8 or are able to
>> upgrade to 1.0.1+. Let's not support it.
>>
>> 4) 1.0.1 is discouraged but still supported until its EOL.
>>
>> 5) 1.0.2 is the recommend version.
>>
>> 6) 1.1 support will be added by #26470 soon.
>>
>> 7) LibreSSL 2.3 is supported but with a slightly limited feature set.
> 
> Can you expand briefly how "limited" the feature set is?  Does it only
> disable some arcane features, so that e.g. asyncio + TLS supports works
> fine?
> 
> Other than that, it all sounds good to me.

I honestly don't know because I lack the expertise and knowledge.
LibreSSL has removed some features (env vars like SSL_CERT_FILE, ENGINE
support) and added some other features. I cannot tell if stdlib ssl +
LibreSSL is even secure. It probably is *if and only if* LibreSSL is
100% compatible to OpenSSL.

Christian

From mal at egenix.com  Wed Aug 31 06:20:28 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 31 Aug 2016 12:20:28 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <91e10464-9c0c-3cb4-499c-a5ee23907f79@python.org>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com>
 <7f12976b-aae0-4a42-e591-86b87cdc47dd@python.org>
 <57C6A47D.4050107@egenix.com>
 <91e10464-9c0c-3cb4-499c-a5ee23907f79@python.org>
Message-ID: <57C6AF6C.1040709@egenix.com>

On 31.08.2016 12:05, Christian Heimes wrote:
> This was my last reply to your mails on this topic. It's clear to me
> that you are not open to Cory's, Nick's or my arguments and that you
> won't change your position. More replies are just a waste of my limited
> time.

I *am* open to arguments, but so far I have not seen a compelling
one which strikes the right balance.

And if you have read my replies, I'm only suggesting to postpone
the switch to 1.0.2 by one release and even there I said that
it's not all that dramatic if this doesn't happen due to the way
the timelines for 3.6 and 3.7 work.

What I am seriously worried about, is the next step ...

> Instead I'm going to focus on a PEP to define OpenSSL support and to
> auto-deprecate unsupported OpenSSL versions. The PEP is a very high
> chance to get accepted. Everybody except you support the plan.

... which would then mean: Python's compatibility roadmap will
be dictated by OpenSSL.

I won't buy into that, sorry. Crypto is a helper in certain
situations, it's not what Python is all about. We should not
let OpenSSL dictate how and when we deprecate platforms or
OS versions.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From steve at pearwood.info  Wed Aug 31 07:34:37 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 31 Aug 2016 21:34:37 +1000
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CAP7+vJLuV_34PtCN5AXueEOoRQFHLLnyDVgFZPfUcbDnZueJNA@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
 <20160831010040.GM26300@ando.pearwood.info>
 <CAP7+vJLuV_34PtCN5AXueEOoRQFHLLnyDVgFZPfUcbDnZueJNA@mail.gmail.com>
Message-ID: <20160831113437.GP26300@ando.pearwood.info>

On Tue, Aug 30, 2016 at 07:15:55PM -0700, Guido van Rossum wrote:
> On Tue, Aug 30, 2016 at 6:00 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> > On Tue, Aug 30, 2016 at 02:20:26PM -0700, Guido van Rossum wrote:
> >> I'm happy to present PEP 526 for your collective review:
> >
> > Are you hoping to get this in before 3.6 beta? Because I'm not sure I
> > can give this much attention before then, but I really want to.
> 
> Yes I am hoping for that. Unlike PEP 484, this PEP is forward-looking
> (more like PEP 492, async/await), and the sooner we can get it in the
> sooner people who want to use it won't have to worry about supporting
> older Python versions. (And am I ever looking forward to the day when
> Python 3.5 counts as "older". :-)

Indeed :-)

Okay, I'll bump it up my list of priorities. (Sleep? Who needs that? :-) 


-- 
Steve

From ncoghlan at gmail.com  Wed Aug 31 07:57:49 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 21:57:49 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C6A487.2090301@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com> <20160831104301.4e61c428@fsol>
 <57C6A487.2090301@egenix.com>
Message-ID: <CADiSq7cOH6CHaupOLywHuDJ9Lm-XyuVRmXQi-8va=P6Txcw0zQ@mail.gmail.com>

On 31 August 2016 at 19:33, M.-A. Lemburg <mal at egenix.com> wrote:
> On 31.08.2016 10:43, Antoine Pitrou wrote:
>> On Wed, 31 Aug 2016 10:31:12 +0200
>> "M.-A. Lemburg" <mal at egenix.com> wrote:
>>>
>>> I am thinking of Python users out there who are running on LTS
>>> OS releases simply because their IT doesn't let them run anything
>>> else.
>>
>> There is a solution nowadays, which is to use Anaconda (or Miniconda).
>
> Sure, or use ActivePython or eGenix PyRun :-)
>
> But is that really what we want to tell people ?

I'm personally entirely comfortable with it, as large organisations
running community supported code without investing back into the
upstream community accordingly is currently a major problem in the
Python ecosystem - many technical folks find it easier to reach out to
the open source community for better support than they do to go into
battle with their own Finance departments to argue for appropriate
investment in managing their supply chain. Unfortunately, while that's
an entirely understandable reaction to an all too common form of
organisational dysfunction, it's also a major contributor to community
volunteer burnout.

Accordingly, we need more of these organisations to either fund paid
upstream development directly (e.g. by assigning their own staff to do
it or hiring existing core developers), or else for them to start
paying commercial redistributors, and making it clear that they expect
those redistributors to fund ongoing upstream development and
maintenance activities on their behalf. For folks that are already
paying commercial redistributors, we need them to be asking pointed
questions of their support managers, like "We're paying you for
commercial CPython support, so why don't you have anyone assigned to
work on it full time?"

Adopting that strategy isn't without its risks - some organisations
may react by banning the use of Python entirely and go looking for a
less assertive community (or one with better established funding
sources), rather than finding ways to pay for suitable infrastructure
support arrangements. However, hopefully folks within such
organisations will understand their political environment well enough
to know whether or not they need to stay under the executive radar.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug 31 08:02:54 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 22:02:54 +1000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <57C6AF6C.1040709@egenix.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com>
 <7f12976b-aae0-4a42-e591-86b87cdc47dd@python.org>
 <57C6A47D.4050107@egenix.com>
 <91e10464-9c0c-3cb4-499c-a5ee23907f79@python.org>
 <57C6AF6C.1040709@egenix.com>
Message-ID: <CADiSq7e48thxkC4JGv-3aEk+F-gwbu9dYWHvP34r2F7EcxSgPA@mail.gmail.com>

On 31 August 2016 at 20:20, M.-A. Lemburg <mal at egenix.com> wrote:
> ... which would then mean: Python's compatibility roadmap will
> be dictated by OpenSSL.
>
> I won't buy into that, sorry. Crypto is a helper in certain
> situations, it's not what Python is all about. We should not
> let OpenSSL dictate how and when we deprecate platforms or
> OS versions.

It won't dictate general support for those platforms, it will dictate
support for the *ssl module* on those platforms. If someone isn't
making secure network connections from Python, things will work fine.
If a redistributor is stepping in to provide the assertion that the
network connection is secure despite our upstream misgivings, things
will work fine.

Connections will only fail in cases where neither we nor a
redistributor are prepared to make the assertion that a requested
secure network connection will actually be secure.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From levkivskyi at gmail.com  Wed Aug 31 08:52:38 2016
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 31 Aug 2016 15:52:38 +0300
Subject: [Python-Dev] PEP 526 ready for review: Syntax for Variable and
 Attribute Annotations
In-Reply-To: <CADiSq7dbBzQ0ZmxrLCvyb3WTz3eByKYQ4kb+yXi=D2=FF_L8oA@mail.gmail.com>
References: <CAP7+vJ+nZg_Yu5RBoSV1CZJPEWrSvMwExS+rXC5vxWsMnNBQEA@mail.gmail.com>
 <20160831010040.GM26300@ando.pearwood.info>
 <CACLn2+2LHn-FmMKGnqxgP-Rd92sBOjwuACqzEsdhkXkhqyeXAA@mail.gmail.com>
 <CAP7+vJJ1o-Kjsj-6gwk0r22s6u=j=5qKz8tAjOqmiLR7Mh+GPQ@mail.gmail.com>
 <CACLn2+0BXNue_R8jB5dvm1jTU_Amn85aoADkhBUS7jDKMs14hA@mail.gmail.com>
 <CADiSq7dCncrJLHWLC259-oxemjgkRsZ2R9C2q5geYnzO-Mk=Vg@mail.gmail.com>
 <CAP7+vJJMvT9CQQfNsEmAV1YeqX99O=0EeeZeORHYOjfS9400Sg@mail.gmail.com>
 <CADiSq7dbBzQ0ZmxrLCvyb3WTz3eByKYQ4kb+yXi=D2=FF_L8oA@mail.gmail.com>
Message-ID: <CAOMjWkkb5ybHrrkS2JYeBcBsbUcO0JkEkbSonqUSbyQK402L+Q@mail.gmail.com>

On 31 August 2016 at 13:09, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I guess as long as they're included somewhere in the AST for the
> function body, I don't mind if the translation to bytecode throws them
> away - that's essentially saying that a function level type annotation
> is effectively interpreted as if it was:
>
>     if False:
>         __annotations__[<varname>] = <annotation>
>
> So the code generator will pick up syntax errors during normal
> execution, but not runtime errors (since the expression never actually
> gets evaluated).
>

Nick, you are right, in current implementation they are included in AST
in exactly the same way as in classes and modules, but compiler throws them
away.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160831/65522754/attachment.html>

From mal at egenix.com  Wed Aug 31 12:56:19 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 31 Aug 2016 18:56:19 +0200
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CADiSq7e48thxkC4JGv-3aEk+F-gwbu9dYWHvP34r2F7EcxSgPA@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com>
 <7f12976b-aae0-4a42-e591-86b87cdc47dd@python.org>
 <57C6A47D.4050107@egenix.com>
 <91e10464-9c0c-3cb4-499c-a5ee23907f79@python.org>
 <57C6AF6C.1040709@egenix.com>
 <CADiSq7e48thxkC4JGv-3aEk+F-gwbu9dYWHvP34r2F7EcxSgPA@mail.gmail.com>
Message-ID: <57C70C33.30106@egenix.com>

On 31.08.2016 14:02, Nick Coghlan wrote:
> On 31 August 2016 at 20:20, M.-A. Lemburg <mal at egenix.com> wrote:
>> ... which would then mean: Python's compatibility roadmap will
>> be dictated by OpenSSL.
>>
>> I won't buy into that, sorry. Crypto is a helper in certain
>> situations, it's not what Python is all about. We should not
>> let OpenSSL dictate how and when we deprecate platforms or
>> OS versions.
> 
> It won't dictate general support for those platforms, it will dictate
> support for the *ssl module* on those platforms. If someone isn't
> making secure network connections from Python, things will work fine.
> If a redistributor is stepping in to provide the assertion that the
> network connection is secure despite our upstream misgivings, things
> will work fine.
> 
> Connections will only fail in cases where neither we nor a
> redistributor are prepared to make the assertion that a requested
> secure network connection will actually be secure.

Yes, you can view it that way. Your car works, but access
to fuel is severely limited ;-)

The way things are developing around Python, if pip doesn't
work, your Python installation cannot be considered
working.

Hmm, perhaps pip ought to fallback to curl or wget if there's
no ssl module. Or we move away entirely from HTTPS and use
properly signed Python packages - ah, nevermind, it's not the
first time that have different views than many other core
devs. That's fine.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From brett at python.org  Wed Aug 31 12:58:17 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 31 Aug 2016 16:58:17 +0000
Subject: [Python-Dev] Supported versions of OpenSSL
In-Reply-To: <CADiSq7e48thxkC4JGv-3aEk+F-gwbu9dYWHvP34r2F7EcxSgPA@mail.gmail.com>
References: <48ae9561-1452-aa43-f003-d8775a546cd0@python.org>
 <57C3EDB0.9030305@egenix.com>
 <F4D91DAB-F774-446D-B3A1-A15168388D94@lukasa.co.uk>
 <57C48D7B.4080200@egenix.com>
 <e6fb8299-7ea8-95ab-4b5c-5c6a11c9490a@python.org>
 <57C5E79B.9010302@egenix.com>
 <CAGE7PNJLdrdzXVirxgNxa3hqeF8YzSHE_VaOk0WU3TUoNg4LTg@mail.gmail.com>
 <57C695D0.4020406@egenix.com>
 <7f12976b-aae0-4a42-e591-86b87cdc47dd@python.org>
 <57C6A47D.4050107@egenix.com>
 <91e10464-9c0c-3cb4-499c-a5ee23907f79@python.org>
 <57C6AF6C.1040709@egenix.com>
 <CADiSq7e48thxkC4JGv-3aEk+F-gwbu9dYWHvP34r2F7EcxSgPA@mail.gmail.com>
Message-ID: <CAP1=2W7WJHDBFDDnWfv37DkNjzaeBrFOb1TuY6a1YvPiCinFRQ@mail.gmail.com>

I think it's time for this thread to stop as everyone seems to be talking
in circles. Christian said he's going to write a PEP so let's wait for that
before discussing this any further so we have a concrete proposal to focus
around.

On Wed, 31 Aug 2016 at 05:04 Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 31 August 2016 at 20:20, M.-A. Lemburg <mal at egenix.com> wrote:
> > ... which would then mean: Python's compatibility roadmap will
> > be dictated by OpenSSL.
> >
> > I won't buy into that, sorry. Crypto is a helper in certain
> > situations, it's not what Python is all about. We should not
> > let OpenSSL dictate how and when we deprecate platforms or
> > OS versions.
>
> It won't dictate general support for those platforms, it will dictate
> support for the *ssl module* on those platforms. If someone isn't
> making secure network connections from Python, things will work fine.
> If a redistributor is stepping in to provide the assertion that the
> network connection is secure despite our upstream misgivings, things
> will work fine.
>
> Connections will only fail in cases where neither we nor a
> redistributor are prepared to make the assertion that a requested
> secure network connection will actually be secure.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160831/c7985e0a/attachment.html>

From christian at python.org  Wed Aug 31 16:31:14 2016
From: christian at python.org (Christian Heimes)
Date: Wed, 31 Aug 2016 22:31:14 +0200
Subject: [Python-Dev] Patch reviews
Message-ID: <nq7eqg$oij$1@blaine.gmane.org>

Hi,

I have 7 patches for 3.6 ready for merging. The new features were
discussed on Security-SIG and reviewed by Victor or GPS. The patches
just need one final review and an ACK. The first three patches should
land in 2.7, 3.4 and 3.5, too.

http://bugs.python.org/issue26470
Make OpenSSL module compatible with OpenSSL 1.1.0

https://bugs.python.org/issue27850
Remove 3DES from cipher list (sweet32 CVE-2016-2183)
Also adds ChaCha20 Poly1305

http://bugs.python.org/issue27691
X509 cert with GEN_RID subject alt name causes SytemError

http://bugs.python.org/issue27866
ssl: get list of enabled ciphers

https://bugs.python.org/issue27744
Add AF_ALG (Linux Kernel crypto) to socket module

http://bugs.python.org/issue16113
Add SHA-3 and SHAKE (Keccak) support

http://bugs.python.org/issue26798
add BLAKE2 to hashlib

Christian