From rosuav at gmail.com  Wed Aug  1 02:53:43 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 1 Aug 2018 16:53:43 +1000
Subject: [Python-Dev] Confused on git commit tree about Lib/datetime.py
In-Reply-To: <CAATxhGekbwXwU1dNEW=Y-5mV7MXZOsC8AEC3dcJ6sSxGmXFP+g@mail.gmail.com>
References: <CAATxhGekbwXwU1dNEW=Y-5mV7MXZOsC8AEC3dcJ6sSxGmXFP+g@mail.gmail.com>
Message-ID: <CAPTjJmoGaDvOBGnXgo7ro8EwLW523q6KhBSmBuMZPfYogY1i4Q@mail.gmail.com>

On Wed, Aug 1, 2018 at 1:16 PM, Jeffrey Zhang <zhang.lei.fly at gmail.com> wrote:
> I found a interesting issue when checking the Lib/datetime.py implementation
> in python3
>
> This patch is introduced by cf86e368ebd17e10f68306ebad314eea31daaa1e [0].
> But if you
> check the github page[0], or using git tag --contains, you will find v2.7.x
> includes this commit too.
>
> $ git tag --contains cf86e368ebd17e10f68306ebad314eea31daaa1e
> 3.2
> v2.7.10
> v2.7.10rc1
> v2.7.11
> v2.7.11rc1
> ...
>
> whereas, if you check the v2.7.x code base, nothing is found
>
> $ git log v2.7.4 -- Lib/datetime.py
> <nothing here>
>
> I guess it maybe a git tool bug, or the commit tree is messed up. Is there
> any guys could explain this
> situation?

I suppose you could say that the commit tree is "messed up", in a
sense, but it's not truly messed up, just a little odd. It's a
consequence of the way merges have been done in the CPython
repository. Nothing is actually broken, except for the ability to
track down a commit the way you're doing.

ChrisA

From chris.jerdonek at gmail.com  Wed Aug  1 03:31:31 2018
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Wed, 1 Aug 2018 00:31:31 -0700
Subject: [Python-Dev] Confused on git commit tree about Lib/datetime.py
In-Reply-To: <CAPTjJmoGaDvOBGnXgo7ro8EwLW523q6KhBSmBuMZPfYogY1i4Q@mail.gmail.com>
References: <CAATxhGekbwXwU1dNEW=Y-5mV7MXZOsC8AEC3dcJ6sSxGmXFP+g@mail.gmail.com>
 <CAPTjJmoGaDvOBGnXgo7ro8EwLW523q6KhBSmBuMZPfYogY1i4Q@mail.gmail.com>
Message-ID: <CAOTb1wftv=W25pGSyOUOrWb5Hhmy488ZA=7kAMVHN6Bf7K9ZYw@mail.gmail.com>

FWIW, it looks like this is the first (earliest) merge commit that
caused the v2.7.4 line to contain
cf86e368ebd17e10f68306ebad314eea31daaa1e:

    $ git show -q d26b658f1433a28b611906c078f47bc804a63dd1
    commit d26b658f1433a28b611906c078f47bc804a63dd1
    Merge: 2d639d5665 f8b9dfd9a1
    Author: Georg Brandl <georg at python.org>
    Date:   Sat Aug 11 11:08:04 2012 +0200

        Graft a89d654adaa2 from 3.2 branch. Fixes #15620.

--Chris


On Tue, Jul 31, 2018 at 11:53 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Wed, Aug 1, 2018 at 1:16 PM, Jeffrey Zhang <zhang.lei.fly at gmail.com> wrote:
>> I found a interesting issue when checking the Lib/datetime.py implementation
>> in python3
>>
>> This patch is introduced by cf86e368ebd17e10f68306ebad314eea31daaa1e [0].
>> But if you
>> check the github page[0], or using git tag --contains, you will find v2.7.x
>> includes this commit too.
>>
>> $ git tag --contains cf86e368ebd17e10f68306ebad314eea31daaa1e
>> 3.2
>> v2.7.10
>> v2.7.10rc1
>> v2.7.11
>> v2.7.11rc1
>> ...
>>
>> whereas, if you check the v2.7.x code base, nothing is found
>>
>> $ git log v2.7.4 -- Lib/datetime.py
>> <nothing here>
>>
>> I guess it maybe a git tool bug, or the commit tree is messed up. Is there
>> any guys could explain this
>> situation?
>
> I suppose you could say that the commit tree is "messed up", in a
> sense, but it's not truly messed up, just a little odd. It's a
> consequence of the way merges have been done in the CPython
> repository. Nothing is actually broken, except for the ability to
> track down a commit the way you're doing.
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com

From J.Demeyer at UGent.be  Wed Aug  1 06:17:34 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Wed, 1 Aug 2018 12:17:34 +0200
Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
In-Reply-To: <f206f6a75e9241d3a1ae3bf9ae060182@xmail101.UGent.be>
References: <edfc113afbfd48888f5317337644f9af@xmail101.UGent.be>
 <5B6021EF.5080908@UGent.be>
 <f206f6a75e9241d3a1ae3bf9ae060182@xmail101.UGent.be>
Message-ID: <5B6188BE.5060708@UGent.be>

On 2018-07-31 11:12, INADA Naoki wrote:
> Any PEP won't be accepted in few month, because we don't have flow to
> accept PEPs for now.

Is that certain? I haven't been following the process discussions, so 
I'm just asking the question. For example, given that you are already 
looking at PEP 580, would it be possible for you to handle PEP 580 as 
official BDFL-Delegate (even if there is no BDFL)?

> So it's worthless that waiting PEP accepted before start PoC.

First of all, it's too early for a proof-of-concept of native C calling. 
We first have to design the theory before we can start implementing 
anything.

But even if we could start to write a proof of concept, I would really 
prefer doing that on top of PEP 580. There are two reasons for this:

1. If PEP 580 is rejected, I very much doubt that the native C calling 
protocol will be accepted.

2. I would be good to use PEP 580 as a framework for the implementation. 
Otherwise we have to implement it twice: once before PEP 580 with the 
proof-of-concept and then again after PEP 580 with the "real" 
implementation.


Jeroen.

From tjreedy at udel.edu  Wed Aug  1 10:45:40 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 1 Aug 2018 10:45:40 -0400
Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
In-Reply-To: <5B6188BE.5060708@UGent.be>
References: <edfc113afbfd48888f5317337644f9af@xmail101.UGent.be>
 <5B6021EF.5080908@UGent.be>
 <f206f6a75e9241d3a1ae3bf9ae060182@xmail101.UGent.be>
 <5B6188BE.5060708@UGent.be>
Message-ID: <pjsgui$2je$1@blaine.gmane.org>

On 8/1/2018 6:17 AM, Jeroen Demeyer wrote:
> On 2018-07-31 11:12, INADA Naoki wrote:
>> Any PEP won't be accepted in few month, because we don't have flow to
>> accept PEPs for now.
> 
> Is that certain? I haven't been following the process discussions, so 
> I'm just asking the question. For example, given that you are already 
> looking at PEP 580, would it be possible for you to handle PEP 580 as 
> official BDFL-Delegate (even if there is no BDFL)?

I think this would be a reasonable thing to discuss either here or on 
the committers list or both.

-- 
Terry Jan Reedy


From aixtools at felt.demon.nl  Wed Aug  1 05:58:32 2018
From: aixtools at felt.demon.nl (Michael)
Date: Wed, 1 Aug 2018 11:58:32 +0200
Subject: [Python-Dev] test_zlib.py suggestion
Message-ID: <9cb0f6b1-26c6-a1f0-44e8-cca0358ef06b@felt.demon.nl>

I have a build_bot running (yeah me!), and was surprised to see 
test_zlib fail on AIX.

There is not an issue with test_zlib, but I do have a suggestion.

I was getting an error with test_flushes(). On python2-2.7.15 the test 
passes and in python3-3.8 (and earlier I expect) the test fails. The 
difference is the addition of the two modes 'Z_PARTIAL_FLUSH' and 'Z_BLOCK'

sync_opt = ['Z_NO_FLUSH', 'Z_SYNC_FLUSH', 'Z_FULL_FLUSH',
 ??????????? 'Z_PARTIAL_FLUSH', 'Z_BLOCK']

On default AIX installs zlib is, sadly, still at version 1.2.3 and 
Z_BLOCK was 'improved' in version 1.2.5. And it fails on Z_BLOCK.

My suggestion is to enhance the test_library_version() so that it 
verifies that the minor (or smaller?) number is at least '5'

When I static link python with zlib-1.2.11 then the test_zlib passes all 
sub-tests.

Again, not a python bug - but a suggestion for improving what is tested.

I can open an issue, and with a bit of assistance/interest from others 
I'll even do a PR.

Michael


From aixtools at felt.demon.nl  Wed Aug  1 07:58:14 2018
From: aixtools at felt.demon.nl (Michael)
Date: Wed, 1 Aug 2018 13:58:14 +0200
Subject: [Python-Dev] spwd and AIX
Message-ID: <06e1963d-ca26-7ea2-6aca-84fad1c51455@felt.demon.nl>

a) I am looking at getting spwd integrated from AIX

b) only the parameter sp_pwdp is my concern - as AIX really does not 
want to reveal the encrypted password. Rather, AIX will say '!' (meaning 
there is, or should be a shadow password, or '*' - no user password). 
Would this horribly break things if only '!' was returned, rather than 
the shadow password?

It does not look terribly hard - but how do you deal with defaults such 
as 0 or -1 for the numeric values?

Regards,

Michael


From christian at python.org  Wed Aug  1 11:52:43 2018
From: christian at python.org (Christian Heimes)
Date: Wed, 1 Aug 2018 17:52:43 +0200
Subject: [Python-Dev] spwd and AIX
In-Reply-To: <06e1963d-ca26-7ea2-6aca-84fad1c51455@felt.demon.nl>
References: <06e1963d-ca26-7ea2-6aca-84fad1c51455@felt.demon.nl>
Message-ID: <pjsks7$l59$1@blaine.gmane.org>

On 2018-08-01 13:58, Michael wrote:
> a) I am looking at getting spwd integrated from AIX
> 
> b) only the parameter sp_pwdp is my concern - as AIX really does not
> want to reveal the encrypted password. Rather, AIX will say '!' (meaning
> there is, or should be a shadow password, or '*' - no user password).
> Would this horribly break things if only '!' was returned, rather than
> the shadow password?
> 
> It does not look terribly hard - but how do you deal with defaults such
> as 0 or -1 for the numeric values?

Please note that I'm planning to deprecate and remove the spwd module.
The module has several limitation. Most importantly it bypasses the PAM
stack. Therefore it's not compatible with password policies or other
password storages like LDAP.

Christian


From brett at python.org  Wed Aug  1 12:03:53 2018
From: brett at python.org (Brett Cannon)
Date: Wed, 1 Aug 2018 09:03:53 -0700
Subject: [Python-Dev] test_zlib.py suggestion
In-Reply-To: <9cb0f6b1-26c6-a1f0-44e8-cca0358ef06b@felt.demon.nl>
References: <9cb0f6b1-26c6-a1f0-44e8-cca0358ef06b@felt.demon.nl>
Message-ID: <CAP1=2W4Dwdv3Z8Ln-uj4iJLgP=XcKTmxBrCgZ2zNQR71_=CHSQ@mail.gmail.com>

Open an issue as this will surely get forgotten otherwise, then people can
discuss on the issue how to handle this.

On Wed, 1 Aug 2018 at 08:40 Michael <aixtools at felt.demon.nl> wrote:

> I have a build_bot running (yeah me!), and was surprised to see
> test_zlib fail on AIX.
>
> There is not an issue with test_zlib, but I do have a suggestion.
>
> I was getting an error with test_flushes(). On python2-2.7.15 the test
> passes and in python3-3.8 (and earlier I expect) the test fails. The
> difference is the addition of the two modes 'Z_PARTIAL_FLUSH' and 'Z_BLOCK'
>
> sync_opt = ['Z_NO_FLUSH', 'Z_SYNC_FLUSH', 'Z_FULL_FLUSH',
>              'Z_PARTIAL_FLUSH', 'Z_BLOCK']
>
> On default AIX installs zlib is, sadly, still at version 1.2.3 and
> Z_BLOCK was 'improved' in version 1.2.5. And it fails on Z_BLOCK.
>
> My suggestion is to enhance the test_library_version() so that it
> verifies that the minor (or smaller?) number is at least '5'
>
> When I static link python with zlib-1.2.11 then the test_zlib passes all
> sub-tests.
>
> Again, not a python bug - but a suggestion for improving what is tested.
>
> I can open an issue, and with a bit of assistance/interest from others
> I'll even do a PR.
>
> Michael
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180801/fcdb8d8a/attachment.html>

From brett at python.org  Wed Aug  1 12:05:52 2018
From: brett at python.org (Brett Cannon)
Date: Wed, 1 Aug 2018 09:05:52 -0700
Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
In-Reply-To: <pjsgui$2je$1@blaine.gmane.org>
References: <edfc113afbfd48888f5317337644f9af@xmail101.UGent.be>
 <5B6021EF.5080908@UGent.be>
 <f206f6a75e9241d3a1ae3bf9ae060182@xmail101.UGent.be>
 <5B6188BE.5060708@UGent.be> <pjsgui$2je$1@blaine.gmane.org>
Message-ID: <CAP1=2W6iz5cbaaqT3NzN76BEbFWZ1xzv0zhMGoZyQ=jbWqMPBQ@mail.gmail.com>

On Wed, 1 Aug 2018 at 07:47 Terry Reedy <tjreedy at udel.edu> wrote:

> On 8/1/2018 6:17 AM, Jeroen Demeyer wrote:
> > On 2018-07-31 11:12, INADA Naoki wrote:
> >> Any PEP won't be accepted in few month, because we don't have flow to
> >> accept PEPs for now.
> >
> > Is that certain? I haven't been following the process discussions, so
> > I'm just asking the question. For example, given that you are already
> > looking at PEP 580, would it be possible for you to handle PEP 580 as
> > official BDFL-Delegate (even if there is no BDFL)?
>
> I think this would be a reasonable thing to discuss either here or on
> the committers list or both.
>

If there was an absolute certainty of who the BDFL delegate would be then
we might be able to not wait, but without that I don't know if enough core
devs will feel comfortable choosing one right now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180801/b5450460/attachment.html>

From brett at python.org  Wed Aug  1 12:09:42 2018
From: brett at python.org (Brett Cannon)
Date: Wed, 1 Aug 2018 09:09:42 -0700
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
Message-ID: <CAP1=2W6cY=_RmhxTk36jWNyEj4eQv5b+ht+K32CvwbpjZFr6wg@mail.gmail.com>

On Tue, 31 Jul 2018 at 15:16 Victor Stinner <vstinner at redhat.com> wrote:

> Hi,
>
> I finished my work on the _PyCoreConfig structure: it's a C structure
> in Include/pystate.h which has many fields used to configure Python
> initialization. In Python 3.6 and older, these parameters were scatted
> around the code, and it was hard to get an exhaustive list of it.
>

That's great! Thanks for doing this! I know I have always found it hard to
track down where stuff in the start-up process ultimately gets set.


>
> This work is linked to the Nick Coghlan's PEP 432 "Restructuring the
> CPython startup sequence":
> https://www.python.org/dev/peps/pep-0432/
>
> Right now, the new API is still private. Nick Coghlan splitted the
> initialization in two parts: "core" and "main". I'm not sure that this
> split is needed. We should see what to do, but it would be nice to
> make the _PyCoreConfig API public! IMHO it's way better than the old
> way to configuration Python initialization.
>
> --
>
> It is now possible to only use _PyCoreConfig to initialize Python: it
> overrides old ways to configure Python like environment variables (ex:
> PYTHONPATH), global configuration variables (ex: Py_BytesWarningFlag)
> and C functions (ex: Py_SetProgramName()).
>
> I added tests to test_embed on the different ways to configure Python
> initialization:
>
> * environment variables (ex: PYTHONPATH)
> * global configuration variables (ex: Py_BytesWarningFlag) and C
> functions (ex: Py_SetProgramName())
> * _PyCoreConfig
>
> I found and fixed many issues when writing these tests :-)
>
> Reading the current configuration, _PyCoreConfig_Read(), no longer
> changes the configuration. Now the code to read the configuration and
> the code to apply the configuration is properly separated.
>
> The work is not fully complete, there are a few remaining corner cases
> and some parameters (ex: Py_FrozenFlag) which cannot be set by
> _PyCoreConfig yet. My latest issue used to work on this API:
>
> https://bugs.python.org/issue34170
>
> I had to refactor a lot of code to implement all of that.
>
> --
>
> The problem is that Python 3.7 got the half-baked implementation, and
> it caused issues:
>
> * Calling Py_Main() after Py_Initialize() fails with a fatal error on
> Python 3.7.0
>   https://bugs.python.org/issue34008
> * PYTHONOPTIMIZE environment variable is ignored by Py_Initialize()
>    https://bugs.python.org/issue34247
>
> I fixed the first issue, I'm now working on the second one to see how
> it can be fixed. Other option would be to backport the code from
> master to the 3.7 branch, since the code in master has a way better
> design. But it requires to backport a lot of changes. I'm not sure yet
> what is the best option.
>

If it is not extremely painful to fix just the issue then I say don't
backport the whole thing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180801/a224c629/attachment.html>

From brett at python.org  Wed Aug  1 12:17:26 2018
From: brett at python.org (Brett Cannon)
Date: Wed, 1 Aug 2018 09:17:26 -0700
Subject: [Python-Dev] Using Cython for the stdlib (was: Let's change to C
 API!)
In-Reply-To: <pjqhcb$u66$1@blaine.gmane.org>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be>
 <20180731094528.118471f9@fsol> <pjqhcb$u66$1@blaine.gmane.org>
Message-ID: <CAP1=2W4HJRka2OAsAAozx+VQ-4Hh+4=rMHv-SpWV6NES_fnsow@mail.gmail.com>

[new thread as this no longer has anything to do with Victor's proposal]

On Tue, 31 Jul 2018 at 13:42 Stefan Behnel <stefan_ml at behnel.de> wrote:

> Antoine Pitrou schrieb am 31.07.2018 um 09:45:
> > On Tue, 31 Jul 2018 09:27:03 +0200
> > Jeroen Demeyer <J.Demeyer at UGent.be> wrote:
> >> On 2018-07-31 08:58, Antoine Pitrou wrote:
> >>> I think Stefan is right that we
> >>> should push people towards Cython and alternatives, rather than direct
> >>> use of the C API (which people often fail to use correctly, in my
> >>> experience).
> >>
> >> I know this probably isn't the correct place to bring it up, but I'm
> >> sure that CPython itself could benefit from using Cython. For example,
> >> most of the C extensions in Modules/ could be written in Cython.
> >
> > We don't depend on any third-party Python modules.  Adding a Cython
> > dependency for CPython development would be a tough sell.
>
> I don't really want to get into that discussion (it's more about processes
> than arguments), but let me note that the CPython development already has a
> couple of dependencies, such as github and its bots, or tools like argument
> clinic (admittedly included), make and a C compiler (not included), and a
> text editor. It's not like it's free of tools that help in writing and
> maintaining the code. That's pretty much the level at which I also see
> Cython. It's more complex than argument clinic, but it otherwise serves a
> similar need.
>

You're right that since Cython generates C code that we could check in like
we do all of our other generated code we could view Cython as another
external tool we rely on for maintaining the code base.


>
>
> > Also, a C extension can be built-in (linked statically into the
> > interpreter), which I think would be hard to do with Cython.
>
> Someone recently contributed a feature of hiding the pyinit function for
> the embedding case, so people do these things already. This could use the
> normal inittab mechanism, for example. What I think you might be referring
> to is that Cython modules require the CPython runtime to be initialised to
> a certain extent, so you couldn't implement "sys" in Cython, for example.
>

I think the key thing is that on Windows all extension modules are built-in
modules, so that use-case would need to be supported (I don't know Cython
well enough to know whether this would be doable if we converted as much as
possible to Cython itself).


> But Jeroen is right, Cython should be a viable option for (most of?) the
> extension modules in the stdlib. Whether the CPython core devs would accept
> it in their workflow or not is a totally different question.
>

I think it's definitely something to consider, although I think Victor's
proposal to clean up the C API so we have a clearly delineated private API,
a version-specific API, and a stable ABI is a separate idea and so this
should be its own thread (plus Victor's is easier to accomplish in the
short-term ;) .
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180801/d0d980d3/attachment.html>

From solipsis at pitrou.net  Wed Aug  1 12:30:14 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 1 Aug 2018 18:30:14 +0200
Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
References: <edfc113afbfd48888f5317337644f9af@xmail101.UGent.be>
 <5B6021EF.5080908@UGent.be>
 <f206f6a75e9241d3a1ae3bf9ae060182@xmail101.UGent.be>
 <5B6188BE.5060708@UGent.be> <pjsgui$2je$1@blaine.gmane.org>
 <CAP1=2W6iz5cbaaqT3NzN76BEbFWZ1xzv0zhMGoZyQ=jbWqMPBQ@mail.gmail.com>
Message-ID: <20180801183014.06932148@fsol>

On Wed, 1 Aug 2018 09:05:52 -0700
Brett Cannon <brett at python.org> wrote:
> On Wed, 1 Aug 2018 at 07:47 Terry Reedy <tjreedy at udel.edu> wrote:
> 
> > On 8/1/2018 6:17 AM, Jeroen Demeyer wrote:  
> > > On 2018-07-31 11:12, INADA Naoki wrote:  
> > >> Any PEP won't be accepted in few month, because we don't have flow to
> > >> accept PEPs for now.  
> > >
> > > Is that certain? I haven't been following the process discussions, so
> > > I'm just asking the question. For example, given that you are already
> > > looking at PEP 580, would it be possible for you to handle PEP 580 as
> > > official BDFL-Delegate (even if there is no BDFL)?  
> >
> > I think this would be a reasonable thing to discuss either here or on
> > the committers list or both.
> >  
> 
> If there was an absolute certainty of who the BDFL delegate would be then
> we might be able to not wait, but without that I don't know if enough core
> devs will feel comfortable choosing one right now.

We could proceed by consensus: the PEP author publicly proposes a PEP
delegate, and if no core developer opposes, that person is officially
accepted as delegate.

Often PEP authors have a pretty good idea of who can be a delegate for
a PEP.  This is especially true on specialized topics which only a
couple core devs are interested in discussing actively.

Regards

Antoine.



From steve.dower at python.org  Wed Aug  1 15:06:10 2018
From: steve.dower at python.org (Steve Dower)
Date: Wed, 1 Aug 2018 20:06:10 +0100
Subject: [Python-Dev] Using Cython for the stdlib (was: Let's change to
 CAPI!)
In-Reply-To: <CAP1=2W4HJRka2OAsAAozx+VQ-4Hh+4=rMHv-SpWV6NES_fnsow@mail.gmail.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <pjqhcb$u66$1@blaine.gmane.org>
 <CAP1=2W4HJRka2OAsAAozx+VQ-4Hh+4=rMHv-SpWV6NES_fnsow@mail.gmail.com>
Message-ID: <mailman.67.1533150372.2858.python-dev@python.org>

I don?t think there would be any trouble linking in the generated C files. The built in modules like this just have their *_init() functions called at startup, so possibly nothing would even change.

Top-posted from my Windows 10 phone

From: Brett Cannon
Sent: Wednesday, 1 August 2018 17:20
To: Stefan Behnel
Cc: python-dev at python.org
Subject: [Python-Dev] Using Cython for the stdlib (was: Let's change to CAPI!)

[new thread as this no longer has anything to do with Victor's proposal]

On Tue, 31 Jul 2018 at 13:42 Stefan Behnel <stefan_ml at behnel.de> wrote:
Antoine Pitrou schrieb am 31.07.2018 um 09:45:
> On Tue, 31 Jul 2018 09:27:03 +0200
> Jeroen Demeyer <J.Demeyer at UGent.be> wrote:
>> On 2018-07-31 08:58, Antoine Pitrou wrote:
>>> I think Stefan is right that we
>>> should push people towards Cython and alternatives, rather than direct
>>> use of the C API (which people often fail to use correctly, in my
>>> experience).? 
>>
>> I know this probably isn't the correct place to bring it up, but I'm 
>> sure that CPython itself could benefit from using Cython. For example, 
>> most of the C extensions in Modules/ could be written in Cython.
> 
> We don't depend on any third-party Python modules.? Adding a Cython
> dependency for CPython development would be a tough sell.

I don't really want to get into that discussion (it's more about processes
than arguments), but let me note that the CPython development already has a
couple of dependencies, such as github and its bots, or tools like argument
clinic (admittedly included), make and a C compiler (not included), and a
text editor. It's not like it's free of tools that help in writing and
maintaining the code. That's pretty much the level at which I also see
Cython. It's more complex than argument clinic, but it otherwise serves a
similar need.

You're right that since Cython generates C code that we could check in like we do all of our other generated code we could view Cython as another external tool we rely on for maintaining the code base.
?


> Also, a C extension can be built-in (linked statically into the
> interpreter), which I think would be hard to do with Cython.

Someone recently contributed a feature of hiding the pyinit function for
the embedding case, so people do these things already. This could use the
normal inittab mechanism, for example. What I think you might be referring
to is that Cython modules require the CPython runtime to be initialised to
a certain extent, so you couldn't implement "sys" in Cython, for example.

I think the key thing is that on Windows all extension modules are built-in modules, so that use-case would need to be supported (I don't know Cython well enough to know whether this would be doable if we converted as much as possible to Cython itself).
?
But Jeroen is right, Cython should be a viable option for (most of?) the
extension modules in the stdlib. Whether the CPython core devs would accept
it in their workflow or not is a totally different question.

I think it's definitely something to consider, although I think Victor's proposal to clean up the C API so we have a clearly delineated private API, a version-specific API, and a stable ABI is a separate idea and so this should be its own thread (plus Victor's is easier to accomplish in the short-term ;) . 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180801/95bfd05f/attachment.html>

From stefan_ml at behnel.de  Wed Aug  1 16:01:55 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 1 Aug 2018 22:01:55 +0200
Subject: [Python-Dev] Using Cython for the stdlib (was: Let's change to
 C API!)
In-Reply-To: <CAP1=2W4HJRka2OAsAAozx+VQ-4Hh+4=rMHv-SpWV6NES_fnsow@mail.gmail.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <pjqhcb$u66$1@blaine.gmane.org>
 <CAP1=2W4HJRka2OAsAAozx+VQ-4Hh+4=rMHv-SpWV6NES_fnsow@mail.gmail.com>
Message-ID: <pjt3fg$46p$1@blaine.gmane.org>

Brett Cannon schrieb am 01.08.2018 um 18:17:
> On Tue, 31 Jul 2018 at 13:42 Stefan Behnel wrote:
>> Antoine Pitrou schrieb am 31.07.2018 um 09:45:
>>> Also, a C extension can be built-in (linked statically into the
>>> interpreter), which I think would be hard to do with Cython.
>>
>> Someone recently contributed a feature of hiding the pyinit function for
>> the embedding case, so people do these things already. This could use the
>> normal inittab mechanism, for example. What I think you might be referring
>> to is that Cython modules require the CPython runtime to be initialised to
>> a certain extent, so you couldn't implement "sys" in Cython, for example.
> 
> I think the key thing is that on Windows all extension modules are built-in
> modules, so that use-case would need to be supported (I don't know Cython
> well enough to know whether this would be doable if we converted as much as
> possible to Cython itself).

As Steve noted, this is probably easy. What Cython produces is just the C
code file for an extension module. Whether you turn that into a shared
library or statically link it into something else (that knows how to
initialise an extension module) is up to you.

I would say, from the point on where CPython is ready to initialise its own
extension modules, it can also initialise Cython generated modules. So,
just to give an example, if you want to compile difflib.py into an
accelerator module and link that into the core, that's probably fine, as
long as you first initialise everything that difflib needs in its module
init code (such as importlib to execute the module level imports) before
you initialise the compiled difflib module.

Stefan


From armin.rigo at gmail.com  Wed Aug  1 17:40:46 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Wed, 1 Aug 2018 23:40:46 +0200
Subject: [Python-Dev] Error message for wrong number of arguments
In-Reply-To: <CALGmxE+gw4ApvLQFRoVFgqYF1THEsajbUx0uFoQdZhF8=PH+8g@mail.gmail.com>
References: <5B5F00A6.8070802@UGent.be>
 <e64de23cbbfd4ca79b545e7b822167b1@xmail101.UGent.be>
 <5B5F5B6E.1020300@UGent.be>
 <CALGmxE+gw4ApvLQFRoVFgqYF1THEsajbUx0uFoQdZhF8=PH+8g@mail.gmail.com>
Message-ID: <CAMSv6X1W7D9NROSKr96Dob-eDm1LRP4S+FZQao3uXOAFsq5QKA@mail.gmail.com>

Hi,

On 30 July 2018 at 22:19, Chris Barker via Python-Dev
<python-dev at python.org> wrote:
> Oh well. This is a serious usability issue -- but what can you do?

I think that argument clinic knows if the built-in function is
supposed to be a method or a function.  It doesn't look too hard to
add a new flag METH_IS_METHOD or something, which would be taken in
consideration in the common cases, and which can be added manually OR
used automatically by argument clinic.  This would not be a 100%
solution out of the box, but if the new wording is right, it shouldn't
be a problem.


A bient?t,

Armin.

From aixtools at felt.demon.nl  Wed Aug  1 17:52:42 2018
From: aixtools at felt.demon.nl (Michael Felt (aixtools))
Date: Wed, 1 Aug 2018 23:52:42 +0200
Subject: [Python-Dev] spwd and AIX
In-Reply-To: <pjsks7$l59$1@blaine.gmane.org>
References: <06e1963d-ca26-7ea2-6aca-84fad1c51455@felt.demon.nl>
 <pjsks7$l59$1@blaine.gmane.org>
Message-ID: <8B9746BE-81B2-43DE-93C5-487C48592892@felt.demon.nl>

Sounds like i can skip this then. Thx. 

Sent from my iPhone

> On 1 Aug 2018, at 17:52, Christian Heimes <christian at python.org> wrote:
> 
>> On 2018-08-01 13:58, Michael wrote:
>> a) I am looking at getting spwd integrated from AIX
>> 
>> b) only the parameter sp_pwdp is my concern - as AIX really does not
>> want to reveal the encrypted password. Rather, AIX will say '!' (meaning
>> there is, or should be a shadow password, or '*' - no user password).
>> Would this horribly break things if only '!' was returned, rather than
>> the shadow password?
>> 
>> It does not look terribly hard - but how do you deal with defaults such
>> as 0 or -1 for the numeric values?
> 
> Please note that I'm planning to deprecate and remove the spwd module.
> The module has several limitation. Most importantly it bypasses the PAM
> stack. Therefore it's not compatible with password policies or other
> password storages like LDAP.
> 
> Christian
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/aixtools%40felt.demon.nl


From ericsnowcurrently at gmail.com  Wed Aug  1 19:18:41 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 1 Aug 2018 17:18:41 -0600
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
Message-ID: <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>

On Tue, Jul 31, 2018 at 4:15 PM Victor Stinner <vstinner at redhat.com> wrote:
> I finished my work on the _PyCoreConfig structure:

\o/

Thanks for all the good work!

> Right now, the new API is still private. Nick Coghlan splitted the
> initialization in two parts: "core" and "main". I'm not sure that this
> split is needed.

The "core" config is basically the config for the runtime.  In fact,
PEP 432 renamed "core" to "runtime".  Please keep the firm distinction
between the runtime and the (main) interpreter.

> We should see what to do, but it would be nice to
> make the _PyCoreConfig API public! IMHO it's way better than the old
> way to configuration Python initialization.

+1

However, shouldn't that happen after PEP 432 is accepted?

> [snip]
>
> The problem is that Python 3.7 got the half-baked implementation, and
> it caused issues:
>
> * Calling Py_Main() after Py_Initialize() fails with a fatal error on
> Python 3.7.0
>   https://bugs.python.org/issue34008
> * PYTHONOPTIMIZE environment variable is ignored by Py_Initialize()
>    https://bugs.python.org/issue34247
>
> I fixed the first issue, I'm now working on the second one to see how
> it can be fixed. Other option would be to backport the code from
> master to the 3.7 branch, since the code in master has a way better
> design. But it requires to backport a lot of changes. I'm not sure yet
> what is the best option.

Backporting shouldn't be so risky since it's all private API and there
are few other changes in the relevant code since 3.7, right?  It
depends on if Ned's okay with it or not. :)

-eric

From barry at python.org  Wed Aug  1 20:15:52 2018
From: barry at python.org (Barry Warsaw)
Date: Wed, 1 Aug 2018 17:15:52 -0700
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
Message-ID: <24BFF505-C8D2-47E8-8BA7-B8E92543506E@python.org>

On Jul 31, 2018, at 15:14, Victor Stinner <vstinner at redhat.com> wrote:
> 
> I finished my work on the _PyCoreConfig structure: it's a C structure
> in Include/pystate.h which has many fields used to configure Python
> initialization. In Python 3.6 and older, these parameters were scatted
> around the code, and it was hard to get an exhaustive list of it.

Great work Victor!  +1 for making _PyCoreConfig public, although I?m sure you?re only proposing that for Python 3.8, not in any future backport.

> I had to refactor a lot of code to implement all of that.
> 
> The problem is that Python 3.7 got the half-baked implementation, and
> it caused issues:
> 
> * Calling Py_Main() after Py_Initialize() fails with a fatal error on
> Python 3.7.0
>  https://bugs.python.org/issue34008
> * PYTHONOPTIMIZE environment variable is ignored by Py_Initialize()
>   https://bugs.python.org/issue34247
> 
> I fixed the first issue, I'm now working on the second one to see how
> it can be fixed. Other option would be to backport the code from
> master to the 3.7 branch, since the code in master has a way better
> design. But it requires to backport a lot of changes. I'm not sure yet
> what is the best option.

Do you have WIP branch for the backport?  I agree that it?s probably low enough risk given the private nature of the API in 3.7, but that it?s up to Ned to decide.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180801/65622f5b/attachment.sig>

From turnbull.stephen.fw at u.tsukuba.ac.jp  Thu Aug  2 05:22:36 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Thu, 2 Aug 2018 18:22:36 +0900
Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
In-Reply-To: <20180801183014.06932148@fsol>
References: <edfc113afbfd48888f5317337644f9af@xmail101.UGent.be>
 <5B6021EF.5080908@UGent.be>
 <f206f6a75e9241d3a1ae3bf9ae060182@xmail101.UGent.be>
 <5B6188BE.5060708@UGent.be> <pjsgui$2je$1@blaine.gmane.org>
 <CAP1=2W6iz5cbaaqT3NzN76BEbFWZ1xzv0zhMGoZyQ=jbWqMPBQ@mail.gmail.com>
 <20180801183014.06932148@fsol>
Message-ID: <23394.52572.518091.536027@turnbull.sk.tsukuba.ac.jp>

Antoine Pitrou writes:

 > We could proceed by consensus: the PEP author publicly proposes a PEP
 > delegate, and if no core developer opposes, that person is officially
 > accepted as delegate.

I think this is too decentralized.  I think there should be enthusiasm
for the delegate, say two "seconds" (with discussion of why they think
the delegate is appropriate) besides the PEP proponent(s).

 > Often PEP authors have a pretty good idea of who can be a delegate for
 > a PEP.  This is especially true on specialized topics which only a
 > couple core devs are interested in discussing actively.

I agree that this is an important point.  I think you (disclaimer: I'm
a social scientist, but not a committer) should be careful that
delegates be broadly respected, and where possible have interests in
other areas of Python development besides the one they're delegate
for.


From vstinner at redhat.com  Thu Aug  2 05:49:50 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 2 Aug 2018 11:49:50 +0200
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
 <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
Message-ID: <CA+3bQGGiQFMkqz_YRommw4OUMKLyySPgGb1B4etB8smptTt_sg@mail.gmail.com>

2018-08-02 1:18 GMT+02:00 Eric Snow <ericsnowcurrently at gmail.com>:
> Backporting shouldn't be so risky since it's all private API and there
> are few other changes in the relevant code since 3.7, right?  It
> depends on if Ned's okay with it or not. :)

I'm still doing further bug fixes and cleanup in the master branch:
https://bugs.python.org/issue34170

I'm doing more and more changes. I just added two new files:
Include/coreconfig.h and Python/coreconfig.h. IMHO it's better to put
similar code in separated files.

FYI Python/coreconfig.c has 954 files and Include/coreconfig.h has 319 lines.

I'm ok to rename the structure and the files if you prefer a different name.

About that, I'm working on a subproject: abandon Py_xxx global
configuration variables to replace them with accessing
interpreter->core_config->xxx. I'm not sure yet if it's a good idea or
not, but it would allow to have two interpreters to have their own
different configuration. Imagine two interpreters with different
sys.path running in isolated mode. Or maybe an interpreter without
importlib?

One of the issue is that we have now two copies of the same option.
For example, Py_BytesWarningFlag and
interpreter->core_config->bytes_warning. That's why I would like to
switch to core_config.

But I'm also trying to make sure that the two variables have the same value:
https://github.com/python/cpython/commit/a4d20b2e5ece2120f129cb4dda951a6c2461e92d

Victor

From aixtools at felt.demon.nl  Thu Aug  2 06:31:42 2018
From: aixtools at felt.demon.nl (Michael)
Date: Thu, 2 Aug 2018 12:31:42 +0200
Subject: [Python-Dev] test_zlib.py suggestion
In-Reply-To: <CAP1=2W4Dwdv3Z8Ln-uj4iJLgP=XcKTmxBrCgZ2zNQR71_=CHSQ@mail.gmail.com>
References: <9cb0f6b1-26c6-a1f0-44e8-cca0358ef06b@felt.demon.nl>
 <CAP1=2W4Dwdv3Z8Ln-uj4iJLgP=XcKTmxBrCgZ2zNQR71_=CHSQ@mail.gmail.com>
Message-ID: <ac99cc70-bd00-3c71-28d0-b9cb80eec05e@felt.demon.nl>

On 01/08/2018 18:03, Brett Cannon wrote:
> Open an issue as this will surely get forgotten otherwise, then people can
> discuss on the issue how to handle this.
Will do. I have more homework - as I just tested AIX 7.1 TL5 as a build 
system. The good news is libz.a is finally updated to something newer. 
The bad news is they introduced libreadline-6, with no support, and that 
it seems it going to be difficult to get it to link.

I so wish they would include "include files" with these standard 
libraries. Not doing so is one of the reasons things break - either 
double files, overwriting files and/or mixed versions. Sigh.
> On Wed, 1 Aug 2018 at 08:40 Michael <aixtools at felt.demon.nl> wrote:
>
>> I have a build_bot running (yeah me!), and was surprised to see
>> test_zlib fail on AIX.
>>
>> There is not an issue with test_zlib, but I do have a suggestion.
>>
>> I was getting an error with test_flushes(). On python2-2.7.15 the test
>> passes and in python3-3.8 (and earlier I expect) the test fails. The
>> difference is the addition of the two modes 'Z_PARTIAL_FLUSH' and 'Z_BLOCK'
>>
>> sync_opt = ['Z_NO_FLUSH', 'Z_SYNC_FLUSH', 'Z_FULL_FLUSH',
>>               'Z_PARTIAL_FLUSH', 'Z_BLOCK']
>>
>> On default AIX installs zlib is, sadly, still at version 1.2.3 and
>> Z_BLOCK was 'improved' in version 1.2.5. And it fails on Z_BLOCK.
>>
>> My suggestion is to enhance the test_library_version() so that it
>> verifies that the minor (or smaller?) number is at least '5'
>>
>> When I static link python with zlib-1.2.11 then the test_zlib passes all
>> sub-tests.
>>
>> Again, not a python bug - but a suggestion for improving what is tested.
>>
>> I can open an issue, and with a bit of assistance/interest from others
>> I'll even do a PR.
>>
>> Michael
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>


From larry at hastings.org  Thu Aug  2 10:00:11 2018
From: larry at hastings.org (Larry Hastings)
Date: Thu, 2 Aug 2018 07:00:11 -0700
Subject: [Python-Dev] [RELEASED] Python 3.4.9 and Python 3.5.6 are now
 available
Message-ID: <c197b678-e6c1-0a54-b3ea-3454cceee987@hastings.org>


On behalf of the Python development community, I'm happy to announce the 
availability of Python 3.4.9 and Python 3.5.6.

Both Python 3.4 and 3.5 are in "security fixes only" mode.? Both 
versions only accept security fixes, not conventional bug fixes, and 
both releases are source-only.


You can find Python 3.4.9 here:

    https://www.python.org/downloads/release/python-349/


And you can find Python 3.5.6 here:

    https://www.python.org/downloads/release/python-356/


We now return you to your pitched debate already in progress,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180802/e20cceb9/attachment.html>

From vstinner at redhat.com  Thu Aug  2 10:17:53 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 2 Aug 2018 16:17:53 +0200
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and
 Python 3.5.6 are now available
In-Reply-To: <c197b678-e6c1-0a54-b3ea-3454cceee987@hastings.org>
References: <c197b678-e6c1-0a54-b3ea-3454cceee987@hastings.org>
Message-ID: <CA+3bQGHiS=idQusNtEiBrZ9BFpL7ANvXDjgBEdAJRH_GuC7TpQ@mail.gmail.com>

Hi,

2018-08-02 16:00 GMT+02:00 Larry Hastings <larry at hastings.org>:
> On behalf of the Python development community, I'm happy to announce the
> availability of Python 3.4.9 and Python 3.5.6.

Great! FYI these versions fix two security vulnerabilities:

(*) CVE-2018-1000117: Buffer overflow vulnerability in os.symlink on Windows
http://python-security.readthedocs.io/vuln/cve-2018-1000117_buffer_overflow_vulnerability_in_os.symlink_on_windows.html

(*) CVE-2018-1060: difflib and poplib catastrophic backtracking
http://python-security.readthedocs.io/vuln/cve-2018-1060_difflib_and_poplib_catastrophic_backtracking.html

3.4.9 and 3.5.6 have no more known security vulnerabilities :-)

Victor

From ericsnowcurrently at gmail.com  Thu Aug  2 11:17:49 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 2 Aug 2018 09:17:49 -0600
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CA+3bQGGiQFMkqz_YRommw4OUMKLyySPgGb1B4etB8smptTt_sg@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
 <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
 <CA+3bQGGiQFMkqz_YRommw4OUMKLyySPgGb1B4etB8smptTt_sg@mail.gmail.com>
Message-ID: <CALFfu7D287SEuYPBGgS4c6Kb_0=KyEzDSy9Oqf7_uBoio43gPw@mail.gmail.com>

On Thu, Aug 2, 2018 at 3:50 AM Victor Stinner <vstinner at redhat.com> wrote:
> I'm still doing further bug fixes and cleanup in the master branch:
> https://bugs.python.org/issue34170
>
> I'm doing more and more changes.

Yeah, it's a question of what you plan to backport.  As Barry
suggested, it would be great if you had a WIP PR for the backport,
just so Ned (and others) has a point of reference.

> I just added two new files:
> Include/coreconfig.h and Python/coreconfig.h. IMHO it's better to put
> similar code in separated files.
>
> FYI Python/coreconfig.c has 954 files and Include/coreconfig.h has 319 lines.

Nick might have a better opinion, particularly when in comes to a C
codebase, but I'm in favor of keeping separate things separate, and
especially when in relates to the runtime.  (Historically we haven't
been great about considering the runtime as a distinct part of
CPython.)  Hence +1 to keeping the runtime config separate, especially
given the size of the files. :)  Presumably pystate.c (and pystate.h)
got smaller by roughly the same amount?

Also, would it make sense for at least some of coreconfig.h to live in
Include/internal, either as coreconfig.h or in the internal pystate.h?

> I'm ok to rename the structure and the files if you prefer a different name.

I'd love to see all the "core" names changed to "runtime", in the same
way that PEP 432 was updated.  It was a point of confusion in the PEP
until we changed the name, and doing so helped.  I thought we had also
changed the code, but apparently not.

For that matter, I'd love to see PEP 432 and the codebase synced up.
While the overall plan is still consistent, a number of details (e.g.
the intent and content of the "core" config) have diverged.

> About that, I'm working on a subproject: abandon Py_xxx global
> configuration variables to replace them with accessing
> interpreter->core_config->xxx.

+1  IMHO it's a natural consequence of having a runtime/core config.
In fact, having both is problematic because it's easy for us to
accidentally ignore a global-var-as-config (prior to Py_Initialize);
the relationship between those global variables and runtime
initialization isn't very obvious in the code (unlike the config
struct).  It's also confusing for embedders if we have both.  As I've
already expressed, I'm definitely in favor of improving encapsulation
of the runtime (and moving away from globals). :)

Note that there are backward compatibility issues to deal with.  AFAIU
if we start ignoring those global variables during initialization then
it's going to cause problems for embedders.  If we get rid of the
variables altogether then it would break extension modules that
currently rely on reading those parts of the runtime config.  So I'm
guessing you planned on deprecating any use of those global variables
and, in the spirit of your goals for the C-API, provide a public API
for extensions to access the info in the runtime config instead.

FWIW, I recall that Nick and I discussed this relative to PEP 432 a
while ago and remember the decision to stay with the status quo for
now (to avoid scope creep in the PEP).  Apparently that consideration
did not get recorded in the PEP (that I could see with a quick skim of
the text).  The mailing lists may have the discussion somewhere.

> I'm not sure yet if it's a good idea or
> not, but it would allow to have two interpreters to have their own
> different configuration. Imagine two interpreters with different
> sys.path running in isolated mode. Or maybe an interpreter without
> importlib?

Yeah, a number of interesting possibilities open up as we further
encapsulate the runtime and move away from globals.

> One of the issue is that we have now two copies of the same option.
> For example, Py_BytesWarningFlag and
> interpreter->core_config->bytes_warning. That's why I would like to
> switch to core_config.

+1

> But I'm also trying to make sure that the two variables have the same value:
> https://github.com/python/cpython/commit/a4d20b2e5ece2120f129cb4dda951a6c2461e92d

Yep.  That is necessary while the global config variables still exist.
It's just risky since it's easy for us to change the config but forget
to update the global config vars (that are shadowing the config).  It
would probably be worth making sure we have tests that verify that the
two stay synchronized.

-eric

From vstinner at redhat.com  Thu Aug  2 11:54:00 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 2 Aug 2018 17:54:00 +0200
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CALFfu7D287SEuYPBGgS4c6Kb_0=KyEzDSy9Oqf7_uBoio43gPw@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
 <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
 <CA+3bQGGiQFMkqz_YRommw4OUMKLyySPgGb1B4etB8smptTt_sg@mail.gmail.com>
 <CALFfu7D287SEuYPBGgS4c6Kb_0=KyEzDSy9Oqf7_uBoio43gPw@mail.gmail.com>
Message-ID: <CA+3bQGH+oVEUHLOufFfsMyfo4dAEepC4-5emu691Dttq_-OSPQ@mail.gmail.com>

2018-08-02 17:17 GMT+02:00 Eric Snow <ericsnowcurrently at gmail.com>:
> Note that there are backward compatibility issues to deal with.  AFAIU
> if we start ignoring those global variables during initialization then
> it's going to cause problems for embedders.

One of the first operation of Py_Initialize(), Py_Main() and
_PyCoreConfig_Read() is to get the current value of all global
configuration variables.

The change is more that modifying a global configuration variable
after Py_Initialize() or Py_Main() may or may not have an effect. And
in the future, it should no longer have effect. In short, these
variables should only be read to populate the initialization
configuration and then it should no longer change.

> So I'm guessing you planned on deprecating any use of those global variables
> and, in the spirit of your goals for the C-API, provide a public API
> for extensions to access the info in the runtime config instead.

There is no *need* to deprecate anything. _PyCoreConfig remains fully
compatible with them and there are now unit tests to make sure that
their value is read at Python startup.

The priority is: core config > global vars > env vars.

>> But I'm also trying to make sure that the two variables have the same value:
>> https://github.com/python/cpython/commit/a4d20b2e5ece2120f129cb4dda951a6c2461e92d
>
> Yep.  That is necessary while the global config variables still exist.
> It's just risky since it's easy for us to change the config but forget
> to update the global config vars (that are shadowing the config).  It
> would probably be worth making sure we have tests that verify that the
> two stay synchronized.

If possible, I would prefer that the configuration is *not* modified
after Python has been initialized. I even hesitate to mark
PyInterpreterState.core_config a constant to prevent such change.

The idea would be to know exactly how Python has been initialize, to
make the initialization more deterministic and explicit.

To come back to a concrete example:
https://github.com/python/cpython/commit/a4d20b2e5ece2120f129cb4dda951a6c2461e92d

We can easily modify core_config->inspect before Python
initialization. For this commit, it's just that I wanted to make tiny
and incremental changes.

Victor

From vstinner at redhat.com  Thu Aug  2 11:59:18 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 2 Aug 2018 17:59:18 +0200
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
 <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
Message-ID: <CA+3bQGFwoNU16CBaKt0h8eJ7QUmBSykxZ17P-a2JTtcrTx6APA@mail.gmail.com>

2018-08-02 1:18 GMT+02:00 Eric Snow <ericsnowcurrently at gmail.com>:
> The "core" config is basically the config for the runtime.  In fact,
> PEP 432 renamed "core" to "runtime".  Please keep the firm distinction
> between the runtime and the (main) interpreter.

There is already something called _PyRuntime but it's shared between
all interpreters.

_PyCoreConfig is already *per* interpreter.

Would you mind to elaborate what you mean by the "main interpreter"? I
don't see anything obvious in the current code about what is a "main
interpreter". Technically, I don't see anything like that.

I'm still not convinced that we need _PyMainInterpreterConfig:
_PyCoreConfig contains the same information. Is it really worth it to
duplicate all _PyCoreConfig (more than 36 fields) in
_PyMainInterpreterConfig? _PyMainInterpreterConfig adds a third copy
of many paramters: another opportunity to introduce an inconsistency.
Right now, an interpreter contains both: core and main
configurations...

I propose to *remove* _PyMainInterpreterConfig and rename
_PyCoreConfig as _PyInterpreterConfig. I would also propose to merge
again Py_Initialize() to have a single step instead of the current
core step + main step: 2 steps.

Victor

From chris.barker at noaa.gov  Thu Aug  2 13:50:58 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 2 Aug 2018 10:50:58 -0700
Subject: [Python-Dev] Error message for wrong number of arguments
In-Reply-To: <CAMSv6X1W7D9NROSKr96Dob-eDm1LRP4S+FZQao3uXOAFsq5QKA@mail.gmail.com>
References: <5B5F00A6.8070802@UGent.be>
 <e64de23cbbfd4ca79b545e7b822167b1@xmail101.UGent.be>
 <5B5F5B6E.1020300@UGent.be>
 <CALGmxE+gw4ApvLQFRoVFgqYF1THEsajbUx0uFoQdZhF8=PH+8g@mail.gmail.com>
 <CAMSv6X1W7D9NROSKr96Dob-eDm1LRP4S+FZQao3uXOAFsq5QKA@mail.gmail.com>
Message-ID: <CALGmxELHynsnAdvO_OkEQACUaBxvUQgoaw3Tt7OZeVNiUME2WQ@mail.gmail.com>

On Wed, Aug 1, 2018 at 2:40 PM, Armin Rigo <armin.rigo at gmail.com> wrote:

> On 30 July 2018 at 22:19, Chris Barker via Python-Dev
> <python-dev at python.org> wrote:
> > Oh well. This is a serious usability issue -- but what can you do?
>
> I think that argument clinic knows if the built-in function is
> supposed to be a method or a function.  It doesn't look too hard to
> add a new flag METH_IS_METHOD or something, which would be taken in
> consideration in the common cases, and which can be added manually OR
> used automatically by argument clinic.  This would not be a 100%
> solution out of the box, but if the new wording is right, it shouldn't
> be a problem.
>

But can you do the same thing with pure-python methods?

After all, this thread started with trying ot unify what error folks get
regardless of how the method was written.


-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180802/c39ff3ee/attachment.html>

From larry at hastings.org  Thu Aug  2 21:22:24 2018
From: larry at hastings.org (Larry Hastings)
Date: Thu, 2 Aug 2018 18:22:24 -0700
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and
 Python 3.5.6 are now available
In-Reply-To: <CA+3bQGHiS=idQusNtEiBrZ9BFpL7ANvXDjgBEdAJRH_GuC7TpQ@mail.gmail.com>
References: <c197b678-e6c1-0a54-b3ea-3454cceee987@hastings.org>
 <CA+3bQGHiS=idQusNtEiBrZ9BFpL7ANvXDjgBEdAJRH_GuC7TpQ@mail.gmail.com>
Message-ID: <26fd54bd-602d-8870-5f5c-094b58a1fb2c@hastings.org>



On 08/02/2018 07:17 AM, Victor Stinner wrote:
> 3.4.9 and 3.5.6 have no more known security vulnerabilities :-)

Well, not to be a complete pill, but...

    https://bugs.python.org/issue17180
    https://bugs.python.org/issue17239
    https://bugs.python.org/issue19050

Sadly, just because they're languishing on bpo doesn't mean they aren't 
valid security vulnerabilities.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180802/b1245b49/attachment.html>

From larry at hastings.org  Fri Aug  3 08:08:07 2018
From: larry at hastings.org (Larry Hastings)
Date: Fri, 3 Aug 2018 05:08:07 -0700
Subject: [Python-Dev] Error message for wrong number of arguments
In-Reply-To: <CAMSv6X1W7D9NROSKr96Dob-eDm1LRP4S+FZQao3uXOAFsq5QKA@mail.gmail.com>
References: <5B5F00A6.8070802@UGent.be>
 <e64de23cbbfd4ca79b545e7b822167b1@xmail101.UGent.be>
 <5B5F5B6E.1020300@UGent.be>
 <CALGmxE+gw4ApvLQFRoVFgqYF1THEsajbUx0uFoQdZhF8=PH+8g@mail.gmail.com>
 <CAMSv6X1W7D9NROSKr96Dob-eDm1LRP4S+FZQao3uXOAFsq5QKA@mail.gmail.com>
Message-ID: <f87c8410-cbb5-2602-6c64-5df7de7d50bc@hastings.org>



On 08/01/2018 02:40 PM, Armin Rigo wrote:
> I think that argument clinic knows if the built-in function is
> supposed to be a method or a function.

Yes, Argument Clinic knows.? Clinic's "Function" instances have a "cls" 
member, and if that's set to a Clinic "Class" instance--and it's not one 
of the special methods like new or init--then it's a normal method.
> It doesn't look too hard to
> add a new flag METH_IS_METHOD or something, which would be taken in
> consideration in the common cases, and which can be added manually OR
> used automatically by argument clinic.

Yes, Clinic could easily automatically generate such a flag.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180803/4c1216c8/attachment.html>

From J.Demeyer at UGent.be  Fri Aug  3 09:32:40 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 3 Aug 2018 15:32:40 +0200
Subject: [Python-Dev] Error message for wrong number of arguments
In-Reply-To: <256b538ca607434ba56b50aeff886bc5@xmail101.UGent.be>
References: <5B5F00A6.8070802@UGent.be>
 <256b538ca607434ba56b50aeff886bc5@xmail101.UGent.be>
Message-ID: <5B645978.8040008@UGent.be>

Actually, I just realized that it's not really possible to fix the error 
messages for built-in methods. The problem is that Argument Clinic does 
not know whether a function or method is being handled. For example, 
there is no indication at all that this is a method (note that the name 
"list.insert" might refer to a function "insert" inside a module called 
"list" or a method "insert" or a class "list"):

/*[clinic input]
list.insert

     index: Py_ssize_t
     object: object
     /

Insert object before index.
[clinic start generated code]*/

From J.Demeyer at UGent.be  Fri Aug  3 09:35:46 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 3 Aug 2018 15:35:46 +0200
Subject: [Python-Dev] Error message for wrong number of arguments
In-Reply-To: <5B645978.8040008@UGent.be>
References: <5B5F00A6.8070802@UGent.be>
 <256b538ca607434ba56b50aeff886bc5@xmail101.UGent.be>
 <5B645978.8040008@UGent.be>
Message-ID: <5B645A32.90409@UGent.be>

Actually, scratch that, I posted too soon.

There is also a block

/*[clinic input]
class list "PyListObject *" "&PyList_Type"
[clinic start generated code]*/

So it could work.

From status at bugs.python.org  Fri Aug  3 12:10:04 2018
From: status at bugs.python.org (Python tracker)
Date: Fri,  3 Aug 2018 18:10:04 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20180803161004.0A0DA57B74@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2018-07-27 - 2018-08-03)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6759 (+19)
  closed 39323 (+63)
  total  46082 (+82)

Open issues with patches: 2692 


Issues opened (60)
==================

#34097: ZIP does not support timestamps before 1980
https://bugs.python.org/issue34097  reopened by vstinner

#34177: test_site fails in macOS-PR VSTS builds for 3.7 branch
https://bugs.python.org/issue34177  reopened by pablogsal

#34252: Bunch of path leaks on Python 3.7 on Release
https://bugs.python.org/issue34252  opened by illera88

#34253: Tkinter- On windows, calling filedialog or messagebox before t
https://bugs.python.org/issue34253  opened by nyt

#34254: Include type annotations in error messages for better errors
https://bugs.python.org/issue34254  opened by xtreak

#34255: test_embed skipped when srcdir != builddir
https://bugs.python.org/issue34255  opened by ronaldoussoren

#34256: Python treats ASCII record seperator ('\x1e') as a newline
https://bugs.python.org/issue34256  opened by timClicks

#34259: Improve docstring of list.sort
https://bugs.python.org/issue34259  opened by timhoffm

#34260: shutil.copy2 is not the same as cp -p
https://bugs.python.org/issue34260  opened by csernazs

#34261: Add description to clinic.py
https://bugs.python.org/issue34261  opened by timhoffm

#34262: Asyncio test fails under Win 7
https://bugs.python.org/issue34262  opened by pansen

#34264: Inconsistency between stack size in main thread and secondary 
https://bugs.python.org/issue34264  opened by ronaldoussoren

#34266: Bad behavior with "restart \" or "restart "" in pdb
https://bugs.python.org/issue34266  opened by ppperry

#34267: find_python.bat doesn't find installed Python 3.7
https://bugs.python.org/issue34267  opened by Segev Finer

#34268: run pool.submit in callback when future.set_result
https://bugs.python.org/issue34268  opened by gaoxinge

#34270: Add names to asyncio tasks
https://bugs.python.org/issue34270  opened by alex.gronholm

#34271: Please support logging of SSL master secret by env variable SS
https://bugs.python.org/issue34271  opened by jmfrank63

#34272: Reorganize C API tests
https://bugs.python.org/issue34272  opened by serhiy.storchaka

#34273: %f is confusingly associated with fixed point format
https://bugs.python.org/issue34273  opened by MikeFoxtrot

#34274: Python launcher behavior with "#!/usr/bin/env python" shebang
https://bugs.python.org/issue34274  opened by Segev Finer

#34276: urllib.parse doesn't round-trip file URI's with multiple leadi
https://bugs.python.org/issue34276  opened by chris.jerdonek

#34278: Documentation: os.waitid and os.WNOHANG
https://bugs.python.org/issue34278  opened by crzbear

#34279: RFC: issue a warning in regrtest when no tests have been execu
https://bugs.python.org/issue34279  opened by vstinner

#34282: Enum._convert shadows members named _convert
https://bugs.python.org/issue34282  opened by orlnub123

#34283: [Windows] Python 2 mishandles console code page after setlocal
https://bugs.python.org/issue34283  opened by Segev Finer

#34284: Nonsensical exception message when calling `__new__` on non-in
https://bugs.python.org/issue34284  opened by ppperry

#34285: regrtest: in case of test failure, add "always look on the bri
https://bugs.python.org/issue34285  opened by vstinner

#34286: lib2to3 tests fail on the 3.7 branch (used to work with 3.7.0)
https://bugs.python.org/issue34286  opened by doko

#34288: Declare sethostname in socketmodule.c for SOLARIS
https://bugs.python.org/issue34288  opened by rbelio

#34290: _ctypes PyCField_new doesn't do anything
https://bugs.python.org/issue34290  opened by bup

#34292: test_compile hangs in AMD Ubuntu buildbots
https://bugs.python.org/issue34292  opened by pablogsal

#34293: DOC: Makefile inherits a Sphinx 1.5 bug regarding PAPER envvar
https://bugs.python.org/issue34293  opened by jfbu

#34294: re.finditer and lookahead bug
https://bugs.python.org/issue34294  opened by beardypig

#34296: Speed up python startup by pre-warming the vm
https://bugs.python.org/issue34296  opened by cykerway

#34297: Windows py.exe launcher fail to handle quote correctly
https://bugs.python.org/issue34297  opened by copelnug

#34300: gcc 7.3 causes a warning when compiling getpath.c in python 2.
https://bugs.python.org/issue34300  opened by tzickel

#34302: Avoid inefficient way to find start point in deque.index
https://bugs.python.org/issue34302  opened by ksg97031

#34303: micro-optimizations in functools.reduce()
https://bugs.python.org/issue34303  opened by sir-sigurd

#34305: inspect.getsourcefile and inspect.getcomments do not work with
https://bugs.python.org/issue34305  opened by Eric.Wieser

#34306: minidom: wrong processing of xmlns attributes
https://bugs.python.org/issue34306  opened by porton

#34308: shutil.copystat fails in dockered Linux (Debian) on Azure (Win
https://bugs.python.org/issue34308  opened by nuarhu

#34309: Trouble when reloading extension modules.
https://bugs.python.org/issue34309  opened by christoph.wiedemann

#34310: Build error with option "--with-pydebug" on Mac OS 10.13.6
https://bugs.python.org/issue34310  opened by yuwu

#34311: locale.format() and locale.format_string() cast Decimals to fl
https://bugs.python.org/issue34311  opened by jemerton

#34312: Allow str.endswith and str.startswith to accept an iterable
https://bugs.python.org/issue34312  opened by brett.cannon

#34313: IDLE crashes with Tk-related error on macOS with ActiveTcl 8.6
https://bugs.python.org/issue34313  opened by taleinat

#34315: Regex not evalauated correctly
https://bugs.python.org/issue34315  opened by ram

#34316: test_socket and test_asyncio timeouts in AMD64 Windows10 3.x b
https://bugs.python.org/issue34316  opened by pablogsal

#34318: Convert deprecated behavior of assertRaises() etc into errors
https://bugs.python.org/issue34318  opened by serhiy.storchaka

#34319: Clarify pathlib.Path("filepath").read_text()
https://bugs.python.org/issue34319  opened by thomas.nyberg

#34320: Creating dict from OrderedDict doesn't preserve order
https://bugs.python.org/issue34320  opened by serhiy.storchaka

#34321: mmap.mmap() should not necessarily clone the file descriptor
https://bugs.python.org/issue34321  opened by manuels

#34322: modification to Lib/distutils/ccompiler.py to simplify handlin
https://bugs.python.org/issue34322  opened by schnaitterm

#34323: False timeout log message on proactor close
https://bugs.python.org/issue34323  opened by johnboy2

#34324: Doc README wrong directory name for venv
https://bugs.python.org/issue34324  opened by lys.nikolaou

#34325: test_zipfile gets OverflowError in multiple buildbots
https://bugs.python.org/issue34325  opened by pablogsal

#34326: test_subprocess.POSIXProcessTestCase fails in AMD64 Ubuntu 3.x
https://bugs.python.org/issue34326  opened by pablogsal

#34329: Document how to remove a suffix with pathlib.Path.with_suffix
https://bugs.python.org/issue34329  opened by sotte

#34330: The decode_header function does not always work correctly.
https://bugs.python.org/issue34330  opened by ???????? ??2

#34331: Incorrectly pluralized abstract class error message
https://bugs.python.org/issue34331  opened by ppperry



Most recent 15 issues with no replies (15)
==========================================

#34331: Incorrectly pluralized abstract class error message
https://bugs.python.org/issue34331

#34330: The decode_header function does not always work correctly.
https://bugs.python.org/issue34330

#34322: modification to Lib/distutils/ccompiler.py to simplify handlin
https://bugs.python.org/issue34322

#34321: mmap.mmap() should not necessarily clone the file descriptor
https://bugs.python.org/issue34321

#34320: Creating dict from OrderedDict doesn't preserve order
https://bugs.python.org/issue34320

#34318: Convert deprecated behavior of assertRaises() etc into errors
https://bugs.python.org/issue34318

#34315: Regex not evalauated correctly
https://bugs.python.org/issue34315

#34310: Build error with option "--with-pydebug" on Mac OS 10.13.6
https://bugs.python.org/issue34310

#34308: shutil.copystat fails in dockered Linux (Debian) on Azure (Win
https://bugs.python.org/issue34308

#34306: minidom: wrong processing of xmlns attributes
https://bugs.python.org/issue34306

#34305: inspect.getsourcefile and inspect.getcomments do not work with
https://bugs.python.org/issue34305

#34303: micro-optimizations in functools.reduce()
https://bugs.python.org/issue34303

#34297: Windows py.exe launcher fail to handle quote correctly
https://bugs.python.org/issue34297

#34296: Speed up python startup by pre-warming the vm
https://bugs.python.org/issue34296

#34290: _ctypes PyCField_new doesn't do anything
https://bugs.python.org/issue34290



Most recent 15 issues waiting for review (15)
=============================================

#34329: Document how to remove a suffix with pathlib.Path.with_suffix
https://bugs.python.org/issue34329

#34325: test_zipfile gets OverflowError in multiple buildbots
https://bugs.python.org/issue34325

#34324: Doc README wrong directory name for venv
https://bugs.python.org/issue34324

#34322: modification to Lib/distutils/ccompiler.py to simplify handlin
https://bugs.python.org/issue34322

#34320: Creating dict from OrderedDict doesn't preserve order
https://bugs.python.org/issue34320

#34319: Clarify pathlib.Path("filepath").read_text()
https://bugs.python.org/issue34319

#34318: Convert deprecated behavior of assertRaises() etc into errors
https://bugs.python.org/issue34318

#34305: inspect.getsourcefile and inspect.getcomments do not work with
https://bugs.python.org/issue34305

#34303: micro-optimizations in functools.reduce()
https://bugs.python.org/issue34303

#34302: Avoid inefficient way to find start point in deque.index
https://bugs.python.org/issue34302

#34293: DOC: Makefile inherits a Sphinx 1.5 bug regarding PAPER envvar
https://bugs.python.org/issue34293

#34284: Nonsensical exception message when calling `__new__` on non-in
https://bugs.python.org/issue34284

#34282: Enum._convert shadows members named _convert
https://bugs.python.org/issue34282

#34272: Reorganize C API tests
https://bugs.python.org/issue34272

#34270: Add names to asyncio tasks
https://bugs.python.org/issue34270



Top 10 most discussed issues (10)
=================================

#34247: PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher
https://bugs.python.org/issue34247  11 msgs

#34309: Trouble when reloading extension modules.
https://bugs.python.org/issue34309  11 msgs

#33083: math.factorial accepts non-integral Decimal instances
https://bugs.python.org/issue33083  10 msgs

#34326: test_subprocess.POSIXProcessTestCase fails in AMD64 Ubuntu 3.x
https://bugs.python.org/issue34326  10 msgs

#34284: Nonsensical exception message when calling `__new__` on non-in
https://bugs.python.org/issue34284   9 msgs

#24255: Replace debuglevel-related logic with logging
https://bugs.python.org/issue24255   8 msgs

#34170: Py_Initialize(): computing path configuration must not have si
https://bugs.python.org/issue34170   8 msgs

#33695: Have shutil.copytree(), copy() and copystat() use cached scand
https://bugs.python.org/issue33695   7 msgs

#33729: Hashlib/blake2* missing 'data' keyword argument
https://bugs.python.org/issue33729   7 msgs

#34276: urllib.parse doesn't round-trip file URI's with multiple leadi
https://bugs.python.org/issue34276   7 msgs



Issues closed (61)
==================

#5978: cProfile and profile don't work with pygtk/pyqt and sys.exit(0
https://bugs.python.org/issue5978  closed by steve.dower

#8145: Documentation about sqlite3 isolation_level
https://bugs.python.org/issue8145  closed by berker.peksag

#11594: 2to3 does not preserve line endings
https://bugs.python.org/issue11594  closed by jason.coombs

#14266: pyunit script as shorthand for python -m unittest
https://bugs.python.org/issue14266  closed by berker.peksag

#23642: Interaction of ModuleSpec and C Extension Modules
https://bugs.python.org/issue23642  closed by petr.viktorin

#24356: venv documentation incorrect / misleading
https://bugs.python.org/issue24356  closed by steve.dower

#24809: Add getprotobynumber to socket module
https://bugs.python.org/issue24809  closed by vstinner

#27163: IDLE entry for What's New in Python 3.6
https://bugs.python.org/issue27163  closed by taleinat

#27201: expose the ABI name as a config variable
https://bugs.python.org/issue27201  closed by doko

#27419: Bugs in PyImport_ImportModuleLevelObject
https://bugs.python.org/issue27419  closed by serhiy.storchaka

#27671: FAQ: len() is still function for good reason.
https://bugs.python.org/issue27671  closed by inada.naoki

#27910: Doc/library/traceback.rst ??? references to tuples should be r
https://bugs.python.org/issue27910  closed by berker.peksag

#29097: [Windows] datetime.fromtimestamp(t) when 0 <= t <= 86399 fails
https://bugs.python.org/issue29097  closed by ammar2

#29710: Incorrect representation caveat on bitwise operation docs
https://bugs.python.org/issue29710  closed by ncoghlan

#30984: traceback.print_exc return value documentation
https://bugs.python.org/issue30984  closed by berker.peksag

#32046: 2to3 fix for operator.isCallable()
https://bugs.python.org/issue32046  closed by berker.peksag

#32545: Unable to install Python 3.7.0a4 on Windows 10 - Error 0x80070
https://bugs.python.org/issue32545  closed by terry.reedy

#32769: Add 'annotations' to the glossary
https://bugs.python.org/issue32769  closed by steve.dower

#33089: Add multi-dimensional Euclidean distance function to the math 
https://bugs.python.org/issue33089  closed by rhettinger

#33113: Query performance is very low and can even lead to denial of s
https://bugs.python.org/issue33113  closed by serhiy.storchaka

#33204: IDLE: remove \b from colorizer string prefix
https://bugs.python.org/issue33204  closed by terry.reedy

#33435: incorrect detection of information of some distributions
https://bugs.python.org/issue33435  closed by petr.viktorin

#33476: String index out of range in get_group(), email/_header_value_
https://bugs.python.org/issue33476  closed by steve.dower

#33566: re.findall() dead locked whent the expected ending char not oc
https://bugs.python.org/issue33566  closed by tim.peters

#33833: ProactorEventLoop raises AssertionError
https://bugs.python.org/issue33833  closed by asvetlov

#33921: Explain that '' can be used to bind to all interfaces for the 
https://bugs.python.org/issue33921  closed by steve.dower

#34035: Several AttributeError in zipfile seek() methods
https://bugs.python.org/issue34035  closed by serhiy.storchaka

#34075: asyncio: We should prohibit setting a ProcessPoolExecutor in w
https://bugs.python.org/issue34075  closed by vstinner

#34086: logging.Handler.handleError regressed in python3
https://bugs.python.org/issue34086  closed by vinay.sajip

#34113: LLTRACE segv
https://bugs.python.org/issue34113  closed by vstinner

#34120: IDLE: Freeze when closing Settings (& About) dialog on MacOS
https://bugs.python.org/issue34120  closed by taleinat

#34182: Lib/test/test_pydoc.py failed when ran as a script
https://bugs.python.org/issue34182  closed by serhiy.storchaka

#34231: PYTHONBREAKPOINT is not documented with python --help
https://bugs.python.org/issue34231  closed by berker.peksag

#34234: Use _PyAnyInt_Check() and _PyAnyInt_CheckExact() in 2.7
https://bugs.python.org/issue34234  closed by serhiy.storchaka

#34239: Convert test_bz2 to use tempfile
https://bugs.python.org/issue34239  closed by tim.golden

#34250: import xmlrpclib not works in file named "xml.py"
https://bugs.python.org/issue34250  closed by berker.peksag

#34251: MSI build fails
https://bugs.python.org/issue34251  closed by berker.peksag

#34257: SSL should accept cert content, instead of just cert file path
https://bugs.python.org/issue34257  closed by njs

#34258: Python shell keeps restarting
https://bugs.python.org/issue34258  closed by terry.reedy

#34263: asyncio: "relative *delay* or absolute *when* should not excee
https://bugs.python.org/issue34263  closed by yselivanov

#34265: Dataclasses test doesn't run independently
https://bugs.python.org/issue34265  closed by kayhayen

#34269: logging in 3.7 behaves different due to caching
https://bugs.python.org/issue34269  closed by vinay.sajip

#34275: IDLE: no calltips on MacOS with tk 8.6
https://bugs.python.org/issue34275  closed by terry.reedy

#34277: EmailPolicy not followed
https://bugs.python.org/issue34277  closed by bryced

#34280: METH_NOARGS: no longer require that second arg is NULL
https://bugs.python.org/issue34280  closed by jdemeyer

#34281: IDLE: closing "Go to Line" on OS X 3.7 activates wrong window
https://bugs.python.org/issue34281  closed by terry.reedy

#34287: bufferedio.c uses unused argument of METH_NOARGS functions
https://bugs.python.org/issue34287  closed by inada.naoki

#34289: System can't find the path of Python
https://bugs.python.org/issue34289  closed by matrixise

#34291: UnboundLocalError raised on call to global
https://bugs.python.org/issue34291  closed by eric.smith

#34295: Avoid inefficient way to find start point in deque.index
https://bugs.python.org/issue34295  closed by rhettinger

#34298: Avoid inefficient way to find start point in deque.index
https://bugs.python.org/issue34298  closed by hacksg

#34299: argparse description formatting
https://bugs.python.org/issue34299  closed by paul.j3

#34301: Add _PyInterpreterState_Get() helper function
https://bugs.python.org/issue34301  closed by vstinner

#34304: clarification on escaping \d in regular expressions
https://bugs.python.org/issue34304  closed by serhiy.storchaka

#34307: NullHandler init refusing arguments
https://bugs.python.org/issue34307  closed by vinay.sajip

#34314: Like __hash__, allow setting MyClass.__init__ to None to preve
https://bugs.python.org/issue34314  closed by rhettinger

#34317: Improve docstring   Environment variables in Windows NT
https://bugs.python.org/issue34317  closed by Mariatta

#34327: test_subprocess fails
https://bugs.python.org/issue34327  closed by matrixise

#34328: Incorrect URL for the Visual C++ Build Tools
https://bugs.python.org/issue34328  closed by ronaldoussoren

#1467929: %-formatting and dicts
https://bugs.python.org/issue1467929  closed by eric.smith

#1617161: Instance methods compare equal when their self's are equal
https://bugs.python.org/issue1617161  closed by serhiy.storchaka

From ericsnowcurrently at gmail.com  Fri Aug  3 14:23:41 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 3 Aug 2018 12:23:41 -0600
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CA+3bQGFwoNU16CBaKt0h8eJ7QUmBSykxZ17P-a2JTtcrTx6APA@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
 <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
 <CA+3bQGFwoNU16CBaKt0h8eJ7QUmBSykxZ17P-a2JTtcrTx6APA@mail.gmail.com>
Message-ID: <CALFfu7B0UjJiwiH1p_vfnmvCa4Lo7e2Ra4nOoT=QiN_4XODVPg@mail.gmail.com>

Before I dive in, I'll say that I'd really like to hear Nick's opinion
on all this. :)

On Thu, Aug 2, 2018 at 9:59 AM Victor Stinner <vstinner at redhat.com> wrote:
> 2018-08-02 1:18 GMT+02:00 Eric Snow <ericsnowcurrently at gmail.com>:
> > The "core" config is basically the config for the runtime.  In fact,
> > PEP 432 renamed "core" to "runtime".  Please keep the firm distinction
> > between the runtime and the (main) interpreter.
>
> There is already something called _PyRuntime but it's shared between
> all interpreters.

_PyRuntime is a static global of type PyRuntimeState.  It is where I
consolidated (nearly) all the global runtime state last September.

> _PyCoreConfig is already *per* interpreter.

This was done as part of the PEP 432 implementation, which I landed
during PyCon 2017.  If PyRuntimeState had existed already I'm sure it
would be there instead.

> Would you mind to elaborate what you mean by the "main interpreter"? I
> don't see anything obvious in the current code about what is a "main
> interpreter". Technically, I don't see anything like that.

The main interpreter is the first one created (during runtime
initialization).  It is special for a variety of reasons.  Here are
the ones I could think of:

1. the "main" thread will always belong to the main interpreter since
it is the first PyThreadState created
2. runtime initialization uses the main interpreter exclusively
3. the first phase of runtime initialization (pre-initialization) ends
with the main interpreter being *partially* configured
4. during the second phase (initializing), the partially-configured
main interpreter facilitates the use of most of the C-API and may be
used by embedders
  * this is the only time that an interpreter may be used in this way,
and it only happens with the main interpreter
5. runtime finalization takes place using the main interpreter
6. the main interpreter is the last one destroyed during finalization
7. the REPL runs only in the main interpreter
8. the Python CLI is run in the main interpreter (i.e. in its __main__ module)
9. the main interpreter cannot be destroyed (except during finalization)
10. in Python code the main interpreter will always exist
11. it is the parent of all subinterpreters created in Python code (via PEP 554)
12. signals are handled only in the main interpreter
13. all single-threaded Python code is run in the main interpreter

Note that there isn't anything special to the interpreter itself, but
rather in where and how it's used.  However, that matters and the
runtime needs to treat it specially.

I expect all this isn't well-documented because it is relevant to very
few people.

> I'm still not convinced that we need _PyMainInterpreterConfig:

Let's step back a moment and consider the course of events:

1. PEP 432 was created nearly 6 years ago to address the tangle that
runtime initialization had become, with the intent of helping both the
CPython maintainers and embedders
2. Nick did some re-organization around then (e.g. factoring out
pylifecycle.c) to facilitate an implementation of the PEP
3. Nick implemented PEP 432, with a plan to merge it as a *private*
API regardless of whether or not the PEP was accepted (with general
consensus that doing so was a good idea)
  * see https://bitbucket.org/ncoghlan/cpython_sandbox/branch/pep432_modular_bootstrap
  * landing the private API would allow us to iron out the details of the PEP
  * work happened in spurts in 2013, 2015, and 2016; I kept poking
Nick because the implementation was a big blocker for my
multi-core/subinterpreters project
4. leading up to (and at) PyCon 2017, I forked Nick's branch, moved it
to github, rebased it onto master, got it working again, created a PR,
and finally landed it
5. since then the implementation has changed a bunch (due to Victor's
much appreciate efforts) and has diverged from the PEP
  * notably it's unclear that code (especially pymain) strictly
conforms to the phases in the PEP

At this point the PEP is out of date.  There have been several mailing
list threads (all python-dev, IIRC) and some BPO issues where Victor
solicited clarification or expressed a desire to change things and
Nick gave feedback.  None of that made it into the PEP. :(
Consequently the PEP is inconsistent with the actual target.
Furthermore, as was intended, we've learned of a few ways that the PEP
could be improved.

We *really* need to get the PEP updated so we can be sure everyone has
all the info.

Regarding the justification for the "main interpreter" config, the
implementation has diverged from the original intent of the PEP:

* the core/runtime config was meant to hold the minimal data needed to
bootstrap/initialize the basic (limited) functionality of the C-API,
including a restricted main interpreter
  + the struct members were strictly C plain-old-types since using
PyObject would require the runtime to already be (partially)
initialized
  + in the last year a lot of data has been added to this config; I
don't know how much is strictly necessary to bootstrap the runtime
(end of phase 1) and how much could be dealt with in phase 2
* the "main interpreter" config was meant to hold all the config
needed to finish initializing the runtime (end of phase 2)
  + the struct members were mostly PyObject* (possible since most
builtin types are available at this point)
  + the PEP proposes a bunch more fields than the implementation has;
we planned on adding them a few at a time

> _PyCoreConfig contains the same information. Is it really worth it to
> duplicate all _PyCoreConfig (more than 36 fields) in
> _PyMainInterpreterConfig? _PyMainInterpreterConfig adds a third copy
> of many paramters: another opportunity to introduce an inconsistency.

TBH, the PEP *should* have a clear answer for your question here,
Victor.  It has some explanation, but clearly it is incomplete (hence
this continuing email thread).

The duplication is partly a consequence of what has happened in the
last year: a bunch of fields were added to the core config that were
not in the PEP.  However, note the key differences between the two
structs:

* core/runtime config
  + minimal
  + simple C fields
  + meant for embedders/pymain to bootstrap a limited runtime
  + not really meant to be used after calling Py_InitializeRuntime
(AKA Py_InitializeCore)
* main interpreter config
  + includes everything needed to finish full runtime initialization
  + has PyObject* fields
  + meant for embedders/pymain to finish initializing the runtime
  + not really meant to be used after calling
Py_ConfigureMainInterpreter (except when initializing a subintepreter)

Originally there wasn't much overlap.  Furthermore, both of them are
kept around so that, via the C-API (or directly in the CPython impl.),
we could expose what data was used to initialize the runtime.  This
fills much the same role as the existing global Py_* variables.

The duplication is due to there being C and PyObject versions.  It is
for the sake of embedders (and a little bit of sanity).  The big
reason why it shouldn't be a problem is because
PyMainInterpreterConfig is generated directly from PyRuntimeConfig
(AKA PyCoreConfig) and only *after* we've used the runtime config to
bootstrap the limited runtime (after which it shouldn't be modified
ever).  So there's no risk of inconsistency, right?

Perhaps it would make sense to only keep a const copy of both, to
avoid modification?

> Right now, an interpreter contains both: core and main
> configurations...

As noted above, the core/runtime config should probably be on
PyRuntimeState instead.

Regarding the "main" config, PyMainInterpreterConfig probably makes
more sense as one of the following:

1. on PyRuntimeState, like the core/runtime config (since it's a one-off)
2. on PyInterpreterState, like now, but set to NULL on all but the
main interpreter (which would allow us to distinguish the main
interpreter from the rest)

Both would require PyInterpreterConfig from PEP 432, but expanded to
cover all config that might be unique to an interpreter.

Also, conceptually there's a different between
the-config-used-to-finish-runtime-init and the
config-used-to-initialize-an-interpreter (including the main
interpreter).  In fact, PEP 432 does include a PyInterpreterConfig.
However, in the current implementation, PyMainInterpreterConfig fills
that role exclusively, which is confusing since we use the "main
interpreter" config to initialize all interpreters (not just the main
one).

So here's what might make sense to do:

1. rename "core" to "runtime" (to reduce confusion)
2. move PyInterpreterState.runtime_config to PyRuntimeState.config
  + prevent modification after Py_InitializeRuntime() is called (e.g.
keep a const copy)?
3. move PyInterpreterState.config to PyRuntimeConfig.main_config
  + prevent modification after Py_ConfigureMainInterpreter() is called
(e.g. keep a const copy)?
  + keep the PyMainInterpreterConfig and Py_ConfigureMainInterpreter names
4. add PyInterpreterConfig with only the parts of
PyMainInterpreterConfig needed to initialize any interpreter
  + add Py_NewInterpreterEx(PyInterpreterConfig) to allow explicitly
passing a config?
5. add PyInterpreterState.config (type PyInterpreterConfig) to record
the config used to initialize that interpreter
  + prevent modification after the interpreter is initialized (e.g.
keep a const copy)?

> I propose to *remove* _PyMainInterpreterConfig and rename
> _PyCoreConfig as _PyInterpreterConfig. I would also propose to merge
> again Py_Initialize() to have a single step instead of the current
> core step + main step: 2 steps.

So you are not in favor of PEP 432 then. :)

-eric

From J.Demeyer at UGent.be  Fri Aug  3 15:34:22 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 3 Aug 2018 21:34:22 +0200
Subject: [Python-Dev] Error message for wrong number of arguments
In-Reply-To: <256b538ca607434ba56b50aeff886bc5@xmail101.UGent.be>
References: <5B5F00A6.8070802@UGent.be>
 <256b538ca607434ba56b50aeff886bc5@xmail101.UGent.be>
Message-ID: <5B64AE3E.1070003@UGent.be>

On 2018-07-30 17:28, Nick Coghlan wrote:
> I would, and I think it would make sense for the PEP to cite improving
> consistency (and reducing code duplication?) in that regard as an
> advantage of the PEP.

I'm not sure to which PEP you are referring (PEP 580 or a new PEP?). 
After thinking a bit about the issue of error messages, I realized that 
PEP 580 would make this easier to fix (to be clear: there are ways to 
fix it without PEP 580, I'm just saying that PEP 580 makes it easier). 
There are two related reasons for this:

* The existing code which calls the actual underlying C function doesn't 
have access to the Python-level function object. So it can't know 
whether it's a function (where self doesn't count) or a method (where 
self counts).

* Armin Rigo suggested to use a new flag to indicate this difference: 
that would certainly work for Argument Clinic (just have Argument Clinic 
add that flag). For methods defined without Argument Clinic, we cannot 
require such a new flag though. We could still add the flag at runtime, 
but it's far from clear if we can freely change the flags inside a 
PyMethodDef at runtime (at least, no existing code that I know does that).

PEP 580 solves the first issue by having the function object available 
and it solves the second issue by not relying on PyMethodDef at all for 
calling functions/methods. The second issue especially can be 
generalized as: PEP 580 makes the implementation of functions/methods 
much less rigid, making it easier to change the implementation.

So maybe this can be seen as yet another advantage of PEP 580.


Jeroen.

From rosuav at gmail.com  Fri Aug  3 17:15:58 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 4 Aug 2018 07:15:58 +1000
Subject: [Python-Dev] Finding Guido's replacement
In-Reply-To: <CAPTjJmoNSV9xEw7q+ZTSCngpjTVuLPtsG_sXAHMwpLa3BORm1w@mail.gmail.com>
References: <CAPTjJmoNSV9xEw7q+ZTSCngpjTVuLPtsG_sXAHMwpLa3BORm1w@mail.gmail.com>
Message-ID: <CAPTjJmoHc1w8YpLbuC5jgqNC_xtrGQ4gXpUnPJO0r2M3Ld6cWQ@mail.gmail.com>

On Mon, Jul 23, 2018 at 6:12 AM, Chris Angelico <rosuav at gmail.com> wrote:
> Guido's term as Benevolent Dictator For Life has been a long one, but
> in the wake of his resignation, we have an opportunity to correct some
> fundamental flaws in the system. Among them:
>
> * Guido lacks patience, as evidenced by the brevity of his acceptance
> posts. See https://mail.python.org/pipermail/python-dev/2017-December/151038.html
> and https://mail.python.org/pipermail/python-dev/2011-November/114545.html
> and particularly
> https://mail.python.org/pipermail/python-dev/2016-May/144646.html
> where Guido specifically cites his own lack of patience.
>
> * Lately, all Guido's actions have been to benefit his employer, not
> the Common Pythonista. We have proof of this from reliable reporting
> sources such as Twitter and social media.
>
> * Finally, "For Life" is far too long. We need to change our rulers
> periodically.
>
> I propose a new way to appoint a project head. All candidates shall be
> flown to an island owned by the Python Secret Underground (which
> emphatically does NOT exist, but an island that would be owned by it
> if it did), whereupon they parachute down, search for guns, and
> proceed to fight each other until only one is left alive. The survivor
> shall be treated to a chicken dinner and proclaimed Patient,
> Understanding, Benevolent Governor, a title which shall be retained
> for one fortnight, after which we repeat the exercise.
>
> If this plan meets with broad approval, I shall write up PEP 3401, in
> honour of the prior art in PEP 401.

For those who didn't pick up on ANY of the hints: THE ABOVE WAS A
JOKE. Please do not take it seriously. I apologize for assuming that
people would think this funny, which was an error of judgement on my
part. Finding Guido's replacement is being handled seriously
elsewhere, not here, and would not be done in any manner akin to the
above.

ChrisA

From rosuav at gmail.com  Fri Aug  3 17:29:00 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 4 Aug 2018 07:29:00 +1000
Subject: [Python-Dev] Finding Guido's replacement
In-Reply-To: <CAPTjJmoNSV9xEw7q+ZTSCngpjTVuLPtsG_sXAHMwpLa3BORm1w@mail.gmail.com>
References: <CAPTjJmoNSV9xEw7q+ZTSCngpjTVuLPtsG_sXAHMwpLa3BORm1w@mail.gmail.com>
Message-ID: <CAPTjJmoEBfK9Z2Sz4nbqU4a3RCV13PsqPNsmm3=NgkB7yfRCJA@mail.gmail.com>

On Mon, Jul 23, 2018 at 6:12 AM, Chris Angelico <rosuav at gmail.com> wrote:
> Guido's term as Benevolent Dictator For Life has been a long one, but
> in the wake of his resignation, we have an opportunity to correct some
> fundamental flaws in the system. Among them:
>
> * Guido lacks patience, as evidenced by the brevity of his acceptance
> posts. See https://mail.python.org/pipermail/python-dev/2017-December/151038.html
> and https://mail.python.org/pipermail/python-dev/2011-November/114545.html
> and particularly
> https://mail.python.org/pipermail/python-dev/2016-May/144646.html
> where Guido specifically cites his own lack of patience.
>
> * Lately, all Guido's actions have been to benefit his employer, not
> the Common Pythonista. We have proof of this from reliable reporting
> sources such as Twitter and social media.
>
> * Finally, "For Life" is far too long. We need to change our rulers
> periodically.
>
> I propose a new way to appoint a project head. All candidates shall be
> flown to an island owned by the Python Secret Underground (which
> emphatically does NOT exist, but an island that would be owned by it
> if it did), whereupon they parachute down, search for guns, and
> proceed to fight each other until only one is left alive. The survivor
> shall be treated to a chicken dinner and proclaimed Patient,
> Understanding, Benevolent Governor, a title which shall be retained
> for one fortnight, after which we repeat the exercise.
>
> If this plan meets with broad approval, I shall write up PEP 3401, in
> honour of the prior art in PEP 401.

More specifically: It has been brought to my attention that the
content of this post may have been offensive to some. This was NOT my
intention, and for this I apologize. It was a failed attempt to make
people laugh, and I did not want at any time to cause pain. Please
accept my apologies, and the withdrawal of the jest.

ChrisA

From vstinner at redhat.com  Fri Aug  3 18:37:18 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Sat, 4 Aug 2018 00:37:18 +0200
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CALFfu7B0UjJiwiH1p_vfnmvCa4Lo7e2Ra4nOoT=QiN_4XODVPg@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
 <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
 <CA+3bQGFwoNU16CBaKt0h8eJ7QUmBSykxZ17P-a2JTtcrTx6APA@mail.gmail.com>
 <CALFfu7B0UjJiwiH1p_vfnmvCa4Lo7e2Ra4nOoT=QiN_4XODVPg@mail.gmail.com>
Message-ID: <CA+3bQGGoz=Wk9YsVnFXzqjZ=FrDXamOn_LbN2T3LXONSyjmH5Q@mail.gmail.com>

It seems like the PEP 432 proposes an API designed from scratch as the
target API. I started from the 28 years old C code and I tried to
cleanup the code. Our method is different, so it's not surprising that
the result is different :-) My intent is to get:

* a function to read *all* configuration with no side effect and put
it into a single structure: _PyCoreConfig
* modify Py_Main() and all variants of Py_Initialize() to always end
in the same code path using _PyCoreConfig

I'm open to change to move the current implementation closer to the
PEP 432. But it seems like I don't understand well the subtle parts of
this PEP.


> * core/runtime config
>   + minimal

I'm not sure of what you mean by "minimum". I collected *all*
parameters need to initialize Python and there are something like 40
parameters or more. But _PyCoreConfig has enough parameters to
initialize a full Python with a working importlib and a REPL, for
example.

Where do you put the limit for "minimal"?


> * main interpreter config
>   + includes everything needed to finish full runtime initialization

For practical reason, I prefer to be able to pass the "path
configuration" at the C level to be able to initialize importlib.

IMHO it makes the current code base simpler, since the path
computation is fully implemented in C.
For example, it allows embedders to use a fixed sys.path in C. IMHO a
good example is to imagine a Python runtime with *no* filesystem
access, where everything is built into the binary. So we have to skip
completly the code computing the path configuration, since this
operating access the filesystem as well!

Maybe something should be changed here?


> The duplication is due to there being C and PyObject versions.  It is
> for the sake of embedders (and a little bit of sanity).  The big
> reason why it shouldn't be a problem is because
> PyMainInterpreterConfig is generated directly from PyRuntimeConfig
> (AKA PyCoreConfig) and only *after* we've used the runtime config to
> bootstrap the limited runtime (after which it shouldn't be modified
> ever).  So there's no risk of inconsistency, right?

Currently, core_config and main_config can be modified, as global
variables: some parameters exist in 3 versions, each is modified. And
it's unclear which one has the highest priority. For example, if we
decide to always rely on core_config, we have to modify the C code to
not longer access Py_VerboseFlag after Py_Initialize().

I'm talking about the current C code, not your theorical API.


> Perhaps it would make sense to only keep a const copy of both, to
> avoid modification?

Maybe. But currently, some flags are modified after Py_Initialize(),
especially Py_InspectFlag in main.c.

I would prefer to keep a read-only configuration to reflect what
sys.flags contains and know how Python has been initialized.


> As noted above, the core/runtime config should probably be on
> PyRuntimeState instead.

For me it doesn't make sense to put all _PyCoreConfig parameters into
PyRuntimeState.

PyRuntimeState seems to be a singleton, so it means that all
interepreters would have the same configuration. Whereas I like the
idea of having a different verbose and/or sys.path per intepreter. Or
maybe I misunderstood what you mean by "core config". I'm talking
about the _current_ _PyCoreConfig in master.

Victor

From ncoghlan at gmail.com  Sat Aug  4 03:55:03 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 4 Aug 2018 17:55:03 +1000
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
Message-ID: <CADiSq7eqUubLK2=3rEmWS1oK9rpqSRE2PoS6cxxGe1r=89AKSg@mail.gmail.com>

On 1 August 2018 at 08:14, Victor Stinner <vstinner at redhat.com> wrote:
> Hi,
>
> I finished my work on the _PyCoreConfig structure: it's a C structure
> in Include/pystate.h which has many fields used to configure Python
> initialization. In Python 3.6 and older, these parameters were scatted
> around the code, and it was hard to get an exhaustive list of it.
>
> This work is linked to the Nick Coghlan's PEP 432 "Restructuring the
> CPython startup sequence":
> https://www.python.org/dev/peps/pep-0432/
>
> Right now, the new API is still private. Nick Coghlan splitted the
> initialization in two parts: "core" and "main". I'm not sure that this
> split is needed.

It is, because one of the aims is to make it clear when frozen
bytecode (ala importlib) and other builtin modules can start to be
used as part of the configuration process. That point is when the core
configuration completes. By contrast, main interpreter configuration
is only complete when you have full filesystem access as well
(including external imports).

That separation also means that an embedding application can choose
*not* to proceed to the second step, and thus limit imports to the
modules built in to the application.

> We should see what to do, but it would be nice to
> make the _PyCoreConfig API public! IMHO it's way better than the old
> way to configuration Python initialization.

Step 1 will be to bring PEP 432 up to date with what actually happened
- we learned a lot from your and Eric's efforts in actually
implementing the draft design as a private API, but the current PEP
still reflects the original design concept.

Thanks for all your work on this! It's exciting to see it finally
coming to fruition :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Aug  4 03:57:36 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 4 Aug 2018 17:57:36 +1000
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CA+3bQGGiQFMkqz_YRommw4OUMKLyySPgGb1B4etB8smptTt_sg@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
 <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
 <CA+3bQGGiQFMkqz_YRommw4OUMKLyySPgGb1B4etB8smptTt_sg@mail.gmail.com>
Message-ID: <CADiSq7cW3rK1wD4HXx0B7m6bMusp-ZVw_SaBBrrjkQWd74HVAg@mail.gmail.com>

On 2 August 2018 at 19:49, Victor Stinner <vstinner at redhat.com> wrote:
> About that, I'm working on a subproject: abandon Py_xxx global
> configuration variables to replace them with accessing
> interpreter->core_config->xxx. I'm not sure yet if it's a good idea or
> not, but it would allow to have two interpreters to have their own
> different configuration. Imagine two interpreters with different
> sys.path running in isolated mode. Or maybe an interpreter without
> importlib?
>
> One of the issue is that we have now two copies of the same option.
> For example, Py_BytesWarningFlag and
> interpreter->core_config->bytes_warning. That's why I would like to
> switch to core_config.

One of the challenges we have around those is the backwards
compatibility implications for embedding applications, so I suspect
the earliest we'll be able to make that change is in the release after
the new initialisation API becomes public.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Aug  4 04:03:06 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 4 Aug 2018 18:03:06 +1000
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CA+3bQGFwoNU16CBaKt0h8eJ7QUmBSykxZ17P-a2JTtcrTx6APA@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
 <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
 <CA+3bQGFwoNU16CBaKt0h8eJ7QUmBSykxZ17P-a2JTtcrTx6APA@mail.gmail.com>
Message-ID: <CADiSq7e-NYCXOqXmLLCsRHH5dhu+4-DxyL2KPf=dYBDU5B8qYQ@mail.gmail.com>

On 3 August 2018 at 01:59, Victor Stinner <vstinner at redhat.com> wrote:
> 2018-08-02 1:18 GMT+02:00 Eric Snow <ericsnowcurrently at gmail.com>:
>> The "core" config is basically the config for the runtime.  In fact,
>> PEP 432 renamed "core" to "runtime".  Please keep the firm distinction
>> between the runtime and the (main) interpreter.
>
> There is already something called _PyRuntime but it's shared between
> all interpreters.
>
> _PyCoreConfig is already *per* interpreter.
>
> Would you mind to elaborate what you mean by the "main interpreter"? I
> don't see anything obvious in the current code about what is a "main
> interpreter". Technically, I don't see anything like that.
>
> I'm still not convinced that we need _PyMainInterpreterConfig:
> _PyCoreConfig contains the same information. Is it really worth it to
> duplicate all _PyCoreConfig (more than 36 fields) in
> _PyMainInterpreterConfig? _PyMainInterpreterConfig adds a third copy
> of many paramters: another opportunity to introduce an inconsistency.
> Right now, an interpreter contains both: core and main
> configurations...

The issue is massive scope creep in the "core config": currently you
need to fully specify everything in order to even use the builtin data
structs. That's not the design goal of PEP 432: the idea is to have an
absolutely bare minimum set of settings that gives a working C API,
but won't let you access the host operating system.

I wasn't reigning you in on it because there were real problems to be
solved in getting the multi-stage start up to work at all given the
current code structure.

Now that it's working though, we should be looking to move settings
back out of coreconfig, and reducing the amount of startup work that
needs to be done using raw C code that can't rely on the CPython C
API.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Aug  4 04:16:46 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 4 Aug 2018 18:16:46 +1000
Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
In-Reply-To: <CA+3bQGGoz=Wk9YsVnFXzqjZ=FrDXamOn_LbN2T3LXONSyjmH5Q@mail.gmail.com>
References: <CA+3bQGGpAjO+KNLRs_QpWAu_U1+Ugcvv4SV_a-dU_yEXvZqm-A@mail.gmail.com>
 <CALFfu7CNO909pvrMzxKDtv7JL3s+v9kvQ9nNitQPvdczSrgV+g@mail.gmail.com>
 <CA+3bQGFwoNU16CBaKt0h8eJ7QUmBSykxZ17P-a2JTtcrTx6APA@mail.gmail.com>
 <CALFfu7B0UjJiwiH1p_vfnmvCa4Lo7e2Ra4nOoT=QiN_4XODVPg@mail.gmail.com>
 <CA+3bQGGoz=Wk9YsVnFXzqjZ=FrDXamOn_LbN2T3LXONSyjmH5Q@mail.gmail.com>
Message-ID: <CADiSq7ccgZ7BwTRG-7hHkrCmc5tW_+46uDu-8yJe7Y4Pm18H0w@mail.gmail.com>

On 4 August 2018 at 08:37, Victor Stinner <vstinner at redhat.com> wrote:
> It seems like the PEP 432 proposes an API designed from scratch as the
> target API. I started from the 28 years old C code and I tried to
> cleanup the code. Our method is different, so it's not surprising that
> the result is different :-)

No, you didn't start from the 28 year old C code - you started from
the private initial implementation of PEP 432, as per
https://www.python.org/dev/peps/pep-0432/#implementation-strategy (and
PEP 432 in turn was born from the Python 3.3 changes needed to
integrate importlib properly into the __main__ initialisation process)

Eric and I merged that [1], because it become apparent while I was
working on the core settings management framework that actually
migrating individual settings needed to happen in-tree, as the
alternative of continuing to maintain it out of tree would at best
result in an enormous unreviewable patch, and more likely never result
in a patch at all (as the churn rate on CPython was high enough to
cause regular conflicts even with the framework branch, let alone once
we started migrating individual settings).

The current private API doesn't meet some of the original design goals
of the PEP, but the time to reconcile that (and figure out whether
it's the API design or the implementation that should change) is while
we're updating and reviewing the PEP against the current
implementation - while everything remained private, it didn't make
sense to throw any additional roadblocks in the way of the excellent
work you were doing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Aug  4 04:20:14 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 4 Aug 2018 18:20:14 +1000
Subject: [Python-Dev] Error message for wrong number of arguments
In-Reply-To: <5B64AE3E.1070003@UGent.be>
References: <5B5F00A6.8070802@UGent.be>
 <256b538ca607434ba56b50aeff886bc5@xmail101.UGent.be>
 <5B64AE3E.1070003@UGent.be>
Message-ID: <CADiSq7d5FuEp4GD1X2ss=z-XqphjFSgzDnTtAD+VQopdTt8qsw@mail.gmail.com>

On 4 August 2018 at 05:34, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
> On 2018-07-30 17:28, Nick Coghlan wrote:
>>
>> I would, and I think it would make sense for the PEP to cite improving
>> consistency (and reducing code duplication?) in that regard as an
>> advantage of the PEP.
>
>
> I'm not sure to which PEP you are referring (PEP 580 or a new PEP?). After
> thinking a bit about the issue of error messages, I realized that PEP 580
> would make this easier to fix (to be clear: there are ways to fix it without
> PEP 580, I'm just saying that PEP 580 makes it easier). There are two
> related reasons for this:
>
> * The existing code which calls the actual underlying C function doesn't
> have access to the Python-level function object. So it can't know whether
> it's a function (where self doesn't count) or a method (where self counts).
>
> * Armin Rigo suggested to use a new flag to indicate this difference: that
> would certainly work for Argument Clinic (just have Argument Clinic add that
> flag). For methods defined without Argument Clinic, we cannot require such a
> new flag though. We could still add the flag at runtime, but it's far from
> clear if we can freely change the flags inside a PyMethodDef at runtime (at
> least, no existing code that I know does that).
>
> PEP 580 solves the first issue by having the function object available and
> it solves the second issue by not relying on PyMethodDef at all for calling
> functions/methods. The second issue especially can be generalized as: PEP
> 580 makes the implementation of functions/methods much less rigid, making it
> easier to change the implementation.
>
> So maybe this can be seen as yet another advantage of PEP 580.

Yes, this is the kind of framing of the issue that I had in mind :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Aug  4 09:13:55 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 4 Aug 2018 23:13:55 +1000
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <20180731094528.118471f9@fsol>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
Message-ID: <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>

On 31 July 2018 at 17:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 31 Jul 2018 09:27:03 +0200
> Jeroen Demeyer <J.Demeyer at UGent.be> wrote:
>> On 2018-07-31 08:58, Antoine Pitrou wrote:
>> > I think Stefan is right that we
>> > should push people towards Cython and alternatives, rather than direct
>> > use of the C API (which people often fail to use correctly, in my
>> > experience).
>>
>> I know this probably isn't the correct place to bring it up, but I'm
>> sure that CPython itself could benefit from using Cython. For example,
>> most of the C extensions in Modules/ could be written in Cython.
>
> We don't depend on any third-party Python modules.  Adding a Cython
> dependency for CPython development would be a tough sell.
>
> Also, a C extension can be built-in (linked statically into the
> interpreter), which I think would be hard to do with Cython.

It'd be *really* nice to at least be able to write some of the C API
tests directly in Cython rather than having to fiddle about with
splitting the test between the regrtest parts that actually define the
test case and the extension module parts that expose the interfaces
that we want to test.

So just as we have a split between the core interpreter components
needed to freeze importlib and the full Python interpreter, it could
be very interesting to have a split between the builtin and extension
modules that are required to bootstrap Cython, and those that can
instead *rely* on Cython as part of their build process.

While actually doing that would likely mean introducing a venv
dependency into the later stages of the build process, we already have
such a dependency for the docs build, and that seems to work OK.

Cheers,
Nick.

P.S. Note that even though static linking Cython generated modules
with CPython doesn't work right just now, experiments like
https://mdqinc.com/blog/2011/08/statically-linking-python-with-cython-generated-modules-and-packages/
suggest that it shouldn't take too many adjustments to the build
process to get it to work by default.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From antoine at python.org  Sat Aug  4 09:57:42 2018
From: antoine at python.org (Antoine Pitrou)
Date: Sat, 4 Aug 2018 15:57:42 +0200
Subject: [Python-Dev] Use of Cython
In-Reply-To: <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
Message-ID: <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>


Hi Nick,

Le 04/08/2018 ? 15:13, Nick Coghlan a ?crit?:
> 
> It'd be *really* nice to at least be able to write some of the C API
> tests directly in Cython rather than having to fiddle about with
> splitting the test between the regrtest parts that actually define the
> test case and the extension module parts that expose the interfaces
> that we want to test.

Actually, I think testing the C API is precisely the kind of area where
you don't want to involve a third-party, especially not a moving target
(Cython is actively maintained and generated code will vary after each
new Cython release).  Besides, Cython itself calls the C API, which
means you might end up involuntarily testing the C API against itself.

If anything, testing the C API using ctypes or cffi would probably be
more reasonable... assuming we get ctypes / cffi to compile everywhere,
which currently isn't the case.

Regards

Antoine.

From stefan_ml at behnel.de  Sat Aug  4 10:46:33 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 4 Aug 2018 16:46:33 +0200
Subject: [Python-Dev] Use of Cython
In-Reply-To: <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
Message-ID: <pk4e46$q92$1@blaine.gmane.org>

Antoine Pitrou schrieb am 04.08.2018 um 15:57:
> Le 04/08/2018 ? 15:13, Nick Coghlan a ?crit?:
>>
>> It'd be *really* nice to at least be able to write some of the C API
>> tests directly in Cython rather than having to fiddle about with
>> splitting the test between the regrtest parts that actually define the
>> test case and the extension module parts that expose the interfaces
>> that we want to test.
> 
> Actually, I think testing the C API is precisely the kind of area where
> you don't want to involve a third-party, especially not a moving target
> (Cython is actively maintained and generated code will vary after each
> new Cython release).  Besides, Cython itself calls the C API, which
> means you might end up involuntarily testing the C API against itself.
> 
> If anything, testing the C API using ctypes or cffi would probably be
> more reasonable... assuming we get ctypes / cffi to compile everywhere,
> which currently isn't the case.

I agree that you would rather not want to let Cython (or another tool)
generate the specific code that tests a specific C-API call, but you could
still use Cython to get around writing the setup, validation and unittest
boilerplate code in C. Basically, a test could then look something like
this (probably works, although I didn't test it):

    from cpython.object cimport PyObject
    from cpython.list cimport PyList_Append

    def test_PyList_Append_on_empty_list():
        # setup code
        l = []
        assert len(l) == 0
        value = "abc"
        pyobj_value = <PyObject*> value
        refcount_before = pyobj_value.ob_refcnt

        # conservative test call, translates to the expected C code,
        # although with exception propagation if it returns -1:
        errcode = PyList_Append(l, value)

        # validation
        assert errcode == 0
        assert len(l) == 1
        assert l[0] is value
        assert pyobj_value.ob_refcnt == refcount_before + 1


If you don't want the exception handling, you can define your own
declaration of PyList_Append() that does not have it. But personally, I'd
rather use try-except in my test code than manually taking care of cleaning
up (unexpected) exceptions.


What we do in Cython, BTW, is write doctests in compiled ".pyx" files. That
allows us to execute certain parts of a test in Python (the doctest code)
and other parts in Cython (the compiled functions/classes that have the
doctests), and thus to do a direct comparison between Python and Cython. An
example that you could find in a test ".pyx" file:

    def test_times2(x):
        """
        doctest that gets executed by Python:

        >>> test_times2(3) == 3 * 2
        True
        """
        # Cython compiled code in a compiled function that gets tested:
        return x * 2


Given that CPython's current "_testcapimodule.c" is only a good 5000 lines
long (divide that by the number of public C-API functions and macros!), I'm
sure the above could help in improving the unit test coverage of the C-API
quite quickly.

Stefan


From ncoghlan at gmail.com  Sat Aug  4 21:15:27 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 5 Aug 2018 11:15:27 +1000
Subject: [Python-Dev] Use of Cython
In-Reply-To: <pk4e46$q92$1@blaine.gmane.org>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
Message-ID: <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>

On 5 August 2018 at 00:46, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Antoine Pitrou schrieb am 04.08.2018 um 15:57:
>> Actually, I think testing the C API is precisely the kind of area where
>> you don't want to involve a third-party, especially not a moving target
>> (Cython is actively maintained and generated code will vary after each
>> new Cython release).  Besides, Cython itself calls the C API, which
>> means you might end up involuntarily testing the C API against itself.
>>
>> If anything, testing the C API using ctypes or cffi would probably be
>> more reasonable... assuming we get ctypes / cffi to compile everywhere,
>> which currently isn't the case.
>
> I agree that you would rather not want to let Cython (or another tool)
> generate the specific code that tests a specific C-API call, but you could
> still use Cython to get around writing the setup, validation and unittest
> boilerplate code in C. Basically, a test could then look something like
> this (probably works, although I didn't test it):
>
>     from cpython.object cimport PyObject
>     from cpython.list cimport PyList_Append
>
>     def test_PyList_Append_on_empty_list():
>         # setup code
>         l = []
>         assert len(l) == 0
>         value = "abc"
>         pyobj_value = <PyObject*> value
>         refcount_before = pyobj_value.ob_refcnt
>
>         # conservative test call, translates to the expected C code,
>         # although with exception propagation if it returns -1:
>         errcode = PyList_Append(l, value)
>
>         # validation
>         assert errcode == 0
>         assert len(l) == 1
>         assert l[0] is value
>         assert pyobj_value.ob_refcnt == refcount_before + 1
>
>
> If you don't want the exception handling, you can define your own
> declaration of PyList_Append() that does not have it. But personally, I'd
> rather use try-except in my test code than manually taking care of cleaning
> up (unexpected) exceptions.

Exactly, that's the kind of thing I had in mind. At the moment,
writing a new dedicated C API test requires designing 4 things:

1. The test case itself (what action to take, which assertions to make about it)
2. The C code to make the API call you want to test
3. The Python->C interface for the test case from 1 to pass test
values in to the code from 2
4. The C->Python interface to get state of interest from 2 back to the
test case from 1

If we were able to use Cython to handle 3 & 4 rather than having to
hand craft it for every test, then I believe it would significantly
lower the barrier to testing the C API directly rather than only
testing it indirectly through the CPython implementation.

Having such a test suite available would then hopefully make it easier
for other implementations to provide robust emulations of the public C
API.

ctypes & cffi likely wouldn't help as much in the case, since they
don't eliminate the need to come up with custom code for parts 3 & 4,
they just let you write that logic in Python rather than C.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ronaldoussoren at mac.com  Sun Aug  5 04:06:43 2018
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sun, 05 Aug 2018 10:06:43 +0200
Subject: [Python-Dev] Use of Cython
In-Reply-To: <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
Message-ID: <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>



> On 5 Aug 2018, at 03:15, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 5 August 2018 at 00:46, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Antoine Pitrou schrieb am 04.08.2018 um 15:57:
>>> Actually, I think testing the C API is precisely the kind of area where
>>> you don't want to involve a third-party, especially not a moving target
>>> (Cython is actively maintained and generated code will vary after each
>>> new Cython release).  Besides, Cython itself calls the C API, which
>>> means you might end up involuntarily testing the C API against itself.
>>> 
>>> If anything, testing the C API using ctypes or cffi would probably be
>>> more reasonable... assuming we get ctypes / cffi to compile everywhere,
>>> which currently isn't the case.
>> 
>> I agree that you would rather not want to let Cython (or another tool)
>> generate the specific code that tests a specific C-API call, but you could
>> still use Cython to get around writing the setup, validation and unittest
>> boilerplate code in C. Basically, a test could then look something like
>> this (probably works, although I didn't test it):
>> 
>>    from cpython.object cimport PyObject
>>    from cpython.list cimport PyList_Append
>> 
>>    def test_PyList_Append_on_empty_list():
>>        # setup code
>>        l = []
>>        assert len(l) == 0
>>        value = "abc"
>>        pyobj_value = <PyObject*> value
>>        refcount_before = pyobj_value.ob_refcnt
>> 
>>        # conservative test call, translates to the expected C code,
>>        # although with exception propagation if it returns -1:
>>        errcode = PyList_Append(l, value)
>> 
>>        # validation
>>        assert errcode == 0
>>        assert len(l) == 1
>>        assert l[0] is value
>>        assert pyobj_value.ob_refcnt == refcount_before + 1
>> 
>> 
>> If you don't want the exception handling, you can define your own
>> declaration of PyList_Append() that does not have it. But personally, I'd
>> rather use try-except in my test code than manually taking care of cleaning
>> up (unexpected) exceptions.
> 
> Exactly, that's the kind of thing I had in mind. At the moment,
> writing a new dedicated C API test requires designing 4 things:
> 
> 1. The test case itself (what action to take, which assertions to make about it)
> 2. The C code to make the API call you want to test
> 3. The Python->C interface for the test case from 1 to pass test
> values in to the code from 2
> 4. The C->Python interface to get state of interest from 2 back to the
> test case from 1
> 
> If we were able to use Cython to handle 3 & 4 rather than having to
> hand craft it for every test, then I believe it would significantly
> lower the barrier to testing the C API directly rather than only
> testing it indirectly through the CPython implementation.
> 
> Having such a test suite available would then hopefully make it easier
> for other implementations to provide robust emulations of the public C
> API.
> 
> ctypes & cffi likely wouldn't help as much in the case, since they
> don't eliminate the need to come up with custom code for parts 3 & 4,
> they just let you write that logic in Python rather than C.

I?m not sure if I understand this, ctypes and cffi are used to access C APIs without writing C code including the CPython API (see for example <https://github.com/abarnert/superhackyinternals/blob/master/internals.py <https://github.com/abarnert/superhackyinternals/blob/master/internals.py>>). 

The code code below should be mostly equivalent to the Cython example posted earlier:

import unittest
import ctypes
from ctypes import pythonapi

class PyObject(ctypes.Structure):
    _fields_ = (
        ('ob_refcnt', ctypes.c_ssize_t),
    )

pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]

def refcount(v):
    return PyObject.from_address(id(v)).ob_refcnt


def test_PyList_Append_on_empty_list():
       # setup code
       l = []
       assert len(l) == 0
       value = "abc"


       refcount_before = refcount(value)

       errcode = pythonapi.PyList_Append(l, value)

       assert errcode == 0
       assert len(l) == 1
       assert l[0] is value
       assert refcount(value) == refcount_before + 1

I write ?mostly? because I rarely use ctypes and am not 100% sure that I use the API correctly.

A problem with using ctypes is that this tests the ABI and to the API, which for example means you cannot test C macros this way.

Ronald

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180805/344db74a/attachment.html>

From ncoghlan at gmail.com  Sun Aug  5 12:14:10 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 6 Aug 2018 02:14:10 +1000
Subject: [Python-Dev] Use of Cython
In-Reply-To: <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
Message-ID: <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>

On 5 August 2018 at 18:06, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> I?m not sure if I understand this, ctypes and cffi are used to access C APIs
> without writing C code including the CPython API (see for example
> <https://github.com/abarnert/superhackyinternals/blob/master/internals.py>).
>
> The code code below should be mostly equivalent to the Cython example posted
> earlier:
>
> import unittest
> import ctypes
> from ctypes import pythonapi
>
> class PyObject(ctypes.Structure):
>     _fields_ = (
>         ('ob_refcnt', ctypes.c_ssize_t),
>     )
>
> pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
>
> def refcount(v):
>     return PyObject.from_address(id(v)).ob_refcnt

The quoted code is what I was referring to in:
====
ctypes & cffi likely wouldn't help as much in the case, since they
don't eliminate the need to come up with custom code for parts 3 & 4,
they just let you write that logic in Python rather than C.
====

Cython has more machinery for accessing the CPython C API correctly
already built in to it, whereas ctypes has no type safety at all,
while cffi doesn't special case CPython's C API in particular.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From aixtools at felt.demon.nl  Sun Aug  5 14:57:40 2018
From: aixtools at felt.demon.nl (Michael)
Date: Sun, 5 Aug 2018 20:57:40 +0200
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and
 Python 3.5.6 are now available
In-Reply-To: <26fd54bd-602d-8870-5f5c-094b58a1fb2c@hastings.org>
References: <c197b678-e6c1-0a54-b3ea-3454cceee987@hastings.org>
 <CA+3bQGHiS=idQusNtEiBrZ9BFpL7ANvXDjgBEdAJRH_GuC7TpQ@mail.gmail.com>
 <26fd54bd-602d-8870-5f5c-094b58a1fb2c@hastings.org>
Message-ID: <eeb74ae1-2d1d-c12e-c090-c2ee3dcee7e0@felt.demon.nl>

On 03/08/2018 03:22, Larry Hastings wrote:
>
>
> On 08/02/2018 07:17 AM, Victor Stinner wrote:
>> 3.4.9 and 3.5.6 have no more known security vulnerabilities :-)
>
> Well, not to be a complete pill, but...
>
> ?? https://bugs.python.org/issue17180
> ?? https://bugs.python.org/issue17239
> ?? https://bugs.python.org/issue19050
>
> Sadly, just because they're languishing on bpo doesn't mean they 
> aren't valid security vulnerabilities.
>
+1 - Sadly, not fixed after 5 years - Why? Because it isn't sexy, or 
fear for breaking things?

Breaking things could be valid - when it is a feature/design change, but 
the whole point of security fixes is because we believe the security 
vulnerability is breakage. Not fixing it keeps everything that depends 
on it (intentional or not) also broken. Any app that depends on 'broken' 
behavior needs to be fixed - rather than let a known vulnerability go 
from 0-day to 1825-day vulnerability (or is it 2000 already?)

Only read the discussion for 17180 - but it seems anything old does not 
get fixed because it did not get fixed years ago.

my two cents!

On a side note: I have been trying to test python on different 
"enterprise" distros of linux and am amazed to see Python2-2.7.5 as the 
'standard'. Rather disheartening for the all the good work that gets 
done. i.e., I am amazed that CVE's like the ones fixed in 3.4.9 and 
3.5.6 (and maybe already/later in 2.7.X) do not motivate distributions 
to update to current levels.

oh my - up to 4 cents! :)

Thanks for the work - I'll get to packaging them for AIX.

>
> //arry/
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/aixtools%40felt.demon.nl


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180805/e0a9710a/attachment.html>

From aixtools at felt.demon.nl  Sun Aug  5 16:59:59 2018
From: aixtools at felt.demon.nl (Michael)
Date: Sun, 5 Aug 2018 22:59:59 +0200
Subject: [Python-Dev] AIX and python tests
Message-ID: <061047f4-6bbc-a662-412d-bb5febfc72bb@felt.demon.nl>

As I have time, I'll dig into these.

I have a couple of PR already 'out there', which I hope someone will be 
looking at when/as he/she/they have time. My time will also be intermittent.

My next test - and I hope not too difficult - would be the test_utf8. 
The test:

FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests) fails - and I am 
wondering if it is as simple as AIX default mode is ISO8559-1 and the 
test looks to be comparing UTF8 with the locale_default. If that is the 
case, obviously this test will never succeed - asis. Am I understanding 
the test properly. If yes, then I'll see what I can come up with for a 
patch to the test for AIX. If no, I'll need some hand holding to help me 
understand the test A bigger challenge, and I think a major issue with 
many of the test failures is test_ssl. Here I already know I'll need so 
assistance. I am quite lost. I know AIX at an expert level, but I do not 
know python (especially python internals, macros, etc..) and after about 
3 levels I am lost. I also find it hard to get 'artifacts' from the 
tests to know what is expected. Looking forward to assistance from 
various people - in understanding the tests, and probably better python 
coding criticism. Michael

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180805/8548bbce/attachment.html>

From python at mrabarnett.plus.com  Sun Aug  5 17:12:24 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 5 Aug 2018 22:12:24 +0100
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and
 Python 3.5.6 are now available
In-Reply-To: <eeb74ae1-2d1d-c12e-c090-c2ee3dcee7e0@felt.demon.nl>
References: <c197b678-e6c1-0a54-b3ea-3454cceee987@hastings.org>
 <CA+3bQGHiS=idQusNtEiBrZ9BFpL7ANvXDjgBEdAJRH_GuC7TpQ@mail.gmail.com>
 <26fd54bd-602d-8870-5f5c-094b58a1fb2c@hastings.org>
 <eeb74ae1-2d1d-c12e-c090-c2ee3dcee7e0@felt.demon.nl>
Message-ID: <2135302d-2937-6f28-1a92-553d1443551f@mrabarnett.plus.com>

On 2018-08-05 19:57, Michael wrote:
> On 03/08/2018 03:22, Larry Hastings wrote:
>>
>>
>> On 08/02/2018 07:17 AM, Victor Stinner wrote:
>>> 3.4.9 and 3.5.6 have no more known security vulnerabilities :-)
>>
>> Well, not to be a complete pill, but...
>>
>> https://bugs.python.org/issue17180
>> https://bugs.python.org/issue17239
>> https://bugs.python.org/issue19050
>>
>> Sadly, just because they're languishing on bpo doesn't mean they 
>> aren't valid security vulnerabilities.
>>
> +1 - Sadly, not fixed after 5 years - Why? Because it isn't sexy, or 
> fear for breaking things?
> 
[snip]Re https://bugs.python.org/issue19050, on Windows 10, Python 3.6 
and Python 3.7 both work OK and Python 3.5 complains about a bad file 
descriptor.

From sjm324 at cornell.edu  Mon Aug  6 01:13:31 2018
From: sjm324 at cornell.edu (Stephen McDowell)
Date: Sun, 5 Aug 2018 22:13:31 -0700
Subject: [Python-Dev] The curious case of 255 function arguments
Message-ID: <CAJGuVd9HLP_78nA-7tVBRMQTMVrH15psWF6vu=L3jb2AVPaX+A@mail.gmail.com>

Hello Python Gurus,

TL;DR: 3.7 released functions having greater than 255 arguments.  Despite
explicit checks for this in 2.x, no such limit is actually imposed -- why?

In the 3.7 release notes "Other Language Changes" section (
https://docs.python.org/3.7/whatsnew/3.7.html#other-language-changes), the
first bullet point denotes

> More than 255 arguments can now be passed to a function, and a function
can now have more than 255 parameters. (Contributed by Serhiy Storchaka in
bpo-12844 <https://bugs.python.org/issue12844> and bpo-18896
<https://bugs.python.org/issue18896>.)

Now lets get something straight: unless I want to exclusively support
Python 3.7 or higher, I must make sure I obey the <255 rule.  Use *args //
**kwargs, etc.  I'm totally ok with that, 2020 is already here in my mind ;)

Curiosity is the reason I'm reaching out.  Upon further investigation and
some discussion with like-minded Python enthusiasts, the code being patched
by Serhiy Storchaka is present in e.g., Python 2.7 (
https://github.com/python/cpython/blob/2.7/Python/ast.c#L2013-L2016)

    if (nargs + nkeywords + ngens > 255) {
      ast_error(n, "more than 255 arguments");
      return NULL;
    }

Despite that code, as demonstrated with the supplemental output in the post
script, *no 2.x versions fail with >255 arguments*.  In contrast, 3.x where
x<7 all do fail (as expected) with a SyntaxError.  To test this, I tried
every minor release of python (excluding v1, arbitrarily choosing the
latest patch release of a minor version) with the following snippet via the
-c flag

    /path/to/pythonX.Y -c 'exec("def foo(" + ", ".join(["a" + str(i) for i
in range(1, 300)]) + "): pass")'

Which tries to construct a function

    def foo(a0, a1, ..., a299): pass

I've looked at the C code for a while and it is entirely non-obvious what
would lead to python *2* *allowing* >255 arguments.  Anybody happen to know
how / why the python *2* versions *succeed*?

Thank you for reading, this is not a problem, just a burning desire for
closure (even if anecdotal) as to how this can be.  I deeply love python,
and am not complaining!  I stumbled across this and found it truly
confounding, and thought the gurus here may happen to recall what changed
in 3.x that lead the the error condition actually being asserted :)

Sincerely,

Stephen McDowell

P.S. On a Fedora 25 box using GCC 6.4.1, I lovingly scripted the
installation of all the python versions just to see if it truly was a 2.x /
3.x divide.  The results of running `python -V` followed by the `python -c
'exec("def foo...")'` described above, with some extra prints for clarity
are as follows (script hackily thrown together in ~30minutes not included,
so as not to make your eyes bleed):

********************************************************************************
Python 2.0.1
==> Greater than 255 Arguments supported
********************************************************************************
Python 2.1.3
==> Greater than 255 Arguments supported
********************************************************************************
Python 2.2.3
==> Greater than 255 Arguments supported
********************************************************************************
Python 2.3.7
==> Greater than 255 Arguments supported
********************************************************************************
Python 2.4.6
==> Greater than 255 Arguments supported
********************************************************************************
Python 2.5.6
==> Greater than 255 Arguments supported
********************************************************************************
Python 2.6.9
==> Greater than 255 Arguments supported
********************************************************************************
Python 2.7.15
==> Greater than 255 Arguments supported
********************************************************************************
Python 3.0.1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1
SyntaxError: more than 255 arguments
********************************************************************************
Python 3.1.5
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1
SyntaxError: more than 255 arguments
********************************************************************************
Python 3.2.6
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1
SyntaxError: more than 255 arguments
********************************************************************************
Python 3.3.7
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1
SyntaxError: more than 255 arguments
********************************************************************************
Python 3.4.9
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1
SyntaxError: more than 255 arguments
********************************************************************************
Python 3.5.6
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1
SyntaxError: more than 255 arguments
********************************************************************************
Python 3.6.6
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1
SyntaxError: more than 255 arguments
********************************************************************************
Python 3.7.0
==> Greater than 255 Arguments supported

P.P.S. Seriously, I LOVE PYTHON <3  It was so easy to download, configure,
build, and install each of these versions, and run the test!  Thank you :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180805/9ddbc2f3/attachment.html>

From agriff at tin.it  Mon Aug  6 03:57:11 2018
From: agriff at tin.it (Andrea Griffini)
Date: Mon, 6 Aug 2018 09:57:11 +0200
Subject: [Python-Dev] The curious case of 255 function arguments
In-Reply-To: <CAJGuVd9HLP_78nA-7tVBRMQTMVrH15psWF6vu=L3jb2AVPaX+A@mail.gmail.com>
References: <CAJGuVd9HLP_78nA-7tVBRMQTMVrH15psWF6vu=L3jb2AVPaX+A@mail.gmail.com>
Message-ID: <CALy7ODvysQAxXHqHwkB2yDkdD_Si7Txwgwh8wXfE-nnPi4c+DA@mail.gmail.com>

With Python 2.7.15 what fails is a call with explicit arguments (e.g.
`foo(0,0,0 ... 0,0)`), not the function definition.
Calling with `foo([0]*300)` instead works.


On Mon, Aug 6, 2018 at 7:18 AM Stephen McDowell <sjm324 at cornell.edu> wrote:

> Hello Python Gurus,
>
> TL;DR: 3.7 released functions having greater than 255 arguments.  Despite
> explicit checks for this in 2.x, no such limit is actually imposed -- why?
>
> In the 3.7 release notes "Other Language Changes" section (
> https://docs.python.org/3.7/whatsnew/3.7.html#other-language-changes),
> the first bullet point denotes
>
> > More than 255 arguments can now be passed to a function, and a function
> can now have more than 255 parameters. (Contributed by Serhiy Storchaka in
> bpo-12844 <https://bugs.python.org/issue12844> and bpo-18896
> <https://bugs.python.org/issue18896>.)
>
> Now lets get something straight: unless I want to exclusively support
> Python 3.7 or higher, I must make sure I obey the <255 rule.  Use *args //
> **kwargs, etc.  I'm totally ok with that, 2020 is already here in my mind ;)
>
> Curiosity is the reason I'm reaching out.  Upon further investigation and
> some discussion with like-minded Python enthusiasts, the code being patched
> by Serhiy Storchaka is present in e.g., Python 2.7 (
> https://github.com/python/cpython/blob/2.7/Python/ast.c#L2013-L2016)
>
>     if (nargs + nkeywords + ngens > 255) {
>       ast_error(n, "more than 255 arguments");
>       return NULL;
>     }
>
> Despite that code, as demonstrated with the supplemental output in the
> post script, *no 2.x versions fail with >255 arguments*.  In contrast,
> 3.x where x<7 all do fail (as expected) with a SyntaxError.  To test
> this, I tried every minor release of python (excluding v1, arbitrarily
> choosing the latest patch release of a minor version) with the following
> snippet via the -c flag
>
>     /path/to/pythonX.Y -c 'exec("def foo(" + ", ".join(["a" + str(i) for i
> in range(1, 300)]) + "): pass")'
>
> Which tries to construct a function
>
>     def foo(a0, a1, ..., a299): pass
>
> I've looked at the C code for a while and it is entirely non-obvious what
> would lead to python *2* *allowing* >255 arguments.  Anybody happen to
> know how / why the python *2* versions *succeed*?
>
> Thank you for reading, this is not a problem, just a burning desire for
> closure (even if anecdotal) as to how this can be.  I deeply love python,
> and am not complaining!  I stumbled across this and found it truly
> confounding, and thought the gurus here may happen to recall what changed
> in 3.x that lead the the error condition actually being asserted :)
>
> Sincerely,
>
> Stephen McDowell
>
> P.S. On a Fedora 25 box using GCC 6.4.1, I lovingly scripted the
> installation of all the python versions just to see if it truly was a 2.x /
> 3.x divide.  The results of running `python -V` followed by the `python
> -c 'exec("def foo...")'` described above, with some extra prints for
> clarity are as follows (script hackily thrown together in ~30minutes not
> included, so as not to make your eyes bleed):
>
>
> ********************************************************************************
> Python 2.0.1
> ==> Greater than 255 Arguments supported
>
> ********************************************************************************
> Python 2.1.3
> ==> Greater than 255 Arguments supported
>
> ********************************************************************************
> Python 2.2.3
> ==> Greater than 255 Arguments supported
>
> ********************************************************************************
> Python 2.3.7
> ==> Greater than 255 Arguments supported
>
> ********************************************************************************
> Python 2.4.6
> ==> Greater than 255 Arguments supported
>
> ********************************************************************************
> Python 2.5.6
> ==> Greater than 255 Arguments supported
>
> ********************************************************************************
> Python 2.6.9
> ==> Greater than 255 Arguments supported
>
> ********************************************************************************
> Python 2.7.15
> ==> Greater than 255 Arguments supported
>
> ********************************************************************************
> Python 3.0.1
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "<string>", line 1
> SyntaxError: more than 255 arguments
>
> ********************************************************************************
> Python 3.1.5
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "<string>", line 1
> SyntaxError: more than 255 arguments
>
> ********************************************************************************
> Python 3.2.6
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "<string>", line 1
> SyntaxError: more than 255 arguments
>
> ********************************************************************************
> Python 3.3.7
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "<string>", line 1
> SyntaxError: more than 255 arguments
>
> ********************************************************************************
> Python 3.4.9
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "<string>", line 1
> SyntaxError: more than 255 arguments
>
> ********************************************************************************
> Python 3.5.6
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "<string>", line 1
> SyntaxError: more than 255 arguments
>
> ********************************************************************************
> Python 3.6.6
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "<string>", line 1
> SyntaxError: more than 255 arguments
>
> ********************************************************************************
> Python 3.7.0
> ==> Greater than 255 Arguments supported
>
> P.P.S. Seriously, I LOVE PYTHON <3  It was so easy to download, configure,
> build, and install each of these versions, and run the test!  Thank you :)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/agriff%40tin.it
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180806/9142ef34/attachment-0001.html>

From agriff at tin.it  Mon Aug  6 03:57:57 2018
From: agriff at tin.it (Andrea Griffini)
Date: Mon, 6 Aug 2018 09:57:57 +0200
Subject: [Python-Dev] The curious case of 255 function arguments
In-Reply-To: <CALy7ODvysQAxXHqHwkB2yDkdD_Si7Txwgwh8wXfE-nnPi4c+DA@mail.gmail.com>
References: <CAJGuVd9HLP_78nA-7tVBRMQTMVrH15psWF6vu=L3jb2AVPaX+A@mail.gmail.com>
 <CALy7ODvysQAxXHqHwkB2yDkdD_Si7Txwgwh8wXfE-nnPi4c+DA@mail.gmail.com>
Message-ID: <CALy7ODsiPyEUueDjCPNi3n7HfoeEtJob5AHRAPP_iob_eKF0kg@mail.gmail.com>

typo... meant of course foo(*([0]*300))

Andrea

On Mon, Aug 6, 2018 at 9:57 AM Andrea Griffini <agriff at tin.it> wrote:

> With Python 2.7.15 what fails is a call with explicit arguments (e.g.
> `foo(0,0,0 ... 0,0)`), not the function definition.
> Calling with `foo([0]*300)` instead works.
>
>
> On Mon, Aug 6, 2018 at 7:18 AM Stephen McDowell <sjm324 at cornell.edu>
> wrote:
>
>> Hello Python Gurus,
>>
>> TL;DR: 3.7 released functions having greater than 255 arguments.  Despite
>> explicit checks for this in 2.x, no such limit is actually imposed -- why?
>>
>> In the 3.7 release notes "Other Language Changes" section (
>> https://docs.python.org/3.7/whatsnew/3.7.html#other-language-changes),
>> the first bullet point denotes
>>
>> > More than 255 arguments can now be passed to a function, and a function
>> can now have more than 255 parameters. (Contributed by Serhiy Storchaka in
>> bpo-12844 <https://bugs.python.org/issue12844> and bpo-18896
>> <https://bugs.python.org/issue18896>.)
>>
>> Now lets get something straight: unless I want to exclusively support
>> Python 3.7 or higher, I must make sure I obey the <255 rule.  Use *args //
>> **kwargs, etc.  I'm totally ok with that, 2020 is already here in my mind ;)
>>
>> Curiosity is the reason I'm reaching out.  Upon further investigation and
>> some discussion with like-minded Python enthusiasts, the code being patched
>> by Serhiy Storchaka is present in e.g., Python 2.7 (
>> https://github.com/python/cpython/blob/2.7/Python/ast.c#L2013-L2016)
>>
>>     if (nargs + nkeywords + ngens > 255) {
>>       ast_error(n, "more than 255 arguments");
>>       return NULL;
>>     }
>>
>> Despite that code, as demonstrated with the supplemental output in the
>> post script, *no 2.x versions fail with >255 arguments*.  In contrast,
>> 3.x where x<7 all do fail (as expected) with a SyntaxError.  To test
>> this, I tried every minor release of python (excluding v1, arbitrarily
>> choosing the latest patch release of a minor version) with the following
>> snippet via the -c flag
>>
>>     /path/to/pythonX.Y -c 'exec("def foo(" + ", ".join(["a" + str(i) for
>> i in range(1, 300)]) + "): pass")'
>>
>> Which tries to construct a function
>>
>>     def foo(a0, a1, ..., a299): pass
>>
>> I've looked at the C code for a while and it is entirely non-obvious what
>> would lead to python *2* *allowing* >255 arguments.  Anybody happen to
>> know how / why the python *2* versions *succeed*?
>>
>> Thank you for reading, this is not a problem, just a burning desire for
>> closure (even if anecdotal) as to how this can be.  I deeply love python,
>> and am not complaining!  I stumbled across this and found it truly
>> confounding, and thought the gurus here may happen to recall what changed
>> in 3.x that lead the the error condition actually being asserted :)
>>
>> Sincerely,
>>
>> Stephen McDowell
>>
>> P.S. On a Fedora 25 box using GCC 6.4.1, I lovingly scripted the
>> installation of all the python versions just to see if it truly was a 2.x /
>> 3.x divide.  The results of running `python -V` followed by the `python
>> -c 'exec("def foo...")'` described above, with some extra prints for
>> clarity are as follows (script hackily thrown together in ~30minutes not
>> included, so as not to make your eyes bleed):
>>
>>
>> ********************************************************************************
>> Python 2.0.1
>> ==> Greater than 255 Arguments supported
>>
>> ********************************************************************************
>> Python 2.1.3
>> ==> Greater than 255 Arguments supported
>>
>> ********************************************************************************
>> Python 2.2.3
>> ==> Greater than 255 Arguments supported
>>
>> ********************************************************************************
>> Python 2.3.7
>> ==> Greater than 255 Arguments supported
>>
>> ********************************************************************************
>> Python 2.4.6
>> ==> Greater than 255 Arguments supported
>>
>> ********************************************************************************
>> Python 2.5.6
>> ==> Greater than 255 Arguments supported
>>
>> ********************************************************************************
>> Python 2.6.9
>> ==> Greater than 255 Arguments supported
>>
>> ********************************************************************************
>> Python 2.7.15
>> ==> Greater than 255 Arguments supported
>>
>> ********************************************************************************
>> Python 3.0.1
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "<string>", line 1
>> SyntaxError: more than 255 arguments
>>
>> ********************************************************************************
>> Python 3.1.5
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "<string>", line 1
>> SyntaxError: more than 255 arguments
>>
>> ********************************************************************************
>> Python 3.2.6
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "<string>", line 1
>> SyntaxError: more than 255 arguments
>>
>> ********************************************************************************
>> Python 3.3.7
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "<string>", line 1
>> SyntaxError: more than 255 arguments
>>
>> ********************************************************************************
>> Python 3.4.9
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "<string>", line 1
>> SyntaxError: more than 255 arguments
>>
>> ********************************************************************************
>> Python 3.5.6
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "<string>", line 1
>> SyntaxError: more than 255 arguments
>>
>> ********************************************************************************
>> Python 3.6.6
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "<string>", line 1
>> SyntaxError: more than 255 arguments
>>
>> ********************************************************************************
>> Python 3.7.0
>> ==> Greater than 255 Arguments supported
>>
>> P.P.S. Seriously, I LOVE PYTHON <3  It was so easy to download,
>> configure, build, and install each of these versions, and run the test!
>> Thank you :)
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/agriff%40tin.it
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180806/c8dab3bf/attachment.html>

From storchaka at gmail.com  Mon Aug  6 05:17:52 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 6 Aug 2018 12:17:52 +0300
Subject: [Python-Dev] The curious case of 255 function arguments
In-Reply-To: <CAJGuVd9HLP_78nA-7tVBRMQTMVrH15psWF6vu=L3jb2AVPaX+A@mail.gmail.com>
References: <CAJGuVd9HLP_78nA-7tVBRMQTMVrH15psWF6vu=L3jb2AVPaX+A@mail.gmail.com>
Message-ID: <pk93ju$4kr$1@blaine.gmane.org>

06.08.18 08:13, Stephen McDowell ????:
> I've looked at the C code for a while and it is entirely non-obvious 
> what would lead to python *2* /allowing/ >255 arguments.? Anybody happen 
> to know how / why the python *2* versions *succeed*?

The error message is misleading. It should be "more than 255 
parameters". This limitation is due to the optimization used in Python 3 
for call variables (see https://bugs.python.org/issue12399 for details).

In all versions <3.7 there is a limitation on the number of explicit 
function arguments because of the limitation of the CALL_FUNCTION opcode.

> Thank you for reading, this is not a problem, just a burning desire for 
> closure (even if anecdotal) as to how this can be.? I deeply love 
> python, and am not complaining!? I stumbled across this and found it 
> truly confounding, and thought the gurus here may happen to recall what 
> changed in 3.x that lead the the error condition actually being asserted :)

Read the history of the code. Commit messages usually contain 
explanations or references to issues.


From cstratak at redhat.com  Mon Aug  6 05:38:29 2018
From: cstratak at redhat.com (Charalampos Stratakis)
Date: Mon, 6 Aug 2018 05:38:29 -0400 (EDT)
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and
 Python 3.5.6 are now available
In-Reply-To: <eeb74ae1-2d1d-c12e-c090-c2ee3dcee7e0@felt.demon.nl>
References: <c197b678-e6c1-0a54-b3ea-3454cceee987@hastings.org>
 <CA+3bQGHiS=idQusNtEiBrZ9BFpL7ANvXDjgBEdAJRH_GuC7TpQ@mail.gmail.com>
 <26fd54bd-602d-8870-5f5c-094b58a1fb2c@hastings.org>
 <eeb74ae1-2d1d-c12e-c090-c2ee3dcee7e0@felt.demon.nl>
Message-ID: <2139465616.54858485.1533548309691.JavaMail.zimbra@redhat.com>

----- Original Message -----

> From: "Michael" <aixtools at felt.demon.nl>
> To: "Larry Hastings" <larry at hastings.org>, python-dev at python.org
> Sent: Sunday, August 5, 2018 8:57:40 PM
> Subject: Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and
> Python 3.5.6 are now available

> On 03/08/2018 03:22, Larry Hastings wrote:

> > On 08/02/2018 07:17 AM, Victor Stinner wrote:
> 

> > > 3.4.9 and 3.5.6 have no more known security vulnerabilities :-)
> > 
> 

> > Well, not to be a complete pill, but...
> 

> > https://bugs.python.org/issue17180
> 
> > https://bugs.python.org/issue17239
> 
> > https://bugs.python.org/issue19050
> 

> > Sadly, just because they're languishing on bpo doesn't mean they aren't
> > valid
> > security vulnerabilities.
> 

> +1 - Sadly, not fixed after 5 years - Why? Because it isn't sexy, or fear for
> breaking things?

> Breaking things could be valid - when it is a feature/design change, but the
> whole point of security fixes is because we believe the security
> vulnerability is breakage. Not fixing it keeps everything that depends on it
> (intentional or not) also broken. Any app that depends on 'broken' behavior
> needs to be fixed - rather than let a known vulnerability go from 0-day to
> 1825-day vulnerability (or is it 2000 already?)

> Only read the discussion for 17180 - but it seems anything old does not get
> fixed because it did not get fixed years ago.

> my two cents!

> On a side note: I have been trying to test python on different "enterprise"
> distros of linux and am amazed to see Python2-2.7.5 as the 'standard'.
> Rather disheartening for the all the good work that gets done. i.e., I am
> amazed that CVE's like the ones fixed in 3.4.9 and 3.5.6 (and maybe
> already/later in 2.7.X) do not motivate distributions to update to current
> levels.

A side note on your side note. Different distro's have different standards, use/customer cases to address etc. In enterprise distributions the usual scheme is that the version that you see is the minimum one and many fixes coming from upstream or the redistributor 
are incorporated on top of that version. Just check the package changelogs. :) CVE's do get fixed and there is actually cooperation with upstream on different levels in regards to those. And speaking here as one of the people doing that for one of the enterprise 
distros. 

> oh my - up to 4 cents! :)

> Thanks for the work - I'll get to packaging them for AIX.

> > //arry/
> 

> > _______________________________________________
> 
> > Python-Dev mailing list Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/aixtools%40felt.demon.nl
> 

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/cstratak%40redhat.com

-- 
Regards, 

Charalampos Stratakis 
Software Engineer 
Python Maintenance Team, Red Hat 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180806/a9a1a75f/attachment.html>

From ronaldoussoren at mac.com  Mon Aug  6 09:25:12 2018
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 06 Aug 2018 15:25:12 +0200
Subject: [Python-Dev] Use of Cython
In-Reply-To: <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
Message-ID: <2B17179F-29B4-40E2-824D-749359E33089@mac.com>



> On 5 Aug 2018, at 18:14, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 5 August 2018 at 18:06, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>> I?m not sure if I understand this, ctypes and cffi are used to access C APIs
>> without writing C code including the CPython API (see for example
>> <https://github.com/abarnert/superhackyinternals/blob/master/internals.py>).
>> 
>> The code code below should be mostly equivalent to the Cython example posted
>> earlier:
>> 
>> import unittest
>> import ctypes
>> from ctypes import pythonapi
>> 
>> class PyObject(ctypes.Structure):
>>    _fields_ = (
>>        ('ob_refcnt', ctypes.c_ssize_t),
>>    )
>> 
>> pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
>> 
>> def refcount(v):
>>    return PyObject.from_address(id(v)).ob_refcnt
> 
> The quoted code is what I was referring to in:
> ====
> ctypes & cffi likely wouldn't help as much in the case, since they
> don't eliminate the need to come up with custom code for parts 3 & 4,
> they just let you write that logic in Python rather than C.
> ====

And earlier Nick wrote:
> 1. The test case itself (what action to take, which assertions to make about it)
> 2. The C code to make the API call you want to test
> 3. The Python->C interface for the test case from 1 to pass test
> values in to the code from 2
> 4. The C->Python interface to get state of interest from 2 back to the
> test case from 1

For all of Cython, ctypes and cffi you almost never have to write (2), and hence (3) and (4), but can write that code in Python. This is at the code of making it harder to know which bits of the CPython API are used in step (2), which makes it harder to review a testcase. 

BTW. In other projects I use tests there almost all of the test code is in C, the unittest runner only calls a C function and uses the result of that function to deduce if the test passed or failed. This only works nicely for fairly simple tests (such as the example test in this thread), not for more complicated and interesting tests due to having to write more C code.

Ronald

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180806/2e041841/attachment.html>

From stefan_ml at behnel.de  Mon Aug  6 11:13:38 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 6 Aug 2018 17:13:38 +0200
Subject: [Python-Dev] Use of Cython
In-Reply-To: <2B17179F-29B4-40E2-824D-749359E33089@mac.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
 <2B17179F-29B4-40E2-824D-749359E33089@mac.com>
Message-ID: <pk9oev$eh3$1@blaine.gmane.org>

Ronald Oussoren via Python-Dev schrieb am 06.08.2018 um 15:25:
>> On 5 Aug 2018, at 18:14, Nick Coghlan wrote:
>> On 5 August 2018 at 18:06, Ronald Oussoren wrote:
>>> I?m not sure if I understand this, ctypes and cffi are used to access C APIs
>>> without writing C code including the CPython API (see for example
>>> <https://github.com/abarnert/superhackyinternals/blob/master/internals.py>).
>>>
>>> The code code below should be mostly equivalent to the Cython example posted
>>> earlier:
>>>
>>> import unittest
>>> import ctypes
>>> from ctypes import pythonapi
>>>
>>> class PyObject(ctypes.Structure):
>>>    _fields_ = (
>>>        ('ob_refcnt', ctypes.c_ssize_t),
>>>    )
>>>
>>> pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
>>>
>>> def refcount(v):
>>>    return PyObject.from_address(id(v)).ob_refcnt
>>
>> The quoted code is what I was referring to in:
>> ====
>> ctypes & cffi likely wouldn't help as much in the case, since they
>> don't eliminate the need to come up with custom code for parts 3 & 4,
>> they just let you write that logic in Python rather than C.
>> ====
> 
> And earlier Nick wrote:
>> 1. The test case itself (what action to take, which assertions to make about it)
>> 2. The C code to make the API call you want to test
>> 3. The Python->C interface for the test case from 1 to pass test
>> values in to the code from 2
>> 4. The C->Python interface to get state of interest from 2 back to the
>> test case from 1
> 
> For all of Cython, ctypes and cffi you almost never have to write (2), and hence (3) and (4), but can write that code in Python.

Which then means that you have a mix of Python and C in many cases. I guess
that's what you meant with your next sentence:

> This is at the code of making it harder to know which bits of the CPython API are used in step (2), which makes it harder to review a testcase. 

Not sure I understand this correctly, but I think we're on the same page
here: writing test code in C is cumbersome, writing test code in a mix of C
and Python across different files is aweful. And making it difficult to
write or even just review test code simply means that people will either
waste their precious contribution time on it, or try to get around it.


> BTW. In other projects I use tests there almost all of the test code is in C, the unittest runner only calls a C function and uses the result of that function to deduce if the test passed or failed. This only works nicely for fairly simple tests (such as the example test in this thread), not for more complicated and interesting tests due to having to write more C code.

I totally agree with that. For less trivial tests, people will often want
to stear the test case at the C level, because some things are really
difficult to do from Python. Good luck making assertions about reference
counts when you're orchestrating the C-API through ctypes. And this is
where Cython shines ? your code *always* ends up running in C, regardless
of how much of it is plain Python. But at any point, you can do pretty
arbitrary C things, all in the same function. And unittest can execute that
function directly for you, without having to write a Python wrapper or
separate test runner.

And for the really hard cases, you can resort to writing a literal C code
snippet in your Cython source file (as a string) and let Cython drop it
into the file it generates, e.g. to quickly define a macro, a little C
function, or an interface wrapper around a C macro that would otherwise be
difficult to test. That little feature removes the last reason for
resorting to a separate C file.

Stefan


From ronaldoussoren at mac.com  Mon Aug  6 11:47:26 2018
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 06 Aug 2018 17:47:26 +0200
Subject: [Python-Dev] Use of Cython
In-Reply-To: <pk9oev$eh3$1@blaine.gmane.org>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
 <2B17179F-29B4-40E2-824D-749359E33089@mac.com> <pk9oev$eh3$1@blaine.gmane.org>
Message-ID: <57363ED3-4851-4A5D-A4B9-F9F0A9053F2C@mac.com>



> On 6 Aug 2018, at 17:13, Stefan Behnel <stefan_ml at behnel.de> wrote:
> 
> Ronald Oussoren via Python-Dev schrieb am 06.08.2018 um 15:25:
>>> On 5 Aug 2018, at 18:14, Nick Coghlan wrote:
>>> On 5 August 2018 at 18:06, Ronald Oussoren wrote:
>>>> I?m not sure if I understand this, ctypes and cffi are used to access C APIs
>>>> without writing C code including the CPython API (see for example
>>>> <https://github.com/abarnert/superhackyinternals/blob/master/internals.py>).
>>>> 
>>>> The code code below should be mostly equivalent to the Cython example posted
>>>> earlier:
>>>> 
>>>> import unittest
>>>> import ctypes
>>>> from ctypes import pythonapi
>>>> 
>>>> class PyObject(ctypes.Structure):
>>>>   _fields_ = (
>>>>       ('ob_refcnt', ctypes.c_ssize_t),
>>>>   )
>>>> 
>>>> pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
>>>> 
>>>> def refcount(v):
>>>>   return PyObject.from_address(id(v)).ob_refcnt
>>> 
>>> The quoted code is what I was referring to in:
>>> ====
>>> ctypes & cffi likely wouldn't help as much in the case, since they
>>> don't eliminate the need to come up with custom code for parts 3 & 4,
>>> they just let you write that logic in Python rather than C.
>>> ====
>> 
>> And earlier Nick wrote:
>>> 1. The test case itself (what action to take, which assertions to make about it)
>>> 2. The C code to make the API call you want to test
>>> 3. The Python->C interface for the test case from 1 to pass test
>>> values in to the code from 2
>>> 4. The C->Python interface to get state of interest from 2 back to the
>>> test case from 1
>> 
>> For all of Cython, ctypes and cffi you almost never have to write (2), and hence (3) and (4), but can write that code in Python.
> 
> Which then means that you have a mix of Python and C in many cases.

Not really, for many cases the only C code required is the actual CPython API. 

> I guess
> that's what you meant with your next sentence:
> 
>> This is at the code of making it harder to know which bits of the CPython API are used in step (2), which makes it harder to review a testcase. 

Not really. What I was trying to say is that when you use C code to make the API call and collect information you have complete control of which APIs are used (in the PyList_Append test case you can be 100% sure that the test code only calls that API), with ctypes/cffi/Cython you have less control over the exact API calls made which could affect the test. Making assertions about refcounts is one example, my test case with ctypes works but you have to think about what the code does before you understand why the test case works and tests what you want to test.  In general that?s not a good quality for test code, that should be as obviously correct as possible. 

Cython is probably the best option in this regard (from what I?ve seen so far) because you can in the end basically write C code with Python syntax where needed.

> 
> Not sure I understand this correctly, but I think we're on the same page
> here: writing test code in C is cumbersome, writing test code in a mix of C
> and Python across different files is aweful. And making it difficult to
> write or even just review test code simply means that people will either
> waste their precious contribution time on it, or try to get around it.

Awful is a strong word. The separation of the test case in two different languages in two different source files is definitely less than ideal. 

> 
> 
>> BTW. In other projects I use tests there almost all of the test code is in C, the unittest runner only calls a C function and uses the result of that function to deduce if the test passed or failed. This only works nicely for fairly simple tests (such as the example test in this thread), not for more complicated and interesting tests due to having to write more C code.
> 
> I totally agree with that. For less trivial tests, people will often want
> to stear the test case at the C level, because some things are really
> difficult to do from Python. Good luck making assertions about reference
> counts when you're orchestrating the C-API through ctypes.

I agree, see above.


> And this is
> where Cython shines ? your code *always* ends up running in C, regardless
> of how much of it is plain Python. But at any point, you can do pretty
> arbitrary C things, all in the same function. And unittest can execute that
> function directly for you, without having to write a Python wrapper or
> separate test runner.
> 
> And for the really hard cases, you can resort to writing a literal C code
> snippet in your Cython source file (as a string) and let Cython drop it
> into the file it generates, e.g. to quickly define a macro, a little C
> function, or an interface wrapper around a C macro that would otherwise be
> difficult to test. That little feature removes the last reason for
> resorting to a separate C file.

Another reason for liking the translation to C is that you can use and test the macros in the CPython API, which is something you definitely cannot do with ctypes.

I have no strong opinion on using Cython for tests or in the stdlib, other than that it is a fairly large dependency.  I do think that adding a ?Cython-lite? tool the CPython distribution would be less ideal, creating and maintaining that tool would be a lot of work without clear benefits over just using Cython.

Ronald


From eelizondo at fb.com  Mon Aug  6 14:02:18 2018
From: eelizondo at fb.com (Eddie Elizondo)
Date: Mon, 6 Aug 2018 18:02:18 +0000
Subject: [Python-Dev] A Subtle Bug in Class Initializations
Message-ID: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>

Background:
Through the implementation of an alternate runtime I've been poking around some of the class initialization routines and I found out that there was a subtle bug with PyType_Ready and the header initializer PyVarObject_HEAD_INIT.

Looking through the codebase, I couldn't really find any pattern of when the type should be defined within PyVarObject_HEAD_INIT. Sometimes it was initialized to NULL (or 0) and PyType_Type (let's ignore Py_True and Py_False from now).

From PyType_Ready it turns out that setting the value PyType_Type is never actually needed outside of PyType_Type and PyBaseObject type. This is clear from the code:
if (Py_TYPE(type) == NULL && base != NULL)
    Py_TYPE(type) = Py_TYPE(base);


Given that any PyTypeObject's base is of type PyType_Type, setting PyVarObject_HEAD_INIT(&PyType_Ready) is superfluous. Therefore, setting all static PyTypeObjects to their ob_type to NULL should be a safe assumption to make.

Uninitialized Types:
A quick s/PyVarObject_HEAD_INIT(&PyType_Type/PyVarObject_HEAD_INIT(NULL/  shows that some objects do need to have their ob_type set from the outset, violating the previous assumption. After writing a quick script, I found that out of the ~300 PyVarObject_HEAD_INIT present in CPython, only these 14 types segfaulted:

PyByteArrayIter_Type
PyBytesIter_Type
PyDictIterKey_Type
PyDictIterValue_Type
PyDictIterItem_Type
PyClassMethod_Type
PyAsyncGen_Type
PyListIter_Type
PyListRevIter_Type
PyODictIter_Type
PyLongRangeIter_Type
PySetIter_Type
PyTupleIter_Type
PyUnicodeIter_Type


Bug:
It turns out that these PyTypeObjects are never initialized through PyType_Ready. However, they are used as fully initialized types. It is by pure chance that the work without having to call the initializer on them. This though is undefined behavior. This not only might result in a weird future bug which is hard to chase down but also, it affects other runtimes as this behavior depends on implementation details of CPython.

This is a pervasive pattern that should be removed from the codebase and ideally extensions should follow as well.

Solution:
Here are my proposed solutions in order from less controversial to most controversial. Note that all of them I already tried in a local branch and are working:

1) Initialize all uninitialized types.

Example: https://github.com/eduardo-elizondo/cpython/commit/bc53db3cf4e5a6923b0b1afa6181305553faf173

2) Move all PyVarObject_HEAD_INIT to NULL except PyType_Type, PyBaseObject_Type and the booleans.

3) Special case the initialization of PyType_Type and PyBaseObject_Type within PyType_Ready to now make all calls to PyVarObject_HEAD_INIT use NULL. To enable this a small change within PyType_Ready is needed to initialize PyType_Type PyBaseObject:
if (base == NULL) {
    Py_TYPE(&PyType_Type) = &PyType_Type;
    Py_TYPE(type) = &PyType_Type;
}

Also, the booleans have to be fully initialized without calling PyVarObject_HEAD_INIT. I propose:

struct _longobject _Py_FalseStruct = {
    PyObject_HEAD_INIT(&PyBool_Type), 0, { 0 }
};

This will clean-up the entire codebase of this anti-pattern.

Example: https://github.com/eduardo-elizondo/cpython/commit/542fd79e4279c64c077c127b175a8d856d3c5f0b

4) Modify PyVarObject_HEAD_INIT to ignore the input and initialize to NULL and 0.
In order to prevent this antipattern within extension code as well, we should make PyVarObject_HEAD_INIT ignore the inputs and just set the value to NULL.

#define PyVarObject_HEAD_INIT(type, size)       \
    { PyObject_HEAD_INIT(NULL) 0 },

This will prevent external code to have a semi-initialized type that is not initialized through PyType_Ready.

5) Finally, I would go even further and suggest making PyVarObject_HEAD_INIT argumentless.

#define PyVarObject_HEAD_INIT       \
    { PyObject_HEAD_INIT(NULL) 0 },

However, this breaks backward compatibility. That being said, this will make extension maintainers aware of this antipattern.

Example: https://github.com/eduardo-elizondo/cpython/commit/3869e53843008ff096764f4adaf26efbb5625996

Thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180806/702762a1/attachment.html>

From aixtools at felt.demon.nl  Tue Aug  7 09:16:41 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Tue, 7 Aug 2018 15:16:41 +0200
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and
 Python 3.5.6 are now available
In-Reply-To: <2139465616.54858485.1533548309691.JavaMail.zimbra@redhat.com>
References: <c197b678-e6c1-0a54-b3ea-3454cceee987@hastings.org>
 <CA+3bQGHiS=idQusNtEiBrZ9BFpL7ANvXDjgBEdAJRH_GuC7TpQ@mail.gmail.com>
 <26fd54bd-602d-8870-5f5c-094b58a1fb2c@hastings.org>
 <eeb74ae1-2d1d-c12e-c090-c2ee3dcee7e0@felt.demon.nl>
 <2139465616.54858485.1533548309691.JavaMail.zimbra@redhat.com>
Message-ID: <84293210-f1f4-624c-f5fc-ee931f16cb71@felt.demon.nl>



On 8/6/2018 11:38 AM, Charalampos Stratakis wrote:
> A side note on your side note. Different distro's have different
> standards, use/customer cases to address etc. In enterprise
> distributions the usual scheme is that the version that you see is the
> minimum one and many fixes coming from upstream or the redistributor
> are incorporated on top of that version. Just check the package
> changelogs. :) CVE's do get fixed and there is actually cooperation
> with upstream on different levels in regards to those. And speaking
> here as one of the people doing that for one of the enterprise
> distros.
>
a) good to hear
b) On AIX they stayed with ssh at version 6.0 for so long, that even
with all the CVE et al included it was still extremely weak compared to
6.7 and later when they tightened the default ciphers. And yes, I fell
over the change - but was glad, in the end, to rid of weak ssh clients.
c) read package changelogs. The :) is because they are hard to read or
non-existent.

I do not mean to criticize any "enterprise" methods. My "enterprise" of
choice is AIX and when it comes to OSS I dare say everyone else does a
better job (which is why I got started with packaging in the first place
- but only what I need and/or someone requests). However, I do find it
very very hard to know what python 2.7.5 has or has not, that 2.7.15 now
has. There are, iirc, quite a few important changes. The "hard" freeze
seems to have come at roughly 2.7.8 or 2.7.9 (just a guess).

Also, as I am trying to test on other platforms it gets a bit
frustrating when the latest python3 I can find is a v3.4.X.

Might be good project developers (in general, not meant as specific to
python) to understand that version number changes are not followed -
blindly - by enterprise patch management and being too quick with
version number changes will make it more difficult for users to know
what they have.

p.s. I do not do this (packaging/patch management) for any "distro". In
that sense I am "just a consumer" who "rolls his own" when/if needed.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: pEpkey.asc
Type: application/pgp-keys
Size: 1765 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180807/23f39bf6/attachment.key>

From parktasi at gmail.com  Tue Aug  7 03:00:19 2018
From: parktasi at gmail.com (=?UTF-8?B?6JSh6YqY5bOv?=)
Date: Tue, 7 Aug 2018 15:00:19 +0800
Subject: [Python-Dev] Refactor __get_builtin_constructor on hasklib.py
Message-ID: <CAAypT1ftkJXpBdTEtKX8i=ENSdUcGwq1n5e6_S6_cWDeaOuM2g@mail.gmail.com>

Hello everybody,
I am Park Tsai. I want to refactor __get_builtin_constructor on hasklib.py
of python 2.7 (https://github.com/python/cpython/blob/2.7/Lib/hashlib.py#L72).

This is the first time that I try to refactor code of CPython on GitHub, so
I am very excited.

This is __get_builtin_constructor code on hasklib.py ,as follows.

def __get_builtin_constructor(name):
    try:
        if name in ('SHA1', 'sha1'):
            import _sha
            return _sha.new
        elif name in ('MD5', 'md5'):
            import _md5
            return _md5.new
        elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
            import _sha256
            bs = name[3:]
            if bs == '256':
                return _sha256.sha256
            elif bs == '224':
                return _sha256.sha224
        elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
            import _sha512
            bs = name[3:]
            if bs == '512':
                return _sha512.sha512
            elif bs == '384':
                return _sha512.sha384
    except ImportError:
        pass  # no extension module, this hash is unsupported.

    raise ValueError('unsupported hash type ' + name)


When I read this code, it looks messy, so I want to refactor it and make it
become more clearly .

Then, it will be like this

def get_builtin_constructor(name):
    try:
        if name[:3] in ('SHA','sha'):
           if(name[3:]=='1'):
               import _sha
               return _sha.new

           elif (name[3:] == '224'):
               import _sha256
               return _sha256.sha224

           elif (name[3:] == '256'):
               import _sha256
               return _sha256.sha256

           elif (name[3:] == '384'):
               import _sha512
               return _sha512.sha384

           elif (name[3:] == '512'):
               import _sha512
               return _sha512.sha512
        elif name in ('MD5', 'md5'):
            import _md5
            return _md5.new

    except ImportError:
        pass # no extension module, this hash is unsupported.

    raise ValueError('unsupported hash type ' + name)

I will be grateful for any help you can provide. I really appreciate any
feedback you can offer!

Best regards,
Park Tsai !!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180807/a5a5df41/attachment.html>

From sanyam.khurana01 at gmail.com  Tue Aug  7 11:26:14 2018
From: sanyam.khurana01 at gmail.com (Sanyam Khurana)
Date: Tue, 7 Aug 2018 20:56:14 +0530
Subject: [Python-Dev] Refactor __get_builtin_constructor on hasklib.py
In-Reply-To: <CAAypT1ftkJXpBdTEtKX8i=ENSdUcGwq1n5e6_S6_cWDeaOuM2g@mail.gmail.com>
References: <CAAypT1ftkJXpBdTEtKX8i=ENSdUcGwq1n5e6_S6_cWDeaOuM2g@mail.gmail.com>
Message-ID: <CAGz_E1mLM1TQ+433bLhoNL0zuuOy3BE-FvyRCWEOggndgwFpLQ@mail.gmail.com>

Hello,

Welcome to the mailing list Park!

On Tue, Aug 7, 2018 at 12:30 PM, ??? <parktasi at gmail.com> wrote:
> Hello everybody,
> I am Park Tsai. I want to refactor __get_builtin_constructor on hasklib.py
> of python 2.7
> (https://github.com/python/cpython/blob/2.7/Lib/hashlib.py#L72).
> This is the first time that I try to refactor code of CPython on GitHub, so
> I am very excited.
>
> This is __get_builtin_constructor code on hasklib.py ,as follows.
>
> def __get_builtin_constructor(name):
>     try:
>         if name in ('SHA1', 'sha1'):
>             import _sha
>             return _sha.new
>         elif name in ('MD5', 'md5'):
>             import _md5
>             return _md5.new
>         elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
>             import _sha256
>             bs = name[3:]
>             if bs == '256':
>                 return _sha256.sha256
>             elif bs == '224':
>                 return _sha256.sha224
>         elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
>             import _sha512
>             bs = name[3:]
>             if bs == '512':
>                 return _sha512.sha512
>             elif bs == '384':
>                 return _sha512.sha384
>     except ImportError:
>         pass  # no extension module, this hash is unsupported.
>
>     raise ValueError('unsupported hash type ' + name)
>
>
> When I read this code, it looks messy, so I want to refactor it and make it
> become more clearly .
>
> Then, it will be like this
>
> def get_builtin_constructor(name):
>     try:
>         if name[:3] in ('SHA','sha'):
>            if(name[3:]=='1'):
>                import _sha
>                return _sha.new
>
>            elif (name[3:] == '224'):
>                import _sha256
>                return _sha256.sha224
>
>            elif (name[3:] == '256'):
>                import _sha256
>                return _sha256.sha256
>
>            elif (name[3:] == '384'):
>                import _sha512
>                return _sha512.sha384
>
>            elif (name[3:] == '512'):
>                import _sha512
>                return _sha512.sha512


IMHO, this decreases the readability of the code. Also, we're doing
string slicing at every conditional statement which doesn't make much
sense.

What do you find not interesting with the current code? What is the
motivation behind this change?

Best,
Sanyam


-- 
Mozilla Rep
http://www.SanyamKhurana.com
Github: CuriousLearner

From mariatta.wijaya at gmail.com  Tue Aug  7 11:51:07 2018
From: mariatta.wijaya at gmail.com (Mariatta Wijaya)
Date: Tue, 7 Aug 2018 08:51:07 -0700
Subject: [Python-Dev] Refactor __get_builtin_constructor on hasklib.py
In-Reply-To: <CAAypT1ftkJXpBdTEtKX8i=ENSdUcGwq1n5e6_S6_cWDeaOuM2g@mail.gmail.com>
References: <CAAypT1ftkJXpBdTEtKX8i=ENSdUcGwq1n5e6_S6_cWDeaOuM2g@mail.gmail.com>
Message-ID: <CAGbohnasYn=ytUE_0o9sHCpROGkduSo-Ga9hQyb8ypWksX_hiw@mail.gmail.com>

2.7 is for bug fixes only. Unless there is a bug to be fixed, I would leave
the code as is.

Mariatta


On Tue, Aug 7, 2018 at 8:14 AM ??? <parktasi at gmail.com> wrote:

> Hello everybody,
> I am Park Tsai. I want to refactor __get_builtin_constructor on hasklib.py
> of python 2.7 (
> https://github.com/python/cpython/blob/2.7/Lib/hashlib.py#L72).
> This is the first time that I try to refactor code of CPython on GitHub,
> so I am very excited.
>
> This is __get_builtin_constructor code on hasklib.py ,as follows.
>
> def __get_builtin_constructor(name):
>     try:
>         if name in ('SHA1', 'sha1'):
>             import _sha
>             return _sha.new
>         elif name in ('MD5', 'md5'):
>             import _md5
>             return _md5.new
>         elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
>             import _sha256
>             bs = name[3:]
>             if bs == '256':
>                 return _sha256.sha256
>             elif bs == '224':
>                 return _sha256.sha224
>         elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
>             import _sha512
>             bs = name[3:]
>             if bs == '512':
>                 return _sha512.sha512
>             elif bs == '384':
>                 return _sha512.sha384
>     except ImportError:
>         pass  # no extension module, this hash is unsupported.
>
>     raise ValueError('unsupported hash type ' + name)
>
>
> When I read this code, it looks messy, so I want to refactor it and make
> it become more clearly .
>
> Then, it will be like this
>
> def get_builtin_constructor(name):
>     try:
>         if name[:3] in ('SHA','sha'):
>            if(name[3:]=='1'):
>                import _sha
>                return _sha.new
>
>            elif (name[3:] == '224'):
>                import _sha256
>                return _sha256.sha224
>
>            elif (name[3:] == '256'):
>                import _sha256
>                return _sha256.sha256
>
>            elif (name[3:] == '384'):
>                import _sha512
>                return _sha512.sha384
>
>            elif (name[3:] == '512'):
>                import _sha512
>                return _sha512.sha512
>         elif name in ('MD5', 'md5'):
>             import _md5
>             return _md5.new
>
>     except ImportError:
>         pass # no extension module, this hash is unsupported.
>
>     raise ValueError('unsupported hash type ' + name)
>
> I will be grateful for any help you can provide. I really appreciate any
> feedback you can offer!
>
> Best regards,
> Park Tsai !!
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mariatta.wijaya%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180807/320f38cb/attachment-0001.html>

From yselivanov.ml at gmail.com  Tue Aug  7 13:34:32 2018
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 7 Aug 2018 13:34:32 -0400
Subject: [Python-Dev] Use of Cython
In-Reply-To: <57363ED3-4851-4A5D-A4B9-F9F0A9053F2C@mac.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
 <2B17179F-29B4-40E2-824D-749359E33089@mac.com> <pk9oev$eh3$1@blaine.gmane.org>
 <57363ED3-4851-4A5D-A4B9-F9F0A9053F2C@mac.com>
Message-ID: <CA+St6D1DNV3j9ieH-M9X0mkMH5mAXKYYOqBeDUWT952ZkCdNyA@mail.gmail.com>

On Mon, Aug 6, 2018 at 11:49 AM Ronald Oussoren via Python-Dev
<python-dev at python.org> wrote:

> I have no strong opinion on using Cython for tests or in the stdlib, other than that it is a fairly large dependency.  I do think that adding a ?Cython-lite? tool the CPython distribution would be less ideal, creating and maintaining that tool would be a lot of work without clear benefits over just using Cython.

Speaking of which, Dropbox is working on a new compiler they call "mypyc".

mypyc will compile type-annotated Python code to an optimized C. The
first goal is to compile mypy with it to make it faster, so I hope
that the project will be completed. Essentially, mypyc will be similar
to Cython, but mypyc is a *subset of Python*, not a superset.
Interfacing with C libraries can be easily achieved with cffi. Being a
strict subset of Python means that mypyc code will execute just fine
in PyPy. They can even apply some optimizations to it eventually, as
it has a strict and static type system.

I'd be more willing to start using mypyc+cffi in CPython stdlib
*eventually*, than Cython now.  Cython is a relatively complex and
still poorly documented language.  I'm speaking from experience after
writing thousands of lines of Cython in uvloop & asyncpg.  In skillful
hands Cython is amazing, but I'd be cautious to advertise and use it
in CPython.

I'm also -1 on using Cython to test C API. While writing C tests is
annoying (I wrote a fair share myself), their very purpose is to make
third-party tools/extensions more stable. Using a third-party tool to
test C API to track regressions that break third-party tools feels
wrong.

Yury

From steve at holdenweb.com  Tue Aug  7 14:04:28 2018
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 7 Aug 2018 19:04:28 +0100
Subject: [Python-Dev] Refactor __get_builtin_constructor on hasklib.py
In-Reply-To: <CAAypT1ftkJXpBdTEtKX8i=ENSdUcGwq1n5e6_S6_cWDeaOuM2g@mail.gmail.com>
References: <CAAypT1ftkJXpBdTEtKX8i=ENSdUcGwq1n5e6_S6_cWDeaOuM2g@mail.gmail.com>
Message-ID: <CAMofdRBbHKE2t0=uRT-9TDLPQdjjzFHPsKRqDjAOtf9EoGXnsA@mail.gmail.com>

Hi there,

Good to see you on python-dev. It's always good to see people getting
excited about helping with Python's development.

The changes you suggest are, unless I've missed something, purely cosmetic
- affecting readability only. This implies that you feel the code as it
stands isn't as readable as it could be. You must admit that "it looks
messy" is a matter of opinion, and alone isn't much of a justification for
making changes to a project that will reach its end of life in less than
eighteen months.

The code in the standard library is mostly fairly well-scrutinised, and as
PEP 8 reminds us, change made for purely stylistic reasons threatens to
introduce new bugs.

It's not obvious how much of the developer documentation you've seen, so it
might be worth mentioning https://devguide.python.org/ as a good starting
point for anyone wanting to contribute.

regards
 Steve

Steve Holden

On Tue, Aug 7, 2018 at 8:00 AM, ??? <parktasi at gmail.com> wrote:

> Hello everybody,
> I am Park Tsai. I want to refactor __get_builtin_constructor on hasklib.py
> of python 2.7 (https://github.com/python/cpython/blob/2.7/Lib/hashlib.
> py#L72).
> This is the first time that I try to refactor code of CPython on GitHub,
> so I am very excited.
>
> This is __get_builtin_constructor code on hasklib.py ,as follows.
>
> def __get_builtin_constructor(name):
>     try:
>         if name in ('SHA1', 'sha1'):
>             import _sha
>             return _sha.new
>         elif name in ('MD5', 'md5'):
>             import _md5
>             return _md5.new
>         elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
>             import _sha256
>             bs = name[3:]
>             if bs == '256':
>                 return _sha256.sha256
>             elif bs == '224':
>                 return _sha256.sha224
>         elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
>             import _sha512
>             bs = name[3:]
>             if bs == '512':
>                 return _sha512.sha512
>             elif bs == '384':
>                 return _sha512.sha384
>     except ImportError:
>         pass  # no extension module, this hash is unsupported.
>
>     raise ValueError('unsupported hash type ' + name)
>
>
> When I read this code, it looks messy, so I want to refactor it and make
> it become more clearly .
>
> Then, it will be like this
>
> def get_builtin_constructor(name):
>     try:
>         if name[:3] in ('SHA','sha'):
>            if(name[3:]=='1'):
>                import _sha
>                return _sha.new
>
>            elif (name[3:] == '224'):
>                import _sha256
>                return _sha256.sha224
>
>            elif (name[3:] == '256'):
>                import _sha256
>                return _sha256.sha256
>
>            elif (name[3:] == '384'):
>                import _sha512
>                return _sha512.sha384
>
>            elif (name[3:] == '512'):
>                import _sha512
>                return _sha512.sha512
>         elif name in ('MD5', 'md5'):
>             import _md5
>             return _md5.new
>
>     except ImportError:
>         pass # no extension module, this hash is unsupported.
>
>     raise ValueError('unsupported hash type ' + name)
>
> I will be grateful for any help you can provide. I really appreciate any
> feedback you can offer!
>
> Best regards,
> Park Tsai !!
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> steve%40holdenweb.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180807/92bbd68a/attachment.html>

From sjm324 at cornell.edu  Tue Aug  7 19:13:45 2018
From: sjm324 at cornell.edu (Stephen McDowell)
Date: Tue, 7 Aug 2018 16:13:45 -0700
Subject: [Python-Dev] The curious case of 255 function arguments
In-Reply-To: <pk93ju$4kr$1@blaine.gmane.org>
References: <CAJGuVd9HLP_78nA-7tVBRMQTMVrH15psWF6vu=L3jb2AVPaX+A@mail.gmail.com>
 <pk93ju$4kr$1@blaine.gmane.org>
Message-ID: <CAJGuVd-T49fg33tAPawvUbTgaVi2WOe22x6ne2ZUB1u2K=HPjQ@mail.gmail.com>

Hi Andrea and Serhiy,

Thank you for your responses and clarifying that it is specifically the
CALL_FUNCTION.  I tested this in my megascript and it will fail when trying
to call the functions directly and receive an error then (Py 2.x: fail at
call invocation, Py 3.y w/ y<7: fail at function definition).

@Serhiy I looked through the commits and had found
https://github.com/python/cpython/commit/5bb8b9134b0bb35a73c76657f41cafa3e4361fcd#diff-4d35cf8992b795c5e97e9c8b6167cb34
but the commit that removed the 255 checks also explains that this is
specifically about the call function (
https://github.com/python/cpython/commit/214678e44bf7773c0ed9c3684818354001d8f9ca#diff-4d35cf8992b795c5e97e9c8b6167cb34
), so indeed I should have been able to answer this myself.

The reason why I originally had encountered this was (as discussed in one
of the bug reports) from code that was generating a class hierarchy to
represent Doxygen's XML schema.  The class constructors had >255 arguments,
but in executing the code it actually does still work in python 2.x.  The
reason is because all of the arguments are defaulted to None, and during
execution of typical sample XML files, the explicit construction with all
>255 arguments virtually never happens.

    f.write("def foo_2({0}):\n".format(",
".join(["a{0}=None".format(str(i)) for i in range(300)])))
    f.write("    print('foo_2 executed')\n\n")
    # ... in generated __main__ ...
    f.write("    foo_2()\n\n")

foo_2() will succeed in python 2.x because the CALL_FUNCTION is not
explicitly getting more than 255 parameters.  Very interesting!

Thank you both again for your responses, I am grateful to finally
understand the way in which success / failure works here :)

-Stephen


On Mon, Aug 6, 2018 at 2:17 AM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> 06.08.18 08:13, Stephen McDowell ????:
>
>> I've looked at the C code for a while and it is entirely non-obvious what
>> would lead to python *2* /allowing/ >255 arguments.  Anybody happen to know
>> how / why the python *2* versions *succeed*?
>>
>
> The error message is misleading. It should be "more than 255 parameters".
> This limitation is due to the optimization used in Python 3 for call
> variables (see https://bugs.python.org/issue12399 for details).
>
> In all versions <3.7 there is a limitation on the number of explicit
> function arguments because of the limitation of the CALL_FUNCTION opcode.
>
> Thank you for reading, this is not a problem, just a burning desire for
>> closure (even if anecdotal) as to how this can be.  I deeply love python,
>> and am not complaining!  I stumbled across this and found it truly
>> confounding, and thought the gurus here may happen to recall what changed
>> in 3.x that lead the the error condition actually being asserted :)
>>
>
> Read the history of the code. Commit messages usually contain explanations
> or references to issues.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/sjm324%
> 40cornell.edu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180807/70b16c13/attachment.html>

From benjamin at python.org  Wed Aug  8 00:13:56 2018
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 07 Aug 2018 21:13:56 -0700
Subject: [Python-Dev] A Subtle Bug in Class Initializations
In-Reply-To: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
References: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
Message-ID: <1533701636.3913944.1467040008.228AE842@webmail.messagingengine.com>

This would be a good thing to fix. The only hard part is dealing with thirdparty extensions.

Note we also have been working around this problem by putting PyType_Ready calls in various generic code paths of the interpreter when an uninitialized type passes through.

On Mon, Aug 6, 2018, at 11:02, Eddie Elizondo wrote:
> Solution:
> Here are my proposed solutions in order from less controversial to most 
> controversial. Note that all of them I already tried in a local branch 
> and are working:
> 
> 1) Initialize all uninitialized types.
> 
> Example: 
> https://github.com/eduardo-elizondo/cpython/commit/bc53db3cf4e5a6923b0b1afa6181305553faf173

That fixes CPython but what about thirdparty extensions?

> 
> 2) Move all PyVarObject_HEAD_INIT to NULL except PyType_Type, 
> PyBaseObject_Type and the booleans.
> 
> 3) Special case the initialization of PyType_Type and PyBaseObject_Type 
> within PyType_Ready to now make all calls to PyVarObject_HEAD_INIT use 
> NULL. To enable this a small change within PyType_Ready is needed to 
> initialize PyType_Type PyBaseObject:
> if (base == NULL) {
>     Py_TYPE(&PyType_Type) = &PyType_Type;
>     Py_TYPE(type) = &PyType_Type;
> }
> 
> Also, the booleans have to be fully initialized without calling 
> PyVarObject_HEAD_INIT. I propose:
> 
> struct _longobject _Py_FalseStruct = {
>     PyObject_HEAD_INIT(&PyBool_Type), 0, { 0 }
> };
> 
> This will clean-up the entire codebase of this anti-pattern.
> 
> Example: 
> https://github.com/eduardo-elizondo/cpython/commit/542fd79e4279c64c077c127b175a8d856d3c5f0b
> 
> 4) Modify PyVarObject_HEAD_INIT to ignore the input and initialize to 
> NULL and 0.
> In order to prevent this antipattern within extension code as well, we 
> should make PyVarObject_HEAD_INIT ignore the inputs and just set the 
> value to NULL.
> 
> #define PyVarObject_HEAD_INIT(type, size)       \
>     { PyObject_HEAD_INIT(NULL) 0 },
> 
> This will prevent external code to have a semi-initialized type that is 
> not initialized through PyType_Ready.

When does that fail, though? The first time someone tries to do anything with the type?

> 
> 5) Finally, I would go even further and suggest making 
> PyVarObject_HEAD_INIT argumentless.
> 
> #define PyVarObject_HEAD_INIT       \
>     { PyObject_HEAD_INIT(NULL) 0 },
> 
> However, this breaks backward compatibility. That being said, this will 
> make extension maintainers aware of this antipattern.

From mcepl at cepl.eu  Wed Aug  8 06:46:55 2018
From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl)
Date: Wed, 08 Aug 2018 12:46:55 +0200
Subject: [Python-Dev] Use of Cython
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
 <2B17179F-29B4-40E2-824D-749359E33089@mac.com>
 <pk9oev$eh3$1@blaine.gmane.org>
 <57363ED3-4851-4A5D-A4B9-F9F0A9053F2C@mac.com>
 <CA+St6D1DNV3j9ieH-M9X0mkMH5mAXKYYOqBeDUWT952ZkCdNyA@mail.gmail.com>
Message-ID: <slrnpmligv.biq.mcepl@milic.suse.cz>

On 2018-08-07, 17:34 GMT, Yury Selivanov wrote:
> Speaking of which, Dropbox is working on a new compiler they 
> call "mypyc".

How does it compare with Nuitka?

Mat?j
-- 
https://matej.ceplovi.cz/blog/, Jabber: mcepl at ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
 
If in desperation, read the documentation!
    -- Brian D. Ripley, on R-help list


From mcepl at cepl.eu  Wed Aug  8 06:44:50 2018
From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl)
Date: Wed, 08 Aug 2018 12:44:50 +0200
Subject: [Python-Dev] Use of Cython
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
 <2B17179F-29B4-40E2-824D-749359E33089@mac.com>
 <pk9oev$eh3$1@blaine.gmane.org>
Message-ID: <slrnpmlid2.biq.mcepl@milic.suse.cz>

On 2018-08-06, 15:13 GMT, Stefan Behnel wrote:
> Not sure I understand this correctly, but I think we're on the 
> same page here: writing test code in C is cumbersome, writing 
> test code in a mix of C and Python across different files is 
> aweful. And making it difficult to write or even just review 
> test code simply means that people will either waste their 
> precious contribution time on it, or try to get around it.

I was thinking about the same when porting M2Crypto to py3k 
(M2Crypto is currently swig-based mix of C-code and Python). Is 
it even possible to make a mix of Cython, swig-based C, and 
Python? In the end I rather stayed with plain C, because the 
combination seems unimaginably awful.

(Also, is Cython the best of all of them? What about cffi or 
Nuitka?)

Best,

Mat?j
-- 
https://matej.ceplovi.cz/blog/, Jabber: mcepl at ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
 
There is no reason to suppose that most human beings are engaged
in maximizing anything unless it be unhappiness, and even this
with incomplete success.
    -- Ronald Coase
       Introduction to ?The Firm, the Market, and the Law?


From solipsis at pitrou.net  Wed Aug  8 07:30:32 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 8 Aug 2018 13:30:32 +0200
Subject: [Python-Dev] Use of Cython
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
 <2B17179F-29B4-40E2-824D-749359E33089@mac.com>
 <pk9oev$eh3$1@blaine.gmane.org>
 <slrnpmlid2.biq.mcepl@milic.suse.cz>
Message-ID: <20180808133032.4ae6ed9c@fsol>

On Wed, 08 Aug 2018 12:44:50 +0200
Mat?j Cepl <mcepl at cepl.eu> wrote:
> On 2018-08-06, 15:13 GMT, Stefan Behnel wrote:
> > Not sure I understand this correctly, but I think we're on the 
> > same page here: writing test code in C is cumbersome, writing 
> > test code in a mix of C and Python across different files is 
> > aweful. And making it difficult to write or even just review 
> > test code simply means that people will either waste their 
> > precious contribution time on it, or try to get around it.  
> 
> I was thinking about the same when porting M2Crypto to py3k 
> (M2Crypto is currently swig-based mix of C-code and Python). Is 
> it even possible to make a mix of Cython, swig-based C, and 
> Python? In the end I rather stayed with plain C, because the 
> combination seems unimaginably awful.

I'm not sure why anyone would want to use swig nowadays.

> (Also, is Cython the best of all of them? What about cffi or 
> Nuitka?)

This sounds like comparing apples to oranges.

Regards

Antoine.



From aixtools at felt.demon.nl  Wed Aug  8 09:50:51 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Wed, 8 Aug 2018 15:50:51 +0200
Subject: [Python-Dev] AIX and python tests
In-Reply-To: <061047f4-6bbc-a662-412d-bb5febfc72bb@felt.demon.nl>
References: <061047f4-6bbc-a662-412d-bb5febfc72bb@felt.demon.nl>
Message-ID: <a63496f5-ca17-0e91-599d-4a7bc0fc9d30@felt.demon.nl>

Try again on this.

a) Victor has said he will look, from time to time - after his vacation.
b) our vacations do not overlap
c) comment was also made privately, re: my starting a worker for
buildbot, that there is not much sense in a bot if noone is working on
the tests.

I'll do my best, in the (limited) time I have to work on c) - but alone
I cannot get anything done.

So, Victor suggested I just ask for others to review for now - so I can
have some semblance of moving forward - before my vacation starts (about
when Victor gets back from his).

In advance - many thanks.

On 8/5/2018 10:59 PM, Michael wrote:
>
> As I have time, I'll dig into these.
>
> I have a couple of PR already 'out there', which I hope someone will
> be looking at when/as he/she/they have time. My time will also be
> intermittent.
>
> My next test - and I hope not too difficult - would be the test_utf8.
> The test:
>
> FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests) fails - and I
> am wondering if it is as simple as AIX default mode is ISO8559-1 and
> the test looks to be comparing UTF8 with the locale_default. If that
> is the case, obviously this test will never succeed - asis. Am I
> understanding the test properly. If yes, then I'll see what I can come
> up with for a patch to the test for AIX. If no, I'll need some hand
> holding to help me understand the test A bigger challenge, and I think
> a major issue with many of the test failures is test_ssl. Here I
> already know I'll need so assistance. I am quite lost. I know AIX at
> an expert level, but I do not know python (especially python
> internals, macros, etc..) and after about 3 levels I am lost. I also
> find it hard to get 'artifacts' from the tests to know what is
> expected. Looking forward to assistance from various people - in
> understanding the tests, and probably better python coding criticism.
> Michael
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/aixtools%40felt.demon.nl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180808/b3325911/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pEpkey.asc
Type: application/pgp-keys
Size: 1765 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180808/b3325911/attachment-0001.key>

From mcepl at cepl.eu  Wed Aug  8 10:09:41 2018
From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl)
Date: Wed, 08 Aug 2018 16:09:41 +0200
Subject: [Python-Dev] Use of Cython
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
 <2B17179F-29B4-40E2-824D-749359E33089@mac.com>
 <pk9oev$eh3$1@blaine.gmane.org> <slrnpmlid2.biq.mcepl@milic.suse.cz>
 <20180808133032.4ae6ed9c@fsol>
Message-ID: <slrnpmlud5.drv.mcepl@milic.suse.cz>

On 2018-08-08, 11:30 GMT, Antoine Pitrou wrote:
> I'm not sure why anyone would want to use swig nowadays.

Legacy reasons, 47k lines of Python code (and 7k lines of swig 
*.i files).

Mat?j
-- 
https://matej.ceplovi.cz/blog/, Jabber: mcepl at ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
 
Do not long for the night, when people vanish in their place.
Be careful, do not turn to evil; for you have preferred this to
affliction.
      -- Job 36:20f (NASB)


From parktasi at gmail.com  Wed Aug  8 04:01:39 2018
From: parktasi at gmail.com (Tsai Park)
Date: Wed, 8 Aug 2018 16:01:39 +0800
Subject: [Python-Dev] Refactor __get_builtin_constructor on hasklib.py
In-Reply-To: <CAMofdRBbHKE2t0=uRT-9TDLPQdjjzFHPsKRqDjAOtf9EoGXnsA@mail.gmail.com>
References: <CAAypT1ftkJXpBdTEtKX8i=ENSdUcGwq1n5e6_S6_cWDeaOuM2g@mail.gmail.com>
 <CAMofdRBbHKE2t0=uRT-9TDLPQdjjzFHPsKRqDjAOtf9EoGXnsA@mail.gmail.com>
Message-ID: <CAAypT1dNqB4fk4BmYmw3QJ3mzh9JZjJiB+AP06gXckOQToBqiw@mail.gmail.com>

Dear Sanyam,Mariatta and Steve,
Thanks for your feedback.
I will read https://devguide.python.org/ carefully.
Best regards,
Park Tsai

2018-08-08 2:04 GMT+08:00 Steve Holden <steve at holdenweb.com>:

> Hi there,
>
> Good to see you on python-dev. It's always good to see people getting
> excited about helping with Python's development.
>
> The changes you suggest are, unless I've missed something, purely cosmetic
> - affecting readability only. This implies that you feel the code as it
> stands isn't as readable as it could be. You must admit that "it looks
> messy" is a matter of opinion, and alone isn't much of a justification for
> making changes to a project that will reach its end of life in less than
> eighteen months.
>
> The code in the standard library is mostly fairly well-scrutinised, and as
> PEP 8 reminds us, change made for purely stylistic reasons threatens to
> introduce new bugs.
>
> It's not obvious how much of the developer documentation you've seen, so
> it might be worth mentioning https://devguide.python.org/ as a good
> starting point for anyone wanting to contribute.
>
> regards
>  Steve
>
> Steve Holden
>
> On Tue, Aug 7, 2018 at 8:00 AM, ??? <parktasi at gmail.com> wrote:
>
>> Hello everybody,
>> I am Park Tsai. I want to refactor __get_builtin_constructor on
>> hasklib.py of python 2.7 (https://github.com/python/cpy
>> thon/blob/2.7/Lib/hashlib.py#L72).
>> This is the first time that I try to refactor code of CPython on GitHub,
>> so I am very excited.
>>
>> This is __get_builtin_constructor code on hasklib.py ,as follows.
>>
>> def __get_builtin_constructor(name):
>>     try:
>>         if name in ('SHA1', 'sha1'):
>>             import _sha
>>             return _sha.new
>>         elif name in ('MD5', 'md5'):
>>             import _md5
>>             return _md5.new
>>         elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
>>             import _sha256
>>             bs = name[3:]
>>             if bs == '256':
>>                 return _sha256.sha256
>>             elif bs == '224':
>>                 return _sha256.sha224
>>         elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
>>             import _sha512
>>             bs = name[3:]
>>             if bs == '512':
>>                 return _sha512.sha512
>>             elif bs == '384':
>>                 return _sha512.sha384
>>     except ImportError:
>>         pass  # no extension module, this hash is unsupported.
>>
>>     raise ValueError('unsupported hash type ' + name)
>>
>>
>> When I read this code, it looks messy, so I want to refactor it and make
>> it become more clearly .
>>
>> Then, it will be like this
>>
>> def get_builtin_constructor(name):
>>     try:
>>         if name[:3] in ('SHA','sha'):
>>            if(name[3:]=='1'):
>>                import _sha
>>                return _sha.new
>>
>>            elif (name[3:] == '224'):
>>                import _sha256
>>                return _sha256.sha224
>>
>>            elif (name[3:] == '256'):
>>                import _sha256
>>                return _sha256.sha256
>>
>>            elif (name[3:] == '384'):
>>                import _sha512
>>                return _sha512.sha384
>>
>>            elif (name[3:] == '512'):
>>                import _sha512
>>                return _sha512.sha512
>>         elif name in ('MD5', 'md5'):
>>             import _md5
>>             return _md5.new
>>
>>     except ImportError:
>>         pass # no extension module, this hash is unsupported.
>>
>>     raise ValueError('unsupported hash type ' + name)
>>
>> I will be grateful for any help you can provide. I really appreciate any
>> feedback you can offer!
>>
>> Best regards,
>> Park Tsai !!
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve%
>> 40holdenweb.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180808/f24ea734/attachment.html>

From aixtools at felt.demon.nl  Thu Aug  9 05:40:38 2018
From: aixtools at felt.demon.nl (Michael)
Date: Thu, 9 Aug 2018 11:40:38 +0200
Subject: [Python-Dev] [issue11191] test_search_cpp error on AIX (with
 xlc)
In-Reply-To: <1533768778.85.0.902498594338.issue11191@psf.upfronthosting.co.za>
References: <1533768778.85.0.902498594338.issue11191@psf.upfronthosting.co.za>
Message-ID: <a5afb2a0-4247-b4f0-7e6e-65d1f3054cb4@felt.demon.nl>

On 09/08/2018 00:52, Michael Felt wrote:
> Change by Michael Felt <aixtools at felt.demon.nl>:
>
>
> ----------
> pull_requests: +8196
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <https://bugs.python.org/issue11191>
> _______________________________________
>
Don't know why it is says +8196. I would have expected +8709.
https://github.com/python/cpython/pull/8709

Would appreciate someone look at/review this. It corrects 6 or 7 errors
from test_distutils (when not using gcc on AIX). And it is very small.
Maybe smaller if the review is to remove the comments.

Thx.


From erik.m.bray at gmail.com  Thu Aug  9 11:18:57 2018
From: erik.m.bray at gmail.com (Erik Bray)
Date: Thu, 9 Aug 2018 17:18:57 +0200
Subject: [Python-Dev] A Subtle Bug in Class Initializations
In-Reply-To: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
References: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
Message-ID: <CAOTD34ZperVL_ZVRkoU55RRrEmCxp0AKSPEYa9jbqArzdaF6bA@mail.gmail.com>

On Mon, Aug 6, 2018 at 8:11 PM Eddie Elizondo <eelizondo at fb.com> wrote:
>
> Background:
>
> Through the implementation of an alternate runtime I've been poking around some of the class initialization routines and I found out that there was a subtle bug with PyType_Ready and the header initializer PyVarObject_HEAD_INIT.
>
>
>
> Looking through the codebase, I couldn't really find any pattern of when the type should be defined within PyVarObject_HEAD_INIT. Sometimes it was initialized to NULL (or 0) and PyType_Type (let's ignore Py_True and Py_False from now).
>
>
>
> From PyType_Ready it turns out that setting the value PyType_Type is never actually needed outside of PyType_Type and PyBaseObject type. This is clear from the code:
>
> if (Py_TYPE(type) == NULL && base != NULL)
>
>     Py_TYPE(type) = Py_TYPE(base);
>
>
>
> Given that any PyTypeObject's base is of type PyType_Type, setting PyVarObject_HEAD_INIT(&PyType_Ready) is superfluous. Therefore, setting all static PyTypeObjects to their ob_type to NULL should be a safe assumption to make.
>
>
>
> Uninitialized Types:
>
> A quick s/PyVarObject_HEAD_INIT(&PyType_Type/PyVarObject_HEAD_INIT(NULL/  shows that some objects do need to have their ob_type set from the outset, violating the previous assumption. After writing a quick script, I found that out of the ~300 PyVarObject_HEAD_INIT present in CPython, only these 14 types segfaulted:
>
>
>
> PyByteArrayIter_Type
>
> PyBytesIter_Type
>
> PyDictIterKey_Type
>
> PyDictIterValue_Type
>
> PyDictIterItem_Type
>
> PyClassMethod_Type
>
> PyAsyncGen_Type
>
> PyListIter_Type
>
> PyListRevIter_Type
>
> PyODictIter_Type
>
> PyLongRangeIter_Type
>
> PySetIter_Type
>
> PyTupleIter_Type
>
> PyUnicodeIter_Type
>
>
>
>
>
> Bug:
>
> It turns out that these PyTypeObjects are never initialized through PyType_Ready. However, they are used as fully initialized types. It is by pure chance that the work without having to call the initializer on them. This though is undefined behavior. This not only might result in a weird future bug which is hard to chase down but also, it affects other runtimes as this behavior depends on implementation details of CPython.
>
>
>
> This is a pervasive pattern that should be removed from the codebase and ideally extensions should follow as well.
>
>
>
> Solution:
>
> Here are my proposed solutions in order from less controversial to most controversial. Note that all of them I already tried in a local branch and are working:
>
>
>
> 1) Initialize all uninitialized types.
>
>
>
> Example: https://github.com/eduardo-elizondo/cpython/commit/bc53db3cf4e5a6923b0b1afa6181305553faf173
>
> 2) Move all PyVarObject_HEAD_INIT to NULL except PyType_Type, PyBaseObject_Type and the booleans.
>
>
>
> 3) Special case the initialization of PyType_Type and PyBaseObject_Type within PyType_Ready to now make all calls to PyVarObject_HEAD_INIT use NULL. To enable this a small change within PyType_Ready is needed to initialize PyType_Type PyBaseObject:

Coincidentally, I just wrote a long-ish blog post explaining in
technical details why PyVarObject_HEAD_INIT(&PyType_Type) pretty much
cannot work, at least for extension modules (it is not a problem in
the core library), on Windows (my post was focused on Cygwin but it is
a problem for Windows in general):
http://iguananaut.net/blog/programming/windows-data-import.html

The TL;DR is that it's not possible on Windows to initialize a struct
with a pointer to some other data that's found in another DLL (i.e.
&PyType_Type), unless it happens to be a function, as a special case.
But &PyType_Type obviously is not, so thinks break.

So I'm +1 for requiring passing NULL to PyVarObject_HEAD_INIT,
requiring PyType_Ready with an explicit base type argument, and maybe
(eventually) making PyVarObject_HEAD_INIT argumentless.

From poornimad2811 at gmail.com  Thu Aug  9 08:45:49 2018
From: poornimad2811 at gmail.com (Poornima .D.)
Date: Thu, 9 Aug 2018 18:15:49 +0530
Subject: [Python-Dev] Problem in importing python packages under python 3.6
 environment
Message-ID: <CAA4oqkNofzF5X2U67=YJ5j68qtv_vtK+U_h59=F3=8TXfiYQSA@mail.gmail.com>

Hi All,


I have limited knowledge on python development.  I am trying to write a
test application which needs to import from many packages across
mutliple directories.


I tried using an environment variable and appending to sys.path variable so
that import Class methods works.


I am trying to replace the above logic by below syntax.


from directory.fileName import ClassName


But this syntax does not work.  Please let me know any solution for this
issue.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180809/ebe1bc2c/attachment.html>

From brett at python.org  Thu Aug  9 12:18:16 2018
From: brett at python.org (Brett Cannon)
Date: Thu, 9 Aug 2018 09:18:16 -0700
Subject: [Python-Dev] Problem in importing python packages under python
 3.6 environment
In-Reply-To: <CAA4oqkNofzF5X2U67=YJ5j68qtv_vtK+U_h59=F3=8TXfiYQSA@mail.gmail.com>
References: <CAA4oqkNofzF5X2U67=YJ5j68qtv_vtK+U_h59=F3=8TXfiYQSA@mail.gmail.com>
Message-ID: <CAP1=2W4s0fERXt6bVWQdr=fkiAw-Y4b4vdnCJ02CXR7FUYqQSw@mail.gmail.com>

This mailing list is for the discussion of the development *of* Python, not
its use. Your best bet for help is either a site like Stack Overflow or
another mailing list like python-tutor or python-list.

On Thu, 9 Aug 2018 at 08:30 Poornima .D. <poornimad2811 at gmail.com> wrote:

> Hi All,
>
>
> I have limited knowledge on python development.  I am trying to write a
> test application which needs to import from many packages across
> mutliple directories.
>
>
> I tried using an environment variable and appending to sys.path variable
> so that import Class methods works.
>
>
> I am trying to replace the above logic by below syntax.
>
>
> from directory.fileName import ClassName
>
>
> But this syntax does not work.  Please let me know any solution for this
> issue.
>
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180809/6d7574fe/attachment.html>

From phd at phdru.name  Thu Aug  9 12:21:11 2018
From: phd at phdru.name (Oleg Broytman)
Date: Thu, 9 Aug 2018 18:21:11 +0200
Subject: [Python-Dev] Problem in importing python packages under python
 3.6 environment
In-Reply-To: <CAA4oqkNofzF5X2U67=YJ5j68qtv_vtK+U_h59=F3=8TXfiYQSA@mail.gmail.com>
References: <CAA4oqkNofzF5X2U67=YJ5j68qtv_vtK+U_h59=F3=8TXfiYQSA@mail.gmail.com>
Message-ID: <20180809162111.drnft4cfy72sp6ly@phdru.name>

Hello.

   This mailing list is to work on developing Python (adding new
features to Python itself and fixing bugs); if you're having problems
learning, understanding or using Python, please find another forum.
Probably python-list/comp.lang.python mailing list/news group is the
best place; there are Python developers who participate in it; you may
get a faster, and probably more complete, answer there. See
https://www.python.org/community/ for other lists/news groups/fora. Thank
you for understanding.

On Thu, Aug 09, 2018 at 06:15:49PM +0530, "Poornima .D." <poornimad2811 at gmail.com> wrote:
> Hi All,
> 
> 
> I have limited knowledge on python development.  I am trying to write a
> test application which needs to import from many packages across
> mutliple directories.
> 
> 
> I tried using an environment variable and appending to sys.path variable so
> that import Class methods works.
> 
> 
> I am trying to replace the above logic by below syntax.
> 
> 
> from directory.fileName import ClassName
> 
> 
> But this syntax does not work.  Please let me know any solution for this
> issue.

   Not enough information to answer. To ask such a question you'd better
prepare a simple exmaple that doesn't work -- just a few small files and
directories.

   Learn about Python modules/packagaes at
https://docs.python.org/3/tutorial/modules.html especially paying
attention to The Module Search Path:
https://docs.python.org/3/tutorial/modules.html#the-module-search-path

Oleg.
-- 
    Oleg Broytman            https://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From mariatta.wijaya at gmail.com  Thu Aug  9 12:21:43 2018
From: mariatta.wijaya at gmail.com (Mariatta Wijaya)
Date: Thu, 9 Aug 2018 09:21:43 -0700
Subject: [Python-Dev] Problem in importing python packages under python
 3.6 environment
In-Reply-To: <CAA4oqkNofzF5X2U67=YJ5j68qtv_vtK+U_h59=F3=8TXfiYQSA@mail.gmail.com>
References: <CAA4oqkNofzF5X2U67=YJ5j68qtv_vtK+U_h59=F3=8TXfiYQSA@mail.gmail.com>
Message-ID: <CAGbohnYriZi6xAY-aFv1somS0=_OOOHSYM1f7QJERnkoLKroTA@mail.gmail.com>

Hi Poornima,

Your question is more appropriate for the python-list mailing list (
https://mail.python.org/mailman/listinfo/python-list) or python-tutors
mailing list (https://mail.python.org/mailman/listinfo/tutor).



On Thu, Aug 9, 2018, 8:30 AM Poornima .D. <poornimad2811 at gmail.com> wrote:

> Hi All,
>
>
> I have limited knowledge on python development.  I am trying to write a
> test application which needs to import from many packages across
> mutliple directories.
>
>
> I tried using an environment variable and appending to sys.path variable
> so that import Class methods works.
>
>
> I am trying to replace the above logic by below syntax.
>
>
> from directory.fileName import ClassName
>
>
> But this syntax does not work.  Please let me know any solution for this
> issue.
>
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mariatta.wijaya%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180809/79fa9d86/attachment.html>

From stefan_ml at behnel.de  Thu Aug  9 12:59:49 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 9 Aug 2018 18:59:49 +0200
Subject: [Python-Dev] Use of Cython
In-Reply-To: <slrnpmlid2.biq.mcepl@milic.suse.cz>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
 <2B17179F-29B4-40E2-824D-749359E33089@mac.com>
 <pk9oev$eh3$1@blaine.gmane.org> <slrnpmlid2.biq.mcepl@milic.suse.cz>
Message-ID: <pkhrq8$u9n$1@blaine.gmane.org>

Hi,

this is getting a bit out of scope for this list. I propose to move further
questions about general Cython usage to he cython-users mailing list.

Mat?j Cepl schrieb am 08.08.2018 um 12:44:
> On 2018-08-06, 15:13 GMT, Stefan Behnel wrote:
>> Not sure I understand this correctly, but I think we're on the 
>> same page here: writing test code in C is cumbersome, writing 
>> test code in a mix of C and Python across different files is 
>> aweful. And making it difficult to write or even just review 
>> test code simply means that people will either waste their 
>> precious contribution time on it, or try to get around it.
> 
> I was thinking about the same when porting M2Crypto to py3k 
> (M2Crypto is currently swig-based mix of C-code and Python). Is 
> it even possible to make a mix of Cython, swig-based C, and 
> Python?

As long as you take the decision at a per-module basis, sure. If you want
to mix them inside of a single module, then it's either Cython+C or Swig+C,
not all three. But as Antoine suggested, unless you really want an
identical mapper for whole range of different languages, Swig is likely not
what you should use these days.


> In the end I rather stayed with plain C, because the 
> combination seems unimaginably awful.

Probably worth expanding your imagination. :)

Stefan


From steve.dower at python.org  Thu Aug  9 13:21:47 2018
From: steve.dower at python.org (Steve Dower)
Date: Thu, 9 Aug 2018 10:21:47 -0700
Subject: [Python-Dev] A Subtle Bug in Class Initializations
In-Reply-To: <CAOTD34ZperVL_ZVRkoU55RRrEmCxp0AKSPEYa9jbqArzdaF6bA@mail.gmail.com>
References: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
 <CAOTD34ZperVL_ZVRkoU55RRrEmCxp0AKSPEYa9jbqArzdaF6bA@mail.gmail.com>
Message-ID: <1422780e-f11c-5a21-88b8-db17c1e78aec@python.org>

On 09Aug2018 0818, Erik Bray wrote:
> On Mon, Aug 6, 2018 at 8:11 PM Eddie Elizondo <eelizondo at fb.com> wrote:
>> 3) Special case the initialization of PyType_Type and PyBaseObject_Type within PyType_Ready to now make all calls to PyVarObject_HEAD_INIT use NULL. To enable this a small change within PyType_Ready is needed to initialize PyType_Type PyBaseObject:
> 
> Coincidentally, I just wrote a long-ish blog post explaining in
> technical details why PyVarObject_HEAD_INIT(&PyType_Type) pretty much
> cannot work, at least for extension modules (it is not a problem in
> the core library), on Windows (my post was focused on Cygwin but it is
> a problem for Windows in general):
> http://iguananaut.net/blog/programming/windows-data-import.html
> 
> The TL;DR is that it's not possible on Windows to initialize a struct
> with a pointer to some other data that's found in another DLL (i.e.
> &PyType_Type), unless it happens to be a function, as a special case.
> But &PyType_Type obviously is not, so thinks break.

Great write-up! I think logically it should make sense that you cannot 
initialize a static value from a dynamically-linked library, but you've 
conclusively shown why that's the case. I'm not clear whether it's also 
the case on other OS's, but I don't see why it wouldn't be (unless they 
compile magic load-time resolution).

> So I'm +1 for requiring passing NULL to PyVarObject_HEAD_INIT,
> requiring PyType_Ready with an explicit base type argument, and maybe
> (eventually) making PyVarObject_HEAD_INIT argumentless.

Since PyVarObject_HEAD_INIT currently requires PyType_Ready() in 
extension modules already, then don't we just need to fix the built-in 
types?

As far as the "eventually" case, I'd hope that eventually extension 
modules are all using PyType_FromSpec() :)

Cheers,
Steve

From tjreedy at udel.edu  Thu Aug  9 13:46:55 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 9 Aug 2018 13:46:55 -0400
Subject: [Python-Dev] Problem in importing python packages under python
 3.6 environment
In-Reply-To: <CAA4oqkNofzF5X2U67=YJ5j68qtv_vtK+U_h59=F3=8TXfiYQSA@mail.gmail.com>
References: <CAA4oqkNofzF5X2U67=YJ5j68qtv_vtK+U_h59=F3=8TXfiYQSA@mail.gmail.com>
Message-ID: <pkhuid$6so$1@blaine.gmane.org>

On 8/9/2018 8:45 AM, Poornima .D. wrote:

> I have limited knowledge on python development.? I am trying to write a 
> test application which needs to import from many packages across  
> mutliple?directories.
> 
> I tried using an?environment variable and appending to sys.path variable 
> so that import Class methods works.
> 
> I am trying to replace the above logic by below syntax.
> 
> from directory.fileName import ClassName
> 
> But this syntax does not work.? Please let me know any solution for this 
> issue.

When you post elsewhere, you must supply more information:
* Copy and paste a minimal example of code that fails, ideally just one 
line.  If your example involves importing code you wrote, add the 
directory structure of your code.
* Copy and paste the actually traceback and error message.


-- 
Terry Jan Reedy



From stefan_ml at behnel.de  Fri Aug 10 05:21:19 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Aug 2018 11:21:19 +0200
Subject: [Python-Dev] Can we split PEP 489 (extension module init) ?
Message-ID: <pkjlac$1g7$1@blaine.gmane.org>

Hi,

coming back to PEP 489 [1], the multi-phase extension module
initialization. We originally designed it as an "all or nothing" feature,
but as it turns out, the "all" part is so difficult to achieve that most
potential users end up with "nothing". So, my question is: could we split
it up so that projects can get at least the main advantages: module spec
and unicode module naming.

PEP 489 is a great protocol in the sense that it allows extension modules
to set themselves up in the same way that Python modules do: load, create
module, execute module code. Without it, creating the module and executing
its code are a single step that is outside of the control of CPython, which
prevents the module from knowing its metadata and CPython from knowing
up-front what the module will actually be.

Now, the problem with PEP 489 is that it requires support for reloading and
subinterpreters at the same time [2]. For this, extension modules must
essentially be free of static global state, which comprises both the module
code itself and any external native libraries that it uses. That is
somewhere between difficult and impossible to achieve. PEP 573 [3] explains
some of the reasons, and lists solutions for some of the issues, but cannot
solve the general problem that some extension modules simply cannot get rid
of their global state, and are therefore inherently incompatible with
reloading and subinterpreters.

I would like the requirement in [2] to be lifted in PEP 489, to make the
main features of the PEP generally available to all extension modules.

The question is then how to opt out of the subinterpreter support. The PEP
explicitly does not allow backporting new init slot functions/feeatures:

"Unknown slot IDs will cause the import to fail with SystemError."

But at least changing this in Py3.8 should be doable and would be really nice.

What do you think?

Stefan



[1] https://www.python.org/dev/peps/pep-0489/
[2]
https://www.python.org/dev/peps/pep-0489/#subinterpreters-and-interpreter-reloading
[3] https://www.python.org/dev/peps/pep-0573/


From encukou at gmail.com  Fri Aug 10 05:51:32 2018
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 10 Aug 2018 11:51:32 +0200
Subject: [Python-Dev] Can we split PEP 489 (extension module init) ?
In-Reply-To: <pkjlac$1g7$1@blaine.gmane.org>
References: <pkjlac$1g7$1@blaine.gmane.org>
Message-ID: <d1cca82b-1db7-cca7-c67e-f68f8aee04f2@gmail.com>

On 08/10/18 11:21, Stefan Behnel wrote:
> Hi,
> 
> coming back to PEP 489 [1], the multi-phase extension module
> initialization. We originally designed it as an "all or nothing" feature,
> but as it turns out, the "all" part is so difficult to achieve that most
> potential users end up with "nothing". So, my question is: could we split
> it up so that projects can get at least the main advantages: module spec
> and unicode module naming.
> 
> PEP 489 is a great protocol in the sense that it allows extension modules
> to set themselves up in the same way that Python modules do: load, create
> module, execute module code. Without it, creating the module and executing
> its code are a single step that is outside of the control of CPython, which
> prevents the module from knowing its metadata and CPython from knowing
> up-front what the module will actually be.
> 
> Now, the problem with PEP 489 is that it requires support for reloading and
> subinterpreters at the same time [2]. For this, extension modules must
> essentially be free of static global state, which comprises both the module
> code itself and any external native libraries that it uses. That is
> somewhere between difficult and impossible to achieve. PEP 573 [3] explains
> some of the reasons, and lists solutions for some of the issues, but cannot
> solve the general problem that some extension modules simply cannot get rid
> of their global state, and are therefore inherently incompatible with
> reloading and subinterpreters.

Are there any issues that aren't explained in PEP 573?
I don't think Python modules should be *inherently* incompatible with 
subinterpreters. Static global state is perhaps unavoidable in some 
cases, but IMO it should be managed when it's exposed to Python.
If there are issues not in the PEPs, I'd like to collect the concrete 
cases in some document.

> I would like the requirement in [2] to be lifted in PEP 489, to make the
> main features of the PEP generally available to all extension modules.
> 
> The question is then how to opt out of the subinterpreter support. The PEP
> explicitly does not allow backporting new init slot functions/feeatures:
> 
> "Unknown slot IDs will cause the import to fail with SystemError."
> 
> But at least changing this in Py3.8 should be doable and would be really nice.

I don't think we can just silently skip unknown slots -- that would mean 
modules wouldn't be getting features they asked for.
Do you have some more sophisticated model for slots in mind, or is this 
something to be designed?


> What do you think?
> 
> Stefan
> 
> 
> 
> [1] https://www.python.org/dev/peps/pep-0489/
> [2]
> https://www.python.org/dev/peps/pep-0489/#subinterpreters-and-interpreter-reloading
> [3] https://www.python.org/dev/peps/pep-0573/


From stefan_ml at behnel.de  Fri Aug 10 06:21:28 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Aug 2018 12:21:28 +0200
Subject: [Python-Dev] Can we split PEP 489 (extension module init) ?
In-Reply-To: <d1cca82b-1db7-cca7-c67e-f68f8aee04f2@gmail.com>
References: <pkjlac$1g7$1@blaine.gmane.org>
 <d1cca82b-1db7-cca7-c67e-f68f8aee04f2@gmail.com>
Message-ID: <pkjor5$vd$1@blaine.gmane.org>

Petr Viktorin schrieb am 10.08.2018 um 11:51:
> On 08/10/18 11:21, Stefan Behnel wrote:
>> coming back to PEP 489 [1], the multi-phase extension module
>> initialization. We originally designed it as an "all or nothing" feature,
>> but as it turns out, the "all" part is so difficult to achieve that most
>> potential users end up with "nothing". So, my question is: could we split
>> it up so that projects can get at least the main advantages: module spec
>> and unicode module naming.
>>
>> PEP 489 is a great protocol in the sense that it allows extension modules
>> to set themselves up in the same way that Python modules do: load, create
>> module, execute module code. Without it, creating the module and executing
>> its code are a single step that is outside of the control of CPython, which
>> prevents the module from knowing its metadata and CPython from knowing
>> up-front what the module will actually be.
>>
>> Now, the problem with PEP 489 is that it requires support for reloading and
>> subinterpreters at the same time [2]. For this, extension modules must
>> essentially be free of static global state, which comprises both the module
>> code itself and any external native libraries that it uses. That is
>> somewhere between difficult and impossible to achieve. PEP 573 [3] explains
>> some of the reasons, and lists solutions for some of the issues, but cannot
>> solve the general problem that some extension modules simply cannot get rid
>> of their global state, and are therefore inherently incompatible with
>> reloading and subinterpreters.
> 
> Are there any issues that aren't explained in PEP 573?
> I don't think Python modules should be *inherently* incompatible with
> subinterpreters. Static global state is perhaps unavoidable in some cases,
> but IMO it should be managed when it's exposed to Python.
> If there are issues not in the PEPs, I'd like to collect the concrete cases
> in some document.

There's always the case where an external native library simply isn't
re-entrant and/or requires configuration to be global. I know, there's
static linking and there are even ways to load an external shared library
multiple times, but that's just adding to the difficulties. Let's just
accept that some things are not easy enough to make for a good requirement.


>> I would like the requirement in [2] to be lifted in PEP 489, to make the
>> main features of the PEP generally available to all extension modules.
>>
>> The question is then how to opt out of the subinterpreter support. The PEP
>> explicitly does not allow backporting new init slot functions/feeatures:
>>
>> "Unknown slot IDs will cause the import to fail with SystemError."
>>
>> But at least changing this in Py3.8 should be doable and would be really
>> nice.
> 
> I don't think we can just silently skip unknown slots -- that would mean
> modules wouldn't be getting features they asked for.
> Do you have some more sophisticated model for slots in mind, or is this
> something to be designed?

Sorry for not being clear here. I was asking for changing the assumptions
that PEP 489 makes about modules that claim to support the multi-step
initialisation part of the PEP. Adding a new (flag?) slot was just one idea
for opting out of multi-initialisation support.

Stefan


From erik.m.bray at gmail.com  Fri Aug 10 06:54:05 2018
From: erik.m.bray at gmail.com (Erik Bray)
Date: Fri, 10 Aug 2018 12:54:05 +0200
Subject: [Python-Dev] A Subtle Bug in Class Initializations
In-Reply-To: <1422780e-f11c-5a21-88b8-db17c1e78aec@python.org>
References: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
 <CAOTD34ZperVL_ZVRkoU55RRrEmCxp0AKSPEYa9jbqArzdaF6bA@mail.gmail.com>
 <1422780e-f11c-5a21-88b8-db17c1e78aec@python.org>
Message-ID: <CAOTD34Zwu=LU4b9L-We4X2hkds=UQ5Xs53vjK6_7m9TQFcWD=w@mail.gmail.com>

On Thu, Aug 9, 2018 at 7:21 PM Steve Dower <steve.dower at python.org> wrote:
>
> On 09Aug2018 0818, Erik Bray wrote:
> > On Mon, Aug 6, 2018 at 8:11 PM Eddie Elizondo <eelizondo at fb.com> wrote:
> >> 3) Special case the initialization of PyType_Type and PyBaseObject_Type within PyType_Ready to now make all calls to PyVarObject_HEAD_INIT use NULL. To enable this a small change within PyType_Ready is needed to initialize PyType_Type PyBaseObject:
> >
> > Coincidentally, I just wrote a long-ish blog post explaining in
> > technical details why PyVarObject_HEAD_INIT(&PyType_Type) pretty much
> > cannot work, at least for extension modules (it is not a problem in
> > the core library), on Windows (my post was focused on Cygwin but it is
> > a problem for Windows in general):
> > http://iguananaut.net/blog/programming/windows-data-import.html
> >
> > The TL;DR is that it's not possible on Windows to initialize a struct
> > with a pointer to some other data that's found in another DLL (i.e.
> > &PyType_Type), unless it happens to be a function, as a special case.
> > But &PyType_Type obviously is not, so thinks break.
>
> Great write-up! I think logically it should make sense that you cannot
> initialize a static value from a dynamically-linked library, but you've
> conclusively shown why that's the case. I'm not clear whether it's also
> the case on other OS's, but I don't see why it wouldn't be (unless they
> compile magic load-time resolution).

Thanks!  I'm not sure what you mean by "on other OS's" though.  Do you
mean other OS's that happen to use Windows-style PE/COFF binaries?
Because other than Windows I'm not sure what we care about there.

For ELF binaries, at least on Linux (and probably elsewhere) it the
runtime loader can perform more sophisticated relocations when loading
a binary into memory, including relocating pointers in the binary's
.data section.  This allows it to initialize data in one executable
"A" with pointers to data in another library "B" *before* "A" is
considered fully loaded and executable.

So this problem never arises, at least on Linux.

> > So I'm +1 for requiring passing NULL to PyVarObject_HEAD_INIT,
> > requiring PyType_Ready with an explicit base type argument, and maybe
> > (eventually) making PyVarObject_HEAD_INIT argumentless.
>
> Since PyVarObject_HEAD_INIT currently requires PyType_Ready() in
> extension modules already, then don't we just need to fix the built-in
> types?
>
> As far as the "eventually" case, I'd hope that eventually extension
> modules are all using PyType_FromSpec() :)

+1 :)

From encukou at gmail.com  Fri Aug 10 07:48:56 2018
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 10 Aug 2018 13:48:56 +0200
Subject: [Python-Dev] Can we split PEP 489 (extension module init) ?
In-Reply-To: <pkjor5$vd$1@blaine.gmane.org>
References: <pkjlac$1g7$1@blaine.gmane.org>
 <d1cca82b-1db7-cca7-c67e-f68f8aee04f2@gmail.com>
 <pkjor5$vd$1@blaine.gmane.org>
Message-ID: <f3cfc548-a60d-b3da-5e25-d9000dfd5638@gmail.com>

On 08/10/18 12:21, Stefan Behnel wrote:
> Petr Viktorin schrieb am 10.08.2018 um 11:51:
>> On 08/10/18 11:21, Stefan Behnel wrote:
>>> coming back to PEP 489 [1], the multi-phase extension module
>>> initialization. We originally designed it as an "all or nothing" feature,
>>> but as it turns out, the "all" part is so difficult to achieve that most
>>> potential users end up with "nothing". So, my question is: could we split
>>> it up so that projects can get at least the main advantages: module spec
>>> and unicode module naming.
>>>
>>> PEP 489 is a great protocol in the sense that it allows extension modules
>>> to set themselves up in the same way that Python modules do: load, create
>>> module, execute module code. Without it, creating the module and executing
>>> its code are a single step that is outside of the control of CPython, which
>>> prevents the module from knowing its metadata and CPython from knowing
>>> up-front what the module will actually be.
>>>
>>> Now, the problem with PEP 489 is that it requires support for reloading and
>>> subinterpreters at the same time [2]. For this, extension modules must
>>> essentially be free of static global state, which comprises both the module
>>> code itself and any external native libraries that it uses. That is
>>> somewhere between difficult and impossible to achieve. PEP 573 [3] explains
>>> some of the reasons, and lists solutions for some of the issues, but cannot
>>> solve the general problem that some extension modules simply cannot get rid
>>> of their global state, and are therefore inherently incompatible with
>>> reloading and subinterpreters.
>>
>> Are there any issues that aren't explained in PEP 573?
>> I don't think Python modules should be *inherently* incompatible with
>> subinterpreters. Static global state is perhaps unavoidable in some cases,
>> but IMO it should be managed when it's exposed to Python.
>> If there are issues not in the PEPs, I'd like to collect the concrete cases
>> in some document.
> 
> There's always the case where an external native library simply isn't
> re-entrant and/or requires configuration to be global. I know, there's
> static linking and there are even ways to load an external shared library
> multiple times, but that's just adding to the difficulties. Let's just
> accept that some things are not easy enough to make for a good requirement.

For that case, I think the right thing to do is for the module to raise 
an extension when it's being initialized for the second time, or when 
the underlying library would be initialized for the second time.

"Avoid static global state" is a good rule of thumb for supporting 
subinterpreters nicely, but other strategies are possible.
If an underlying library just expects to be initialized once, and then 
work from several modules, the Python wrapper should ensure that (using 
global state, most likely). Other ways of handling things should be 
possible, depending on the underlying library.

>>> I would like the requirement in [2] to be lifted in PEP 489, to make the
>>> main features of the PEP generally available to all extension modules.
>>>
>>> The question is then how to opt out of the subinterpreter support. The PEP
>>> explicitly does not allow backporting new init slot functions/feeatures:
>>>
>>> "Unknown slot IDs will cause the import to fail with SystemError."
>>>
>>> But at least changing this in Py3.8 should be doable and would be really
>>> nice.
>>
>> I don't think we can just silently skip unknown slots -- that would mean
>> modules wouldn't be getting features they asked for.
>> Do you have some more sophisticated model for slots in mind, or is this
>> something to be designed?
> 
> Sorry for not being clear here. I was asking for changing the assumptions
> that PEP 489 makes about modules that claim to support the multi-step
> initialisation part of the PEP. Adding a new (flag?) slot was just one idea
> for opting out of multi-initialisation support.

Would this be better than a flag + raising an error on init?
One big disadvantage of a big opt-out-of-everything button is that it 
doesn't encourage people to think about what the actual non-reentrant 
piece of code is.

From status at bugs.python.org  Fri Aug 10 12:10:08 2018
From: status at bugs.python.org (Python tracker)
Date: Fri, 10 Aug 2018 18:10:08 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20180810161008.BFE435685B@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2018-08-03 - 2018-08-10)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6756 ( -3)
  closed 39367 (+44)
  total  46123 (+41)

Open issues with patches: 2685 


Issues opened (26)
==================

#34333: Path.with_suffix() raises TypeError when doing %-formatting
https://bugs.python.org/issue34333  opened by berker.peksag

#34334: QueueHandler logs exc_info twice
https://bugs.python.org/issue34334  opened by avdd

#34338: abstractmethod can run on classes
https://bugs.python.org/issue34338  opened by Michael Hooreman

#34340: mimetypes libmagic compatibility
https://bugs.python.org/issue34340  opened by bpypy

#34341: Appending to ZIP archive blows up existing Central Directory e
https://bugs.python.org/issue34341  opened by serhiy.storchaka

#34344: Fix the docstring for AbstractEventLoopPolicy.get_event_loop
https://bugs.python.org/issue34344  opened by drtyrsa

#34345: Add tests for PEP 468 and PEP 520
https://bugs.python.org/issue34345  opened by serhiy.storchaka

#34346: dir() hangs interpreter
https://bugs.python.org/issue34346  opened by sfaleron

#34347: AIX: test_utf8_mode.test_cmd_line fails
https://bugs.python.org/issue34347  opened by Michael.Felt

#34349: asyncio.wait should accept generator of tasks as first argumen
https://bugs.python.org/issue34349  opened by jnwatson

#34354: Memory leak on _testCongestion
https://bugs.python.org/issue34354  opened by Vinicius Pacheco

#34355: SIGSEGV (Address boundary error)
https://bugs.python.org/issue34355  opened by ybon

#34356: Add support for args and kwargs in logging.conf
https://bugs.python.org/issue34356  opened by xavier.hardy

#34357: situation where urllib3 works, but urllib does not work
https://bugs.python.org/issue34357  opened by deivid

#34360: urllib.parse doesn't fully comply to RFC 3986
https://bugs.python.org/issue34360  opened by The Compiler

#34362: User-created types with wrong __new__ can be instantiated
https://bugs.python.org/issue34362  opened by ppperry

#34363: dataclasses.asdict() mishandles dataclass instance attributes 
https://bugs.python.org/issue34363  opened by alexdelorenzo

#34364: problem with traceback for syntax error in f-string
https://bugs.python.org/issue34364  opened by bgailer

#34365: datetime's documentation refers to "comparison [...] falling b
https://bugs.python.org/issue34365  opened by Kevin.Norris

#34366: _uuid module fails to compile on FreeBSD when libuuid is insta
https://bugs.python.org/issue34366  opened by mgorny

#34367: AsyncResult.get() only notifies one thread
https://bugs.python.org/issue34367  opened by AlexWithBeard

#34368: ftplib __init__ function can't handle 120 or 4xy reply when co
https://bugs.python.org/issue34368  opened by H-ZeX

#34369: kqueue.control() documentation and implementation mismatch
https://bugs.python.org/issue34369  opened by a.badger

#34370: Tkinter scroll issues on macOS
https://bugs.python.org/issue34370  opened by vtudorache

#34371: File reading gets stuck if you read at eof on macos
https://bugs.python.org/issue34371  opened by sverrirab

#34372: Compiler could output more accurate line numbers
https://bugs.python.org/issue34372  opened by Arusekk



Most recent 15 issues with no replies (15)
==========================================

#34372: Compiler could output more accurate line numbers
https://bugs.python.org/issue34372

#34370: Tkinter scroll issues on macOS
https://bugs.python.org/issue34370

#34368: ftplib __init__ function can't handle 120 or 4xy reply when co
https://bugs.python.org/issue34368

#34367: AsyncResult.get() only notifies one thread
https://bugs.python.org/issue34367

#34366: _uuid module fails to compile on FreeBSD when libuuid is insta
https://bugs.python.org/issue34366

#34365: datetime's documentation refers to "comparison [...] falling b
https://bugs.python.org/issue34365

#34357: situation where urllib3 works, but urllib does not work
https://bugs.python.org/issue34357

#34356: Add support for args and kwargs in logging.conf
https://bugs.python.org/issue34356

#34354: Memory leak on _testCongestion
https://bugs.python.org/issue34354

#34345: Add tests for PEP 468 and PEP 520
https://bugs.python.org/issue34345

#34344: Fix the docstring for AbstractEventLoopPolicy.get_event_loop
https://bugs.python.org/issue34344

#34341: Appending to ZIP archive blows up existing Central Directory e
https://bugs.python.org/issue34341

#34340: mimetypes libmagic compatibility
https://bugs.python.org/issue34340

#34334: QueueHandler logs exc_info twice
https://bugs.python.org/issue34334

#34333: Path.with_suffix() raises TypeError when doing %-formatting
https://bugs.python.org/issue34333



Most recent 15 issues waiting for review (15)
=============================================

#34366: _uuid module fails to compile on FreeBSD when libuuid is insta
https://bugs.python.org/issue34366

#34341: Appending to ZIP archive blows up existing Central Directory e
https://bugs.python.org/issue34341

#34333: Path.with_suffix() raises TypeError when doing %-formatting
https://bugs.python.org/issue34333

#34331: Incorrectly pluralized abstract class error message
https://bugs.python.org/issue34331

#34322: modification to Lib/distutils/ccompiler.py to simplify handlin
https://bugs.python.org/issue34322

#34320: Creating dict from OrderedDict doesn't preserve order
https://bugs.python.org/issue34320

#34319: Clarify pathlib.Path("filepath").read_text()
https://bugs.python.org/issue34319

#34318: Convert deprecated behavior of assertRaises() etc into errors
https://bugs.python.org/issue34318

#34311: locale.format() and locale.format_string() cast Decimals to fl
https://bugs.python.org/issue34311

#34305: inspect.getsourcefile and inspect.getcomments do not work with
https://bugs.python.org/issue34305

#34303: micro-optimizations in functools.reduce()
https://bugs.python.org/issue34303

#34302: Avoid inefficient way to find start point in deque.index
https://bugs.python.org/issue34302

#34293: DOC: Makefile inherits a Sphinx 1.5 bug regarding PAPER envvar
https://bugs.python.org/issue34293

#34284: Nonsensical exception message when calling `__new__` on non-in
https://bugs.python.org/issue34284

#34282: Enum._convert shadows members named _convert
https://bugs.python.org/issue34282



Top 10 most discussed issues (10)
=================================

#32797: Tracebacks from Cython modules no longer work
https://bugs.python.org/issue32797  33 msgs

#34296: Speed up python startup by pre-warming the vm
https://bugs.python.org/issue34296  13 msgs

#34363: dataclasses.asdict() mishandles dataclass instance attributes 
https://bugs.python.org/issue34363   7 msgs

#34247: PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher
https://bugs.python.org/issue34247   6 msgs

#34284: Nonsensical exception message when calling `__new__` on non-in
https://bugs.python.org/issue34284   6 msgs

#34355: SIGSEGV (Address boundary error)
https://bugs.python.org/issue34355   5 msgs

#24255: Replace debuglevel-related logic with logging
https://bugs.python.org/issue24255   4 msgs

#32073: Add copy_directory_metadata parameter to shutil.copytree
https://bugs.python.org/issue32073   4 msgs

#34272: Reorganize C API tests
https://bugs.python.org/issue34272   4 msgs

#34319: Clarify pathlib.Path("filepath").read_text()
https://bugs.python.org/issue34319   4 msgs



Issues closed (41)
==================

#2651: Strings passed to KeyError do not round trip
https://bugs.python.org/issue2651  closed by lukasz.langa

#6952: deprecated conversion from string constant to char *
https://bugs.python.org/issue6952  closed by berker.peksag

#13574: refresh example in doc for Extending and Embedding
https://bugs.python.org/issue13574  closed by berker.peksag

#18540: imaplib.IMAP4() ends with "Name or service not known" on Fedor
https://bugs.python.org/issue18540  closed by berker.peksag

#23876: Fix mkdir() call for Watcom compilers on UNIX-like platforms
https://bugs.python.org/issue23876  closed by Jeffrey.Armstrong

#28044: Make the sidebar in the documentation follow the section autom
https://bugs.python.org/issue28044  closed by Mariatta

#29036: logging module: Add `full_module_name` to LogRecord details
https://bugs.python.org/issue29036  closed by vinay.sajip

#31047: Windows: os.path.isabs(os.path.abspath(" ")) == False
https://bugs.python.org/issue31047  closed by steve.dower

#32638: distutils test errors with AIX and xlc
https://bugs.python.org/issue32638  closed by Michael.Felt

#33161: Refactor of pathlib's _WindowsBehavior.gethomedir
https://bugs.python.org/issue33161  closed by berker.peksag

#33736: Improve the documentation of asyncio stream API
https://bugs.python.org/issue33736  closed by berker.peksag

#33839: IDLE tooltips.py: refactor and add docstrings and tests
https://bugs.python.org/issue33839  closed by taleinat

#34047: IDLE: on macOS, scroll slider 'sticks' at bottom of file
https://bugs.python.org/issue34047  closed by taleinat

#34236: Test6012 in test_capi is not run as part of make test
https://bugs.python.org/issue34236  closed by serhiy.storchaka

#34242: configparser: SectionProxy.get is silent on missing options
https://bugs.python.org/issue34242  closed by lukasz.langa

#34244: Add support of check logger
https://bugs.python.org/issue34244  closed by vinay.sajip

#34253: Tkinter- On windows, calling filedialog or messagebox before t
https://bugs.python.org/issue34253  closed by terry.reedy

#34254: Include type annotations in error messages for better errors
https://bugs.python.org/issue34254  closed by xtreak

#34270: Add names to asyncio tasks
https://bugs.python.org/issue34270  closed by yselivanov

#34273: %f is confusingly associated with fixed point format
https://bugs.python.org/issue34273  closed by terry.reedy

#34310: Build error with option "--with-pydebug" on Mac OS 10.13.6
https://bugs.python.org/issue34310  closed by matrixise

#34312: Allow str.endswith and str.startswith to accept an iterable
https://bugs.python.org/issue34312  closed by brett.cannon

#34324: Doc README wrong directory name for venv
https://bugs.python.org/issue34324  closed by Mariatta

#34325: test_zipfile gets OverflowError in multiple buildbots
https://bugs.python.org/issue34325  closed by vstinner

#34326: test_subprocess.POSIXProcessTestCase fails in AMD64 Ubuntu 3.x
https://bugs.python.org/issue34326  closed by matrixise

#34329: Document how to remove a suffix with pathlib.Path.with_suffix
https://bugs.python.org/issue34329  closed by berker.peksag

#34332: Suggestion for a new loop type
https://bugs.python.org/issue34332  closed by zach.ware

#34335: Fix examples in asyncio docs (suppliment to bpo-32258)
https://bugs.python.org/issue34335  closed by Mariatta

#34336: Don't promote possibility to omit Optional on optional/default
https://bugs.python.org/issue34336  closed by levkivskyi

#34337: Fail to get a right answer for 1.2%0.2
https://bugs.python.org/issue34337  closed by mark.dickinson

#34339: Argument unpacking syntax for lambdas
https://bugs.python.org/issue34339  closed by danijar

#34342: Fix the broken links in CPythonVmInternals wiki page
https://bugs.python.org/issue34342  closed by berker.peksag

#34343: Why is turtle still present in python embedded for Windows?
https://bugs.python.org/issue34343  closed by vtudorache

#34348: Python 3.7 - Issues Installing Scikit Learn
https://bugs.python.org/issue34348  closed by mark.dickinson

#34350: Non obvious logging handler behaviour
https://bugs.python.org/issue34350  closed by vinay.sajip

#34351: Python 3.7 - Issues Installing Scikit Learn
https://bugs.python.org/issue34351  closed by mark.dickinson

#34352: Using tailf with python3.4
https://bugs.python.org/issue34352  closed by zach.ware

#34353: stat's python implementation of filemode (fallback) doesn't be
https://bugs.python.org/issue34353  closed by benjamin.peterson

#34358: round() combined with product outputs ugly result
https://bugs.python.org/issue34358  closed by steven.daprano

#34359: Wrong virtual environment found
https://bugs.python.org/issue34359  closed by paul.moore

#34361: An error should be returned when there are spaces in between f
https://bugs.python.org/issue34361  closed by r.david.murray

From steve.dower at python.org  Fri Aug 10 12:49:32 2018
From: steve.dower at python.org (Steve Dower)
Date: Fri, 10 Aug 2018 09:49:32 -0700
Subject: [Python-Dev] A Subtle Bug in Class Initializations
In-Reply-To: <CAOTD34Zwu=LU4b9L-We4X2hkds=UQ5Xs53vjK6_7m9TQFcWD=w@mail.gmail.com>
References: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
 <CAOTD34ZperVL_ZVRkoU55RRrEmCxp0AKSPEYa9jbqArzdaF6bA@mail.gmail.com>
 <1422780e-f11c-5a21-88b8-db17c1e78aec@python.org>
 <CAOTD34Zwu=LU4b9L-We4X2hkds=UQ5Xs53vjK6_7m9TQFcWD=w@mail.gmail.com>
Message-ID: <61704013-0bfd-1af9-b87b-42ab6a30d11e@python.org>

On 10Aug2018 0354, Erik Bray wrote:
> Thanks!  I'm not sure what you mean by "on other OS's" though.  Do you
> mean other OS's that happen to use Windows-style PE/COFF binaries?
> Because other than Windows I'm not sure what we care about there.
> 
> For ELF binaries, at least on Linux (and probably elsewhere) it the
> runtime loader can perform more sophisticated relocations when loading
> a binary into memory, including relocating pointers in the binary's
> .data section.  This allows it to initialize data in one executable
> "A" with pointers to data in another library "B" *before* "A" is
> considered fully loaded and executable.
> 
> So this problem never arises, at least on Linux.

That's exactly what I meant. I simply didn't know how/whether other 
loaders handled this case :) I recognise it's nothing to do with the 
binary format and everything to do with whether the loader knows what to 
do or not.

>>> So I'm +1 for requiring passing NULL to PyVarObject_HEAD_INIT,
>>> requiring PyType_Ready with an explicit base type argument, and maybe
>>> (eventually) making PyVarObject_HEAD_INIT argumentless.
>>
>> Since PyVarObject_HEAD_INIT currently requires PyType_Ready() in
>> extension modules already, then don't we just need to fix the built-in
>> types?
>>
>> As far as the "eventually" case, I'd hope that eventually extension
>> modules are all using PyType_FromSpec() :)
> 
> +1 :)

Is that just a +1 for PyType_FromSpec(), or are you agreeing that we 
only need to fix the built-in types?

Cheers,
Steve

From armin.rigo at gmail.com  Fri Aug 10 13:15:11 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Fri, 10 Aug 2018 19:15:11 +0200
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <20180731135545.24c4d427@fsol>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <20180731085816.7bf79326@fsol>
 <CA+3bQGHq_iTmOos9GPh10cUCSH5C78888UF8BYcz24bbjgSEaQ@mail.gmail.com>
 <20180731135545.24c4d427@fsol>
Message-ID: <CAMSv6X2CSS1emJ97UprUehROqiHE2OGXEf3zsAuCNXOspF+jOg@mail.gmail.com>

Hi,

On 31 July 2018 at 13:55, Antoine Pitrou <solipsis at pitrou.net> wrote:
> It's just that I disagree that removing the C API will make CPython 2x
> faster.
>
> Actually, important modern optimizations for dynamic languages (such as
> inlining, type specialization, inline caches, object unboxing) don't
> seem to depend on the C API at all.

These are optimizations typically talked about in papers about dynamic
languages in general.  In my opinion, in the specific case of CPython,
they are all secondary to the following: (1) JIT, (2) GC, (3) object
model, (4) multithreading.

Currently, the C API only allows Psyco-style JITting (much slower than
PyPy).  All three other points might not be possible at all without a
seriously modified C API.  Why?  I have no proof, but only
circumstantial evidence.  Each of (2), (3), (4) has been done in at
least one other implementation: PyPy, Jython and IronPython.  Each of
these implementation has also got its share of troubles with emulating
the CPython C API.  You can continue to think that the C API has got
nothing to do with it.  I tend to think the opposite.  The continued
absence of major performance improvements for either CPython itself or
for any alternative Python implementation that *does* support the C
API natively is probably proof enough---I think that enough time has
passed, by now, to make this argument.


A bient?t,

Armin.

From J.Demeyer at UGent.be  Sat Aug 11 08:31:59 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Sat, 11 Aug 2018 14:31:59 +0200
Subject: [Python-Dev] Can we split PEP 489 (extension module init) ?
Message-ID: <5B6ED73F.9000508@UGent.be>

 > Would this be better than a flag + raising an error on init?

Exactly. PEP 489 only says "Extensions using the new initialization 
scheme are expected to support subinterpreters". What's wrong with 
raising an exception when the module is initialized the second time?


Jeroen.

From solipsis at pitrou.net  Sat Aug 11 09:19:40 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 11 Aug 2018 15:19:40 +0200
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <CAMSv6X2CSS1emJ97UprUehROqiHE2OGXEf3zsAuCNXOspF+jOg@mail.gmail.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <20180731085816.7bf79326@fsol>
 <CA+3bQGHq_iTmOos9GPh10cUCSH5C78888UF8BYcz24bbjgSEaQ@mail.gmail.com>
 <20180731135545.24c4d427@fsol>
 <CAMSv6X2CSS1emJ97UprUehROqiHE2OGXEf3zsAuCNXOspF+jOg@mail.gmail.com>
Message-ID: <20180811151940.7f52ba97@fsol>


Hi Armin,

On Fri, 10 Aug 2018 19:15:11 +0200
Armin Rigo <armin.rigo at gmail.com> wrote:
> Currently, the C API only allows Psyco-style JITting (much slower than
> PyPy).  All three other points might not be possible at all without a
> seriously modified C API.  Why?  I have no proof, but only
> circumstantial evidence.  Each of (2), (3), (4) has been done in at
> least one other implementation: PyPy, Jython and IronPython.  Each of
> these implementation has also got its share of troubles with emulating
> the CPython C API.  You can continue to think that the C API has got
> nothing to do with it.  I tend to think the opposite.  The continued
> absence of major performance improvements for either CPython itself or
> for any alternative Python implementation that *does* support the C
> API natively is probably proof enough---I think that enough time has
> passed, by now, to make this argument.

Jython and IronPython never got significant manpower AFAIK, so even
without being hindered by the C API, chances are they would never have
gotten very far.  Both do not even seem to have stable releases
implementing the Python 3 language...

That leaves us with CPython and PyPy, which are only two data points.
And there are enough differences, AFAIK, between those two that picking
up "supports the C API natively" as the primary factor leading to a
performance difference sounds arbitrary.

(the major difference being IMHO that PyPy is written in RPython, which
opens up possibilities that are not realistic with a C implementation,
such as the JIT being automatically able to inspect implementations of
core / stdlib primitives; in a CPython-based JIT such as Numba, you
have to reimplement all those primitives in a form that's friendly to
the JIT compiler)

Regards

Antoine.

From stefan_ml at behnel.de  Sat Aug 11 10:40:56 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 11 Aug 2018 16:40:56 +0200
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <20180811151940.7f52ba97@fsol>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <20180731085816.7bf79326@fsol>
 <CA+3bQGHq_iTmOos9GPh10cUCSH5C78888UF8BYcz24bbjgSEaQ@mail.gmail.com>
 <20180731135545.24c4d427@fsol>
 <CAMSv6X2CSS1emJ97UprUehROqiHE2OGXEf3zsAuCNXOspF+jOg@mail.gmail.com>
 <20180811151940.7f52ba97@fsol>
Message-ID: <pkmsdl$mmd$1@blaine.gmane.org>

Antoine Pitrou schrieb am 11.08.2018 um 15:19:
> On Fri, 10 Aug 2018 19:15:11 +0200 Armin Rigo wrote:
>> Currently, the C API only allows Psyco-style JITting (much slower than
>> PyPy).  All three other points might not be possible at all without a
>> seriously modified C API.  Why?  I have no proof, but only
>> circumstantial evidence.  Each of (2), (3), (4) has been done in at
>> least one other implementation: PyPy, Jython and IronPython.  Each of
>> these implementation has also got its share of troubles with emulating
>> the CPython C API.  You can continue to think that the C API has got
>> nothing to do with it.  I tend to think the opposite.  The continued
>> absence of major performance improvements for either CPython itself or
>> for any alternative Python implementation that *does* support the C
>> API natively is probably proof enough---I think that enough time has
>> passed, by now, to make this argument.
> [...]
> That leaves us with CPython and PyPy, which are only two data points.
> And there are enough differences, AFAIK, between those two that picking
> up "supports the C API natively" as the primary factor leading to a
> performance difference sounds arbitrary.

IMHO, while it's not clear to what extent the C-API hinders performance
improvements or jittability of code in CPython, I think it's fair to assume
that it's easier to improve internals when they are internal and not part
of a public API. Whether it's worth the effort to design a new C-API, or at
least make major changes to it, I cannot say, lacking an actual comparable
implementation of such a design that specifically targets better performance.

As it stands, extensions can actually make good use of the fact that the
C-API treats them (mostly, see e.g. PEPs 575/580) as first class citizens
in the CPython ecosystem. So, the status quo is at least a tradeoff.

Stefan


From stefan_ml at behnel.de  Sat Aug 11 10:47:12 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 11 Aug 2018 16:47:12 +0200
Subject: [Python-Dev] Can we split PEP 489 (extension module init) ?
In-Reply-To: <f3cfc548-a60d-b3da-5e25-d9000dfd5638@gmail.com>
References: <pkjlac$1g7$1@blaine.gmane.org>
 <d1cca82b-1db7-cca7-c67e-f68f8aee04f2@gmail.com>
 <pkjor5$vd$1@blaine.gmane.org>
 <f3cfc548-a60d-b3da-5e25-d9000dfd5638@gmail.com>
Message-ID: <pkmspd$q21$1@blaine.gmane.org>

Petr Viktorin schrieb am 10.08.2018 um 13:48:
> Would this be better than a flag + raising an error on init?

Ok, I've implemented this in Cython for now, to finally move the PEP-489
support forward. The somewhat annoying drawback is that module reloading
previously *seemed* to work, simply because it didn't actually do anything.
Now, people will get an exception in cases that previously worked silently.
An exception would probably have been better from the beginning, because it
clearly tells people that what they are trying is not supported. Now it's a
bit of a breaking change. I'll see what it gives.

Thanks for your feedback on this.

Stefan


From santiago.basulto at gmail.com  Sat Aug 11 16:08:06 2018
From: santiago.basulto at gmail.com (Santiago Basulto)
Date: Sat, 11 Aug 2018 17:08:06 -0300
Subject: [Python-Dev] Class decorators can't be pickled,
 which breaks multiprocessing and concurrent.futures. Any plans of
 improving this?
Message-ID: <CAPmap1cDGpcj5KLNdcnB_HG04QGRTBOaVkizzxVd+Xi=BRZhgQ@mail.gmail.com>

Hello folks! I'm using the `concurrent.futures.ProcessPoolExecutor` with a
couple of functions that have been decorated with a class decorator. Both
`concurrent.futures` and `multiprocessing` breaks because "the object's
can't be pickled". There's a really simple fix for this, which is just,
instead of "decorating" the function (with the @), instantiate the
decorator and use it directly.

Example. This is my (very simple, for demonstration purposes) decorator:

    class CheckOnlyIntegers:
        def __init__(self, fn):
            self.fn = fn

        def __call__(self, *args):
            if not all([type(arg) == int for arg in args]):
                raise ValueError("Invalid param is not an integer")
            return self.fn(*args)

If I define a simple `add` function and decorate it using the
`CheckOnlyIntegers` decorator:

    @CheckOnlyIntegers
    def add(x, y):
        return x + y

and try using a regular `ProcessPoolExecutor().submit(add, 2, 3)`, it fails
with:

```
Can't pickle <function add at 0x10ace3e18>: it's not the same object as
__main__.add.
```

The fix for this is simple, instead of "decorating" the function,
instantiate the class and use a different name:

    def add(x, y):
        return x + y

    add_2 = CheckOnlyIntegers(add)

In this case `ProcessPoolExecutor().submit(add_2, 2, 3)` works correctly.
(here's the full sample code
<https://gist.github.com/santiagobasulto/2bc51e7b0b0c7c2a74fd6d5748778dd7>)

I know this is an issue with the pickle module (not concurrent.futures or
multiprocessing). But are there any ways of improving this in future
versions? Not being able to pickle functions decorated with Class
Decorators seems like an unnecessary limitation.

Thanks for your feedback!

-- 
Santiago Basulto.-
Up!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180811/6e08dec1/attachment.html>

From storchaka at gmail.com  Sun Aug 12 01:33:00 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 12 Aug 2018 08:33:00 +0300
Subject: [Python-Dev] Class decorators can't be pickled,
 which breaks multiprocessing and concurrent.futures. Any plans of
 improving this?
In-Reply-To: <CAPmap1cDGpcj5KLNdcnB_HG04QGRTBOaVkizzxVd+Xi=BRZhgQ@mail.gmail.com>
References: <CAPmap1cDGpcj5KLNdcnB_HG04QGRTBOaVkizzxVd+Xi=BRZhgQ@mail.gmail.com>
Message-ID: <pkogm9$5j0$1@blaine.gmane.org>

11.08.18 23:08, Santiago Basulto ????:
> Hello folks! I'm using the `concurrent.futures.ProcessPoolExecutor` with 
> a couple of functions that have been decorated with a class decorator. 
> Both `concurrent.futures` and `multiprocessing` breaks because "the 
> object's can't be pickled". There's a really simple fix for this, which 
> is just, instead of "decorating" the function (with the @), instantiate 
> the decorator and use it directly.
> 
> Example. This is my (very simple, for demonstration purposes) decorator:
> 
>  ? ? class CheckOnlyIntegers:
>  ? ? ? ? def __init__(self, fn):
>  ? ? ? ? ? ? self.fn = fn
> 
>  ? ? ? ? def __call__(self, *args):
>  ? ? ? ? ? ? if not all([type(arg) == int for arg in args]):
>  ? ? ? ? ? ? ? ? raise ValueError("Invalid param is not an integer")
>  ? ? ? ? ? ? return self.fn(*args)
> 
> If I define a simple `add` function and decorate it using the 
> `CheckOnlyIntegers` decorator:
> 
>  ? ? @CheckOnlyIntegers
>  ? ? def add(x, y):
>  ? ? ? ? return x + y
> 
> and try using a regular `ProcessPoolExecutor().submit(add, 2, 3)`, it 
> fails with:
> 
> ```
> Can't pickle <function add at 0x10ace3e18>: it's not the same object as 
> __main__.add.
> ```

By default instances of Python classes are pickled by pickling a set of 
their attributes. Functions are pickled by name. But since the name of 
self.fn corresponds to the decorated function, not the function itself, 
it can't be pickled.

You can implement the explicit pickle support for your decorator that 
bypass this limitation.

     def __reduce__(self):
         return self.fn.__qualname__

Now the decorated function will be pickled by name.

It may help to set also self.__module__ = self.fn.__module__ in the 
constructor.


From armin.rigo at gmail.com  Sun Aug 12 04:03:09 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sun, 12 Aug 2018 10:03:09 +0200
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <20180811151940.7f52ba97@fsol>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <20180731085816.7bf79326@fsol>
 <CA+3bQGHq_iTmOos9GPh10cUCSH5C78888UF8BYcz24bbjgSEaQ@mail.gmail.com>
 <20180731135545.24c4d427@fsol>
 <CAMSv6X2CSS1emJ97UprUehROqiHE2OGXEf3zsAuCNXOspF+jOg@mail.gmail.com>
 <20180811151940.7f52ba97@fsol>
Message-ID: <CAMSv6X1d_WoA-kYNRTV8+yEpjrtL1kH8P8xH8KYAfUt-WOFSgA@mail.gmail.com>

Hi Antoine,

On 11 August 2018 at 15:19, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Jython and IronPython never got significant manpower AFAIK, so even
> without being hindered by the C API, chances are they would never have
> gotten very far.  Both do not even seem to have stable releases
> implementing the Python 3 language...

I included IronPython and Jython because they are also rather complete
implementations of (some version of) Python that are actively used in
some contexts.  During the past 20 years, these two and PyPy are the
only generally-useful rather-complete alternate Python
implementations, and they each improve on some of the pain points (2)
(3) (4) hitting CPython.  Neither of them supports the C API
efficiently.  Whatever you argue, my opinion is that they got where
they are by first completely ignoring the C API.  Even Pyston did
that.

About its C API, CPython can continue to prefer the status quo.  I
tend to think that it's exactly what will occur, so I'm staying away
from capi-sig.


A bient?t,

Armin.

From larry at hastings.org  Mon Aug 13 05:49:17 2018
From: larry at hastings.org (Larry Hastings)
Date: Mon, 13 Aug 2018 02:49:17 -0700
Subject: [Python-Dev] Winding down 3.4
Message-ID: <55df745c-bf49-9849-0e64-b54b797ca0f1@hastings.org>



We of the core dev community commit to supporting Python releases for 
five years.? Releases get eighteen months of active bug fixes, followed 
by three and a half years of security fixes.? Python 3.4 turns 5 next 
March--at which point we'll stop supporting it, and I'll retire as 3.4 
release manager.

My plan is to make one final release on or around its fifth birthday 
containing the last round of security fixes.? That's about seven months 
from now.? Nothing has been merged since the releases of 3.4.9 and 3.5.6 
last week, and there are no open PRs against either of those releases.

But!? There are still a couple languishing "critical" bugs:

    "shutil copy* unsafe on POSIX - they preserve setuid/setgit bits"
    https://bugs.python.org/issue17180

    "XML vulnerabilities in Python"
    https://bugs.python.org/issue17239

    "fflush called on pointer to potentially closed file" (Windows only)
    https://bugs.python.org/issue19050

It'd be nice to resolve all those issues, one way or another, before we 
retire 3.4.


See you next March,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180813/d8c842c3/attachment.html>

From antoine at python.org  Mon Aug 13 05:55:32 2018
From: antoine at python.org (Antoine Pitrou)
Date: Mon, 13 Aug 2018 11:55:32 +0200
Subject: [Python-Dev] [python-committers] Winding down 3.4
In-Reply-To: <55df745c-bf49-9849-0e64-b54b797ca0f1@hastings.org>
References: <55df745c-bf49-9849-0e64-b54b797ca0f1@hastings.org>
Message-ID: <f5d4fa80-7cfa-08e1-d2c2-96d015be99dc@python.org>


Le 13/08/2018 ? 11:49, Larry Hastings a ?crit?:
> 
> 
> We of the core dev community commit to supporting Python releases for
> five years.? Releases get eighteen months of active bug fixes, followed
> by three and a half years of security fixes.? Python 3.4 turns 5 next
> March--at which point we'll stop supporting it, and I'll retire as 3.4
> release manager.
> 
> My plan is to make one final release on or around its fifth birthday
> containing the last round of security fixes.? That's about seven months
> from now.? Nothing has been merged since the releases of 3.4.9 and 3.5.6
> last week, and there are no open PRs against either of those releases.
> 
> But!? There are still a couple languishing "critical" bugs:
> 
>     "shutil copy* unsafe on POSIX - they preserve setuid/setgit bits"
>     https://bugs.python.org/issue17180
> 
>     "XML vulnerabilities in Python"
>     https://bugs.python.org/issue17239
> 
>     "fflush called on pointer to potentially closed file" (Windows only)
>     https://bugs.python.org/issue19050
> 
> It'd be nice to resolve all those issues, one way or another, before we
> retire 3.4.

So that 3.4 dies in good health?

Regards

Antoine.

From erik.m.bray at gmail.com  Mon Aug 13 07:01:31 2018
From: erik.m.bray at gmail.com (Erik Bray)
Date: Mon, 13 Aug 2018 13:01:31 +0200
Subject: [Python-Dev] A Subtle Bug in Class Initializations
In-Reply-To: <61704013-0bfd-1af9-b87b-42ab6a30d11e@python.org>
References: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
 <CAOTD34ZperVL_ZVRkoU55RRrEmCxp0AKSPEYa9jbqArzdaF6bA@mail.gmail.com>
 <1422780e-f11c-5a21-88b8-db17c1e78aec@python.org>
 <CAOTD34Zwu=LU4b9L-We4X2hkds=UQ5Xs53vjK6_7m9TQFcWD=w@mail.gmail.com>
 <61704013-0bfd-1af9-b87b-42ab6a30d11e@python.org>
Message-ID: <CAOTD34b_w1f46FR4hJf7P1kU7TVWOHy-9qV0sLu3VkELCK09rw@mail.gmail.com>

On Fri, Aug 10, 2018 at 6:49 PM Steve Dower <steve.dower at python.org> wrote:
>
> On 10Aug2018 0354, Erik Bray wrote:
> > Thanks!  I'm not sure what you mean by "on other OS's" though.  Do you
> > mean other OS's that happen to use Windows-style PE/COFF binaries?
> > Because other than Windows I'm not sure what we care about there.
> >
> > For ELF binaries, at least on Linux (and probably elsewhere) it the
> > runtime loader can perform more sophisticated relocations when loading
> > a binary into memory, including relocating pointers in the binary's
> > .data section.  This allows it to initialize data in one executable
> > "A" with pointers to data in another library "B" *before* "A" is
> > considered fully loaded and executable.
> >
> > So this problem never arises, at least on Linux.
>
> That's exactly what I meant. I simply didn't know how/whether other
> loaders handled this case :) I recognise it's nothing to do with the
> binary format and everything to do with whether the loader knows what to
> do or not.

Ah, that's not exactly what *I* meant, but you are also right: In
principle it shouldn't have anything to do with the binary formation.
You could stuff a more sophisticated dynamic relocation section into a
PE/COFF binary too but Windows wouldn't know what to do with it.

So you're right that this kind of problem could affect other OSes, I
just have no idea if it does.

> >>> So I'm +1 for requiring passing NULL to PyVarObject_HEAD_INIT,
> >>> requiring PyType_Ready with an explicit base type argument, and maybe
> >>> (eventually) making PyVarObject_HEAD_INIT argumentless.
> >>
> >> Since PyVarObject_HEAD_INIT currently requires PyType_Ready() in
> >> extension modules already, then don't we just need to fix the built-in
> >> types?
> >>
> >> As far as the "eventually" case, I'd hope that eventually extension
> >> modules are all using PyType_FromSpec() :)
> >
> > +1 :)
>
> Is that just a +1 for PyType_FromSpec(), or are you agreeing that we
> only need to fix the built-in types?

Both! I think we should fix the built-in types, but it would be nice
if more extension modules used PyType_FromSpec, if nothing else
because I find it much more readable, and (I'm guessing) it's better
from the perspective of Victor's C-API work.  But I admit I'm not
fully versed in the downsides (if there are any).

From ncoghlan at gmail.com  Mon Aug 13 08:02:07 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 13 Aug 2018 22:02:07 +1000
Subject: [Python-Dev] Can we split PEP 489 (extension module init) ?
In-Reply-To: <pkmspd$q21$1@blaine.gmane.org>
References: <pkjlac$1g7$1@blaine.gmane.org>
 <d1cca82b-1db7-cca7-c67e-f68f8aee04f2@gmail.com>
 <pkjor5$vd$1@blaine.gmane.org>
 <f3cfc548-a60d-b3da-5e25-d9000dfd5638@gmail.com>
 <pkmspd$q21$1@blaine.gmane.org>
Message-ID: <CADiSq7fLaPbz_McsUHY27H9Y=aYjU9pRHeFrCSi6-FJ1OuPP4g@mail.gmail.com>

On Sun, 12 Aug 2018 at 00:50, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Petr Viktorin schrieb am 10.08.2018 um 13:48:
> > Would this be better than a flag + raising an error on init?
>
> Ok, I've implemented this in Cython for now, to finally move the PEP-489
> support forward. The somewhat annoying drawback is that module reloading
> previously *seemed* to work, simply because it didn't actually do anything.
> Now, people will get an exception in cases that previously worked silently.
> An exception would probably have been better from the beginning, because it
> clearly tells people that what they are trying is not supported. Now it's a
> bit of a breaking change. I'll see what it gives.

If the breakage turns out to cause problems, some potential ways to
mitigate it would be:

1. Emulate the old "extension modules are singletons" behaviour in the
module creation slot, and only emit a deprecation warning when it gets
called multiple times rather than failing outright.
2. As for 1, but error out when the active interpreter changes (that
will let a module support reloading, while explicitly erroring out if
you attempt to load it from multiple subinterpreters)

Both of those would need some level of PEP 3121 support to clear out
the singleton state when the module gets outright destroyed, but
they'd still provide a middle ground between the ill-defined behaviour
of single-phase initialisation and full-fledged support multiple
concurrently loaded independent copies of the same extension module.

Under that interpretation, the required clarification in PEP 489 would
be to replace the term "supports" with "has clearly defined and
documented behaviour", and then make it clear that that defined
behaviour is allowed to be "Fails with an exception explaining the
limitation", or "manages some internal state as a process-level
singleton".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve.dower at python.org  Mon Aug 13 11:04:08 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 13 Aug 2018 08:04:08 -0700
Subject: [Python-Dev] [python-committers] Winding down 3.4
In-Reply-To: <f5d4fa80-7cfa-08e1-d2c2-96d015be99dc@python.org>
References: <55df745c-bf49-9849-0e64-b54b797ca0f1@hastings.org>
 <f5d4fa80-7cfa-08e1-d2c2-96d015be99dc@python.org>
Message-ID: <mailman.28.1534172648.29263.python-dev@python.org>

?So that 3.4 dies in good health??

More like getting all its evil deeds off its chest on the death bed, I think :)

Top-posted from my Windows 10 phone

From: Antoine Pitrou
Sent: Monday, 13 August 2018 2:59
To: Larry Hastings; python-committers; Python-Dev
Subject: Re: [python-committers] Winding down 3.4


Le 13/08/2018 ? 11:49, Larry Hastings a ?crit?:
> 
> 
> We of the core dev community commit to supporting Python releases for
> five years.? Releases get eighteen months of active bug fixes, followed
> by three and a half years of security fixes.? Python 3.4 turns 5 next
> March--at which point we'll stop supporting it, and I'll retire as 3.4
> release manager.
> 
> My plan is to make one final release on or around its fifth birthday
> containing the last round of security fixes.? That's about seven months
> from now.? Nothing has been merged since the releases of 3.4.9 and 3.5.6
> last week, and there are no open PRs against either of those releases.
> 
> But!? There are still a couple languishing "critical" bugs:
> 
>     "shutil copy* unsafe on POSIX - they preserve setuid/setgit bits"
>     https://bugs.python.org/issue17180
> 
>     "XML vulnerabilities in Python"
>     https://bugs.python.org/issue17239
> 
>     "fflush called on pointer to potentially closed file" (Windows only)
>     https://bugs.python.org/issue19050
> 
> It'd be nice to resolve all those issues, one way or another, before we
> retire 3.4.

So that 3.4 dies in good health?

Regards

Antoine.
_______________________________________________
python-committers mailing list
python-committers at python.org
https://mail.python.org/mailman/listinfo/python-committers
Code of Conduct: https://www.python.org/psf/codeofconduct/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180813/b37a936d/attachment.html>

From marko.ristin at gmail.com  Wed Aug 15 04:25:17 2018
From: marko.ristin at gmail.com (Marko Ristin-Kaufmann)
Date: Wed, 15 Aug 2018 10:25:17 +0200
Subject: [Python-Dev] Pre- and post-conditions
Message-ID: <CAGu4bVA-8LqyT=Foh0+znR4cPgZFqcKRpiyc7a3xD9nwg+_4ow@mail.gmail.com>

Hi python devs,

I would be very interested to bring design-by-contract into python 3. I
looked at some of the packages I found on pypi and also we rolled our own
solution (https://github.com/Parquery/icontract/). I also looked into
https://www.python.org/dev/peps/pep-0316/.

However, all these solutions seem quite clunky to me. The decorators
involve an unnecessary computational overhead and the implementation of
icontract became quite tricky once we wanted to get the default values of
the decorated function.

What do you think about the following solution as an extension to python
compiler / interpreter?

* We specify pre- and post-conditions in the docstring.
* Python interpreter parses these conditions and adapts the corresponding
byte codes at the begining and the end of the function body.

I'm very grateful for any feedback on this!

Marko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180815/8c604e93/attachment.html>

From marko.ristin at gmail.com  Wed Aug 15 04:28:05 2018
From: marko.ristin at gmail.com (Marko Ristin-Kaufmann)
Date: Wed, 15 Aug 2018 10:28:05 +0200
Subject: [Python-Dev] Hi!
Message-ID: <CAGu4bVDKT=QfUzP4oUV-OA4c0Mo=ZK-L9y+SerhbqfV+pPv92A@mail.gmail.com>

Hi python devs,
Please let me briefly introduce myself. I'm Marko and live in Zurich,
Switzerland. I learned Python and Django during an intership in 2007 and
followed its development ever since.

I did research in computer vision (at ETH Zurich). I work now at Parquery
AG (Zurich, Switzerland) where we implement most of our infra-structure in
Python 3.

Cheers,
Marko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180815/9003c0ce/attachment.html>

From chris.barker at noaa.gov  Wed Aug 15 12:34:56 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 15 Aug 2018 09:34:56 -0700
Subject: [Python-Dev] Pre- and post-conditions
In-Reply-To: <CAGu4bVA-8LqyT=Foh0+znR4cPgZFqcKRpiyc7a3xD9nwg+_4ow@mail.gmail.com>
References: <CAGu4bVA-8LqyT=Foh0+znR4cPgZFqcKRpiyc7a3xD9nwg+_4ow@mail.gmail.com>
Message-ID: <CALGmxEJ2J5FjniaANLkqVBxGH7YHJOb-XD1HQSKZo4iBYAAHhg@mail.gmail.com>

This is an appropriate topic for pyton-ideas:

https://mail.python.org/mailman/listinfo/python-ideas

not python-dev.

I'm sure you'll find interest in your idea there.

-CHB


On Wed, Aug 15, 2018 at 1:25 AM, Marko Ristin-Kaufmann <
marko.ristin at gmail.com> wrote:

> Hi python devs,
>
> I would be very interested to bring design-by-contract into python 3. I
> looked at some of the packages I found on pypi and also we rolled our own
> solution (https://github.com/Parquery/icontract/). I also looked into
> https://www.python.org/dev/peps/pep-0316/.
>
> However, all these solutions seem quite clunky to me. The decorators
> involve an unnecessary computational overhead and the implementation of
> icontract became quite tricky once we wanted to get the default values of
> the decorated function.
>
> What do you think about the following solution as an extension to python
> compiler / interpreter?
>
> * We specify pre- and post-conditions in the docstring.
> * Python interpreter parses these conditions and adapts the corresponding
> byte codes at the begining and the end of the function body.
>
> I'm very grateful for any feedback on this!
>
> Marko
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180815/d7c7a9c6/attachment.html>

From marko.ristin at gmail.com  Wed Aug 15 12:56:07 2018
From: marko.ristin at gmail.com (Marko Ristin-Kaufmann)
Date: Wed, 15 Aug 2018 18:56:07 +0200
Subject: [Python-Dev] Pre- and post-conditions
In-Reply-To: <CALGmxEJ2J5FjniaANLkqVBxGH7YHJOb-XD1HQSKZo4iBYAAHhg@mail.gmail.com>
References: <CAGu4bVA-8LqyT=Foh0+znR4cPgZFqcKRpiyc7a3xD9nwg+_4ow@mail.gmail.com>
 <CALGmxEJ2J5FjniaANLkqVBxGH7YHJOb-XD1HQSKZo4iBYAAHhg@mail.gmail.com>
Message-ID: <CAGu4bVC6s4JcXSqunfA7CH=80t_G8K71va1PGRahG2KRkx9oVQ@mail.gmail.com>

Hi Chris,
Thanks for pointing me to the other list! I'll write there.

Cheers,
Marko

Le mer. 15 ao?t 2018 ? 18:35, Chris Barker <chris.barker at noaa.gov> a ?crit :

> This is an appropriate topic for pyton-ideas:
>
> https://mail.python.org/mailman/listinfo/python-ideas
>
> not python-dev.
>
> I'm sure you'll find interest in your idea there.
>
> -CHB
>
>
> On Wed, Aug 15, 2018 at 1:25 AM, Marko Ristin-Kaufmann <
> marko.ristin at gmail.com> wrote:
>
>> Hi python devs,
>>
>> I would be very interested to bring design-by-contract into python 3. I
>> looked at some of the packages I found on pypi and also we rolled our own
>> solution (https://github.com/Parquery/icontract/). I also looked into
>> https://www.python.org/dev/peps/pep-0316/.
>>
>> However, all these solutions seem quite clunky to me. The decorators
>> involve an unnecessary computational overhead and the implementation of
>> icontract became quite tricky once we wanted to get the default values of
>> the decorated function.
>>
>> What do you think about the following solution as an extension to python
>> compiler / interpreter?
>>
>> * We specify pre- and post-conditions in the docstring.
>> * Python interpreter parses these conditions and adapts the corresponding
>> byte codes at the begining and the end of the function body.
>>
>> I'm very grateful for any feedback on this!
>>
>> Marko
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>>
>>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180815/464cc139/attachment.html>

From aixtools at felt.demon.nl  Thu Aug 16 05:07:19 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Thu, 16 Aug 2018 11:07:19 +0200
Subject: [Python-Dev] [issue17180] shutil copy* unsafe on POSIX - they
 preserve setuid/setgit bits
In-Reply-To: <1534330869.85.0.56676864532.issue17180@psf.upfronthosting.co.za>
References: <1534330869.85.0.56676864532.issue17180@psf.upfronthosting.co.za>
Message-ID: <3836f08b-afe3-fa03-a73f-7bdf69f4fdf1@felt.demon.nl>

I want to believe this can be resolved - without breakage on POSIX.

Clarification: while Mac/OS falls under "posix" in python terms - maybe
"breakage" will need to be accepted,
or, for "back-ports" Mac/OS will be "as if root or super-user" and use
an additional (optional) argument in 3.8 and beyond
to keep backwards compatibility.

Short text: to proceed I think we should start with getting some
additional tests into test_shutil.py asap so that we can see how systems
respond without any changes.

My experience is that cp -r[pP] behaves the same as shutil.copy*() when
the EUID==0, aka superuser,
but strips special bits from files and cannot copy the UID/GID owner
bits of the inode.

I would appreciate someone helping me writing more extensive testing.

We need to test:
* root
* not-root, but owner
* not-root, not owner, but in group
* not-root, "other", other "read" access exists

* if the test does not already exist - also check behavior when directories
? have/do not have "search" (x-bit) enabled.

I am working on a patch to address these different conditions.

Ideally, the "not-owner" and "not-group" tests can be run
by creating the "copy.me" area as root, setting perms, etc.
and then using su -c to run the shutil.copy*() call
and back as root make the verification.

??? Perspective ???

If this is too much discussion, please reply with suggestions - privately -
on what I could do better to not waste your time.

The issue seems unchanged since original posting.

The original report states:
hen copying the mode of a file with copy, copy2, copymode, copystat or
copytree, all permission bits are copied (including setuid and setgit),
but the owner of the file is not. This can be used for privilege escalation.

...snip...

The behaviour of copymode/copystat in this case is the same as `chmod
--reference', and there can be some expectation of unsafety, but
copy/copy2/copytree's behaviour differs from that of `cp -p', and this
is a non-obvious difference.

For clarity: GNU chmod states:

--reference=RFILE
??? use RFILE's mode instead of MODE values

Additionally, the chmod man page reminds us the "special bit" masking
behavior is different for files and directries.
Specifically, SUID, SGID and SVTX should not be cleared unless
specifically requested by a chmod "u-s,g-s" specification.

"... a directory's unmentioned set user and group ID bits are not affected"

Additional comments discuss:
short window of opportunity (files are copied first, then mode bits copied)
breakage with the past (copytree used as "backup", regardless of version)

And the comment/opinion that shutil.copy() should emulate cp (implies
emulate "cp -r", so neither -p nor -P)

it seems shutil.copy2() is adding the -p (or -P if follow_symlinks=false)

There was a modification to test_shutil.py suggested as part of a patch.
I added that to verify the issue is still current.

???
diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py
index 7e0a3292e0..7ceefd1ebc 100644
--- a/Lib/test/test_shutil.py
+++ b/Lib/test/test_shutil.py
@@ -1471,6 +1471,24 @@ class TestShutil(unittest.TestCase):
???????? rv = shutil.copytree(src_dir, dst_dir)
???????? self.assertEqual(['foo'], os.listdir(rv))

+??? @unittest.skipUnless((os.name == "posix" and os.geteuid() != 0),
"Requires POSIX compatible OS and non-root userid")
+??? def test_copy_remove_setuid(self):
+??????? src_dir = self.mkdtemp()
+??????? src_file = os.path.join(src_dir, 'foo')
+??????? write_file(src_file, 'foo')
+??????? dst_file = os.path.join(src_dir, 'bar')
+??????? harmful_mode = stat.S_IRUSR | stat.S_IXUSR | stat.S_ISUID
+??????? harmless_mode = stat.S_IRUSR | stat.S_IXUSR
+
+??????? # set mode and verify
+??????? os.chmod(src_file, harmful_mode)
+??????? mode = stat.S_IMODE(os.stat(src_file).st_mode)
+??????? self.assertTrue(oct(mode), oct(harmful_mode))
+
+??????? # check that copy does not preserve harmful bits
+??????? shutil.copy(src_file, dst_file)
+??????? mode = stat.S_IMODE(os.stat(dst_file).st_mode)
+??????? self.assertEqual(oct(mode), oct(harmless_mode))

?class TestWhich(unittest.TestCase):
???
The result is:
root at x066:[/data/prj/python/python3-3.8]./python -m test -v test_shutil
== CPython 3.8.0a0 (heads/master:cca4eec3c0, Aug 13 2018, 04:53:15) [C]
== AIX-1-00C291F54C00-powerpc-32bit big-endian
== cwd: /data/prj/python/python3-3.8/build/test_python_10944516
== CPU count: 8
== encodings: locale=ISO8859-1, FS=iso8859-1
Run tests sequentially
...
test_copy_remove_setuid (test.test_shutil.TestShutil) ... FAIL
...
======================================================================
FAIL: test_copy_remove_setuid (test.test_shutil.TestShutil)
----------------------------------------------------------------------
Traceback (most recent call last):
? File "/data/prj/python/git/python3-3.8/Lib/test/test_shutil.py", line
1491, in test_copy_remove_setuid
??? self.assertEqual(oct(mode), oct(harmless_mode))
AssertionError: '0o4500' != '0o500'
- 0o4500
??? -
+ 0o500


----------------------------------------------------------------------


On 8/15/2018 1:01 PM, Michael Felt wrote:
> Michael Felt <aixtools at felt.demon.nl> added the comment:
>
> I am looking at this.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180816/fad82cae/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pEpkey.asc
Type: application/pgp-keys
Size: 1765 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180816/fad82cae/attachment.key>

From christian at python.org  Thu Aug 16 09:27:52 2018
From: christian at python.org (Christian Heimes)
Date: Thu, 16 Aug 2018 15:27:52 +0200
Subject: [Python-Dev] TLS 1.3 support
Message-ID: <8b2dde26-7b78-f864-1c3d-a9170a7e6967@python.org>

Hi,

I have some exciting news. RFC 8446 [1] for TLS 1.3 was finalized a
couple of days ago. The new TLS standard comes with several major
improvements, but also with major changes.

First some good news:
Python's ssl module not compiles with OpenSSL 1.1.1-pre. It's also able
to negotiate and establish TLS 1.3 connections successfully. Almost all
unit tests are passing, too. Yesterday I pushed some patches to fix 2.7,
3.6, 3.7, and master.

Of course, there is a catch. While TLS 1.0 to 1.2 were just gradual
improvements over SSLv3, TLS 1.3 behaves differently on several levels.
The article [2] from Nick Sullivan explains the biggest changes well.
Most improvements like 1-RTT handshake, enforced perfect forward secrecy
and improved cryptography are not visible to applications. However some
changes and new features are going to need additional work and new APIs.

I'm open to discuss the matter at the core dev sprints in Seattle next
month.

I'm still in the process of reviewing and assessing changes. OpenSSL
1.1.1's APIs are not finalized yet, too. I'm working closely with Red
Hat's crypto team and OpenSSL devs on the topic. I'll walk you through
some changes and possibly new APIs that I have identified so far.


New cipher suites
=================

Old ciphers suites in TLS 1.2 and earlier looked like

   TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

The parameters are key agreement/key exchange (ECDHE), authentication
(RSA), bulk encryption (AES-256) with block and/or authenticated
encryption mode (GCM) and finally a pseudo random function for MAC.

In TLS 1.3, the cipher suites only specify bulk encryption and PRF. The
parameters for key agreement and authentication are configured
differently. TLS 1.3 uses dedicated TLS extensions for that, too. This
allows more fine grained control to specific supported EC curves,
signature algorithms (RSA PKCS#1.5, RSASSA-PSS) and so on. Some settings
also apply to TLS 1.2 already.

Further more, OpenSSL has introduced new APIs to configure TLS 1.3
cipher suites. SSLContext.set_ciphers() cannot be used to disable TLS
1.3 suites. For now, that is fine. All default algorithms are secure and
state-of-the-art.

For 3.8, I'm planning to make some of the settings configurable. I might
need to backport the new APIs to 3.7 and 3.6, though.


post handshake authentication
=============================

TLS 1.3 introduced the concept of post handshake authentication (PHA). A
server can either request a client to provide client cert authentication
during the handshake, directly after the handshake, or at any later
point. It makes optional authentication for HTTP much more sane. For
example a server may only require authentication for POST requests and
GET requests to /secure/* path. In TLS 1.2, optional client cert auth
requires a complete renegotiation of the handshake.

OpenSSL 1.1.1-pre9 will most likely disable PHA by default [3], because
PHA breaks some assumptions of TLS 1.2 clients and servers. But PHA
might be required for path specific authentication. I need to wait and
see how mod_ssl implements the feature.

I might have to add SSL_VERIFY_POST_HANDSHAKE,
SSL_verify_client_post_handshake(), and
SSL_CTX_set_post_handshake_auth() from [3][4] to Python 2.7, 3.6, 3.7
and master.


session handling
================

TLS 1.3 uses a different approach to handle sessions. Simple speaking,
session data is exchanged after the handshake, there can be more than
one ticket, tickets must not be reused, and session resumption involved
DHE. ssl.SSLSession [5] feature only applies to TLS 1.2 and earlier. I
need to come up with a different approach here. For the time being, TLS
1.3 won't be compatible with manual session control. It's not a big deal
since it is an advanced feature any way.

The new session system also seems to break FTP over TLS. FTP uses two
different TCP connections (control channel, data channel). For FTP over
TLS, the data channel must reuse a session of the control channel. For
now, I'm plannung to restrict FTP to TLSv1.2.


0-RTT resumption
=================

0 round trip time handshake on session resumption is an optional and
advanced feature. I might expose the feature in Python 3.8.

Regards,
Christian


[1] https://tools.ietf.org/html/rfc8446
[2] https://blog.cloudflare.com/rfc-8446-aka-tls-1-3/
[3] https://github.com/openssl/openssl/issues/6933
[4] https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_verify.html
[5] https://docs.python.org/3/library/ssl.html#ssl-session

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180816/ea4c9bec/attachment.sig>

From christian at python.org  Thu Aug 16 10:07:59 2018
From: christian at python.org (Christian Heimes)
Date: Thu, 16 Aug 2018 16:07:59 +0200
Subject: [Python-Dev] TLS 1.3 support
In-Reply-To: <8b2dde26-7b78-f864-1c3d-a9170a7e6967@python.org>
References: <8b2dde26-7b78-f864-1c3d-a9170a7e6967@python.org>
Message-ID: <dc477f28-5274-48b5-2c4e-73d55ec29155@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Argh, typo! Thanks for pointing that out, Miro. I meant to write "now
compiles with OpenSSL 1.1.1-pre".

On 2018-08-16 15:27, Christian Heimes wrote:
> First some good news: Python's ssl module not compiles with OpenSSL
> 1.1.1-pre. It's also able
                      ^^^
  Python's ssl module NOW compiles with OpenSSL 1.1.1-pre.

The latest release OpenSSL 1.1.1-pre8 from June has some bugs, e.g.
twp way shutdown is broken under some circumstances [1]. The current
git master of CPython is mostly compatible with the latest git
snapshot of OpenSSL. I'm waiting for OpenSSL devs to merge PR 6938 [2]
and release pre9.

Christian

[1] https://github.com/openssl/openssl/issues/6262
[2] https://github.com/openssl/openssl/pull/6938
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEL7+ElqIWjf0BBClLhmhSSSEttokFAlt1hTIACgkQhmhSSSEt
tokiMAgApG/ruPxq0WW5FsOuQquH+jH6coifKzEsHsZfIhHLGQwVlpUjRhtQdj+T
YpprzjRU6AS+R7J+/fr7t032TPfogSSpCHSh22RIeY5BlYkNBjjyhxuItwGu8SiP
pvKThAsLiu4x4SCJHMJHgjv/M9ljAsUM9VgUjCy8UQ8p6zfKjC2B/Vwg8WLO0XOn
90ggMEpMfyb6t0daUIEe2eEPBmfwN5K0cctWg1rb98LUbfOdq3MMEWjKgYBBsCS7
vk3RIKE1ZMtzDJaO3M09wHYLu2QXtUpo9k3VU2CABC65Jw9SkyMDUGyUFZjV5GU/
dW0K5DyHr/6JHfHWfq37Y3y/yBXNsA==
=gVcQ
-----END PGP SIGNATURE-----

From nmav at redhat.com  Thu Aug 16 10:00:52 2018
From: nmav at redhat.com (Nikos Mavrogiannopoulos)
Date: Thu, 16 Aug 2018 16:00:52 +0200
Subject: [Python-Dev] TLS 1.3 support
In-Reply-To: <8b2dde26-7b78-f864-1c3d-a9170a7e6967@python.org>
References: <8b2dde26-7b78-f864-1c3d-a9170a7e6967@python.org>
Message-ID: <CADh2w8Tkn2-huQV8Xof2aCDLHRXE0sdeNdHm45YsUhQ2VHBJAg@mail.gmail.com>

Thank you Christian for your great work!

On Thu, Aug 16, 2018 at 3:28 PM Christian Heimes <christian at python.org>
wrote:

> Hi,
>
> I have some exciting news. RFC 8446 [1] for TLS 1.3 was finalized a
> couple of days ago. The new TLS standard comes with several major
> improvements, but also with major changes.
>
> First some good news:
> Python's ssl module not compiles with OpenSSL 1.1.1-pre. It's also able
> to negotiate and establish TLS 1.3 connections successfully. Almost all
> unit tests are passing, too. Yesterday I pushed some patches to fix 2.7,
> 3.6, 3.7, and master.
>
> Of course, there is a catch. While TLS 1.0 to 1.2 were just gradual
> improvements over SSLv3, TLS 1.3 behaves differently on several levels.
> The article [2] from Nick Sullivan explains the biggest changes well.
> Most improvements like 1-RTT handshake, enforced perfect forward secrecy
> and improved cryptography are not visible to applications. However some
> changes and new features are going to need additional work and new APIs.
>
> I'm open to discuss the matter at the core dev sprints in Seattle next
> month.
>
> I'm still in the process of reviewing and assessing changes. OpenSSL
> 1.1.1's APIs are not finalized yet, too. I'm working closely with Red
> Hat's crypto team and OpenSSL devs on the topic. I'll walk you through
> some changes and possibly new APIs that I have identified so far.
>
>
> New cipher suites
> =================
>
> Old ciphers suites in TLS 1.2 and earlier looked like
>
>    TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
>
> The parameters are key agreement/key exchange (ECDHE), authentication
> (RSA), bulk encryption (AES-256) with block and/or authenticated
> encryption mode (GCM) and finally a pseudo random function for MAC.
>
> In TLS 1.3, the cipher suites only specify bulk encryption and PRF. The
> parameters for key agreement and authentication are configured
> differently. TLS 1.3 uses dedicated TLS extensions for that, too. This
> allows more fine grained control to specific supported EC curves,
> signature algorithms (RSA PKCS#1.5, RSASSA-PSS) and so on. Some settings
> also apply to TLS 1.2 already.
>
> Further more, OpenSSL has introduced new APIs to configure TLS 1.3
> cipher suites. SSLContext.set_ciphers() cannot be used to disable TLS
> 1.3 suites. For now, that is fine. All default algorithms are secure and
> state-of-the-art.
>
> For 3.8, I'm planning to make some of the settings configurable. I might
> need to backport the new APIs to 3.7 and 3.6, though.
>
>
> post handshake authentication
> =============================
>
> TLS 1.3 introduced the concept of post handshake authentication (PHA). A
> server can either request a client to provide client cert authentication
> during the handshake, directly after the handshake, or at any later
> point. It makes optional authentication for HTTP much more sane. For
> example a server may only require authentication for POST requests and
> GET requests to /secure/* path. In TLS 1.2, optional client cert auth
> requires a complete renegotiation of the handshake.
>
> OpenSSL 1.1.1-pre9 will most likely disable PHA by default [3], because
> PHA breaks some assumptions of TLS 1.2 clients and servers. But PHA
> might be required for path specific authentication. I need to wait and
> see how mod_ssl implements the feature.
>
> I might have to add SSL_VERIFY_POST_HANDSHAKE,
> SSL_verify_client_post_handshake(), and
> SSL_CTX_set_post_handshake_auth() from [3][4] to Python 2.7, 3.6, 3.7
> and master.
>
>
> session handling
> ================
>
> TLS 1.3 uses a different approach to handle sessions. Simple speaking,
> session data is exchanged after the handshake, there can be more than
> one ticket, tickets must not be reused, and session resumption involved
> DHE. ssl.SSLSession [5] feature only applies to TLS 1.2 and earlier. I
> need to come up with a different approach here. For the time being, TLS
> 1.3 won't be compatible with manual session control. It's not a big deal
> since it is an advanced feature any way.
>
> The new session system also seems to break FTP over TLS. FTP uses two
> different TCP connections (control channel, data channel). For FTP over
> TLS, the data channel must reuse a session of the control channel. For
> now, I'm plannung to restrict FTP to TLSv1.2.
>
>
> 0-RTT resumption
> =================
>
> 0 round trip time handshake on session resumption is an optional and
> advanced feature. I might expose the feature in Python 3.8.
>
> Regards,
> Christian
>
>
> [1] https://tools.ietf.org/html/rfc8446
> [2] https://blog.cloudflare.com/rfc-8446-aka-tls-1-3/
> [3] https://github.com/openssl/openssl/issues/6933
> [4] https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_verify.html
> [5] https://docs.python.org/3/library/ssl.html#ssl-session
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180816/bc630148/attachment.html>

From arj.python at gmail.com  Thu Aug 16 13:25:43 2018
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Thu, 16 Aug 2018 21:25:43 +0400
Subject: [Python-Dev] TLS 1.3 support
In-Reply-To: <8b2dde26-7b78-f864-1c3d-a9170a7e6967@python.org>
References: <8b2dde26-7b78-f864-1c3d-a9170a7e6967@python.org>
Message-ID: <CADrxXXn0Nb6w1osteombtGVjRP1oReDiC25Ec5guCQ_RHaKtVw@mail.gmail.com>

my country, mauritius, had a participation in the dev,

internet society vid :
https://youtu.be/u6rz4PWA_As?t=4524

the red blue yellow green flag is our flag

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ
Mauritius

On Thu, 16 Aug 2018, 17:27 Christian Heimes, <christian at python.org> wrote:

> Hi,
>
> I have some exciting news. RFC 8446 [1] for TLS 1.3 was finalized a
> couple of days ago. The new TLS standard comes with several major
> improvements, but also with major changes.
>
> First some good news:
> Python's ssl module not compiles with OpenSSL 1.1.1-pre. It's also able
> to negotiate and establish TLS 1.3 connections successfully. Almost all
> unit tests are passing, too. Yesterday I pushed some patches to fix 2.7,
> 3.6, 3.7, and master.
>
> Of course, there is a catch. While TLS 1.0 to 1.2 were just gradual
> improvements over SSLv3, TLS 1.3 behaves differently on several levels.
> The article [2] from Nick Sullivan explains the biggest changes well.
> Most improvements like 1-RTT handshake, enforced perfect forward secrecy
> and improved cryptography are not visible to applications. However some
> changes and new features are going to need additional work and new APIs.
>
> I'm open to discuss the matter at the core dev sprints in Seattle next
> month.
>
> I'm still in the process of reviewing and assessing changes. OpenSSL
> 1.1.1's APIs are not finalized yet, too. I'm working closely with Red
> Hat's crypto team and OpenSSL devs on the topic. I'll walk you through
> some changes and possibly new APIs that I have identified so far.
>
>
> New cipher suites
> =================
>
> Old ciphers suites in TLS 1.2 and earlier looked like
>
>    TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
>
> The parameters are key agreement/key exchange (ECDHE), authentication
> (RSA), bulk encryption (AES-256) with block and/or authenticated
> encryption mode (GCM) and finally a pseudo random function for MAC.
>
> In TLS 1.3, the cipher suites only specify bulk encryption and PRF. The
> parameters for key agreement and authentication are configured
> differently. TLS 1.3 uses dedicated TLS extensions for that, too. This
> allows more fine grained control to specific supported EC curves,
> signature algorithms (RSA PKCS#1.5, RSASSA-PSS) and so on. Some settings
> also apply to TLS 1.2 already.
>
> Further more, OpenSSL has introduced new APIs to configure TLS 1.3
> cipher suites. SSLContext.set_ciphers() cannot be used to disable TLS
> 1.3 suites. For now, that is fine. All default algorithms are secure and
> state-of-the-art.
>
> For 3.8, I'm planning to make some of the settings configurable. I might
> need to backport the new APIs to 3.7 and 3.6, though.
>
>
> post handshake authentication
> =============================
>
> TLS 1.3 introduced the concept of post handshake authentication (PHA). A
> server can either request a client to provide client cert authentication
> during the handshake, directly after the handshake, or at any later
> point. It makes optional authentication for HTTP much more sane. For
> example a server may only require authentication for POST requests and
> GET requests to /secure/* path. In TLS 1.2, optional client cert auth
> requires a complete renegotiation of the handshake.
>
> OpenSSL 1.1.1-pre9 will most likely disable PHA by default [3], because
> PHA breaks some assumptions of TLS 1.2 clients and servers. But PHA
> might be required for path specific authentication. I need to wait and
> see how mod_ssl implements the feature.
>
> I might have to add SSL_VERIFY_POST_HANDSHAKE,
> SSL_verify_client_post_handshake(), and
> SSL_CTX_set_post_handshake_auth() from [3][4] to Python 2.7, 3.6, 3.7
> and master.
>
>
> session handling
> ================
>
> TLS 1.3 uses a different approach to handle sessions. Simple speaking,
> session data is exchanged after the handshake, there can be more than
> one ticket, tickets must not be reused, and session resumption involved
> DHE. ssl.SSLSession [5] feature only applies to TLS 1.2 and earlier. I
> need to come up with a different approach here. For the time being, TLS
> 1.3 won't be compatible with manual session control. It's not a big deal
> since it is an advanced feature any way.
>
> The new session system also seems to break FTP over TLS. FTP uses two
> different TCP connections (control channel, data channel). For FTP over
> TLS, the data channel must reuse a session of the control channel. For
> now, I'm plannung to restrict FTP to TLSv1.2.
>
>
> 0-RTT resumption
> =================
>
> 0 round trip time handshake on session resumption is an optional and
> advanced feature. I might expose the feature in Python 3.8.
>
> Regards,
> Christian
>
>
> [1] https://tools.ietf.org/html/rfc8446
> [2] https://blog.cloudflare.com/rfc-8446-aka-tls-1-3/
> [3] https://github.com/openssl/openssl/issues/6933
> [4] https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_verify.html
> [5] https://docs.python.org/3/library/ssl.html#ssl-session
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/arj.python%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180816/234f33c9/attachment.html>

From status at bugs.python.org  Fri Aug 17 12:09:51 2018
From: status at bugs.python.org (Python tracker)
Date: Fri, 17 Aug 2018 18:09:51 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20180817160951.B6D2B116803@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2018-08-10 - 2018-08-17)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6775 (+19)
  closed 39396 (+29)
  total  46171 (+48)

Open issues with patches: 2697 


Issues opened (31)
==================

#34217: windows: cross compilation fails due to headers with uppercase
https://bugs.python.org/issue34217  reopened by erikjanss

#34373: test_time errors on AIX
https://bugs.python.org/issue34373  opened by Michael.Felt

#34374: Shouldn't shutil.copyfile replace link in dst with follow_syml
https://bugs.python.org/issue34374  opened by theger

#34381: Make tests with outbound connection optional
https://bugs.python.org/issue34381  opened by michael-o

#34382: test_os.test_mode fails when directory base directory has g+s 
https://bugs.python.org/issue34382  opened by Michael.Felt

#34383: asyncio passes SSL certificate error to loop.call_exception_ha
https://bugs.python.org/issue34383  opened by dandrei

#34384: os.readlink does not accept pathlib.Path on Windows
https://bugs.python.org/issue34384  opened by girtsf

#34388: collect_gdb fails for test.pythoninfo in several AMD64 FreeBSD
https://bugs.python.org/issue34388  opened by pablogsal

#34389: CPython may fail to build in the presence of a ~/.pydistutils.
https://bugs.python.org/issue34389  opened by Antony.Lee

#34391: test_ftplib is failing with TLS 1.3
https://bugs.python.org/issue34391  opened by christian.heimes

#34392: Add sys.isinterned()
https://bugs.python.org/issue34392  opened by serhiy.storchaka

#34394: Descriptors HowTo doesn't mention __set_name__
https://bugs.python.org/issue34394  opened by MarSoft

#34395: memory leaks in error handling in csv and pickle modules
https://bugs.python.org/issue34395  opened by sir-sigurd

#34396: Certain methods that heap allocated subtypes inherit suffer a 
https://bugs.python.org/issue34396  opened by bup

#34397: remove redundant overflow checks in tuple and list implementat
https://bugs.python.org/issue34397  opened by sir-sigurd

#34398: Docs search does not index glossary items
https://bugs.python.org/issue34398  opened by jfine2358

#34401: [SOLUTION] Make test_gdb work on HP-UX
https://bugs.python.org/issue34401  opened by michael-o

#34402: [SOLUTION] strftime fails on HP-UX
https://bugs.python.org/issue34402  opened by michael-o

#34403: test_utf8_mode.test_cmd_line() fails on HP-UX due to false ass
https://bugs.python.org/issue34403  opened by michael-o

#34404: test_time incorrectly defined
https://bugs.python.org/issue34404  opened by michael-o

#34405: Upgrade to OpenSSL 1.1.0i / 1.0.2p
https://bugs.python.org/issue34405  opened by christian.heimes

#34406: Typo in documentation
https://bugs.python.org/issue34406  opened by Alisue Lambda

#34407: datetime.time.isoformat function has inconsistent behavior wit
https://bugs.python.org/issue34407  opened by Maksym Shalenyi (Enkidulan)

#34408: possible null pointer dereference in pystate.c
https://bugs.python.org/issue34408  opened by pablogsal

#34409: Add a way to customize iteration over fields in asdict() for t
https://bugs.python.org/issue34409  opened by mkurnikov

#34410: Segfault/TimeoutError: itertools.tee of multiprocessing.pool.i
https://bugs.python.org/issue34410  opened by carlorosati

#34411: ProactorEventLoop should use implement GetAddrInfo, GetNameInf
https://bugs.python.org/issue34411  opened by ysangkok+launchpad

#34412: strsignal(3) does not exist on HP-UX
https://bugs.python.org/issue34412  opened by michael-o

#34413: Porting section of 3.5 "What's New" doc does not mention bytec
https://bugs.python.org/issue34413  opened by eric.snow

#34415: Typo in logging.Formatter docstring
https://bugs.python.org/issue34415  opened by MarSoft

#34416: PyCapsule_Import seems to release the GIL without acquiring it
https://bugs.python.org/issue34416  opened by p-ganssle



Most recent 15 issues with no replies (15)
==========================================

#34415: Typo in logging.Formatter docstring
https://bugs.python.org/issue34415

#34408: possible null pointer dereference in pystate.c
https://bugs.python.org/issue34408

#34407: datetime.time.isoformat function has inconsistent behavior wit
https://bugs.python.org/issue34407

#34403: test_utf8_mode.test_cmd_line() fails on HP-UX due to false ass
https://bugs.python.org/issue34403

#34401: [SOLUTION] Make test_gdb work on HP-UX
https://bugs.python.org/issue34401

#34389: CPython may fail to build in the presence of a ~/.pydistutils.
https://bugs.python.org/issue34389

#34383: asyncio passes SSL certificate error to loop.call_exception_ha
https://bugs.python.org/issue34383

#34382: test_os.test_mode fails when directory base directory has g+s 
https://bugs.python.org/issue34382

#34373: test_time errors on AIX
https://bugs.python.org/issue34373

#34368: ftplib __init__ function can't handle 120 or 4xy reply when co
https://bugs.python.org/issue34368

#34367: AsyncResult.get() only notifies one thread
https://bugs.python.org/issue34367

#34366: _uuid module fails to compile on FreeBSD when libuuid is insta
https://bugs.python.org/issue34366

#34365: datetime's documentation refers to "comparison [...] falling b
https://bugs.python.org/issue34365

#34354: Memory leak on _testCongestion
https://bugs.python.org/issue34354

#34345: Add tests for PEP 468 and PEP 520
https://bugs.python.org/issue34345



Most recent 15 issues waiting for review (15)
=============================================

#34412: strsignal(3) does not exist on HP-UX
https://bugs.python.org/issue34412

#34408: possible null pointer dereference in pystate.c
https://bugs.python.org/issue34408

#34405: Upgrade to OpenSSL 1.1.0i / 1.0.2p
https://bugs.python.org/issue34405

#34401: [SOLUTION] Make test_gdb work on HP-UX
https://bugs.python.org/issue34401

#34398: Docs search does not index glossary items
https://bugs.python.org/issue34398

#34397: remove redundant overflow checks in tuple and list implementat
https://bugs.python.org/issue34397

#34395: memory leaks in error handling in csv and pickle modules
https://bugs.python.org/issue34395

#34392: Add sys.isinterned()
https://bugs.python.org/issue34392

#34391: test_ftplib is failing with TLS 1.3
https://bugs.python.org/issue34391

#34384: os.readlink does not accept pathlib.Path on Windows
https://bugs.python.org/issue34384

#34382: test_os.test_mode fails when directory base directory has g+s 
https://bugs.python.org/issue34382

#34381: Make tests with outbound connection optional
https://bugs.python.org/issue34381

#34373: test_time errors on AIX
https://bugs.python.org/issue34373

#34366: _uuid module fails to compile on FreeBSD when libuuid is insta
https://bugs.python.org/issue34366

#34363: dataclasses.asdict() mishandles dataclass instance attributes 
https://bugs.python.org/issue34363



Top 10 most discussed issues (10)
=================================

#34217: windows: cross compilation fails due to headers with uppercase
https://bugs.python.org/issue34217  13 msgs

#34398: Docs search does not index glossary items
https://bugs.python.org/issue34398   7 msgs

#34410: Segfault/TimeoutError: itertools.tee of multiprocessing.pool.i
https://bugs.python.org/issue34410   7 msgs

#17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit
https://bugs.python.org/issue17180   6 msgs

#34384: os.readlink does not accept pathlib.Path on Windows
https://bugs.python.org/issue34384   6 msgs

#34395: memory leaks in error handling in csv and pickle modules
https://bugs.python.org/issue34395   6 msgs

#34363: dataclasses.asdict() mishandles dataclass instance attributes 
https://bugs.python.org/issue34363   5 msgs

#34391: test_ftplib is failing with TLS 1.3
https://bugs.python.org/issue34391   5 msgs

#34411: ProactorEventLoop should use implement GetAddrInfo, GetNameInf
https://bugs.python.org/issue34411   5 msgs

#34369: kqueue.control() documentation and implementation mismatch
https://bugs.python.org/issue34369   4 msgs



Issues closed (29)
==================

#8478: tokenize.untokenize first token missing failure case
https://bugs.python.org/issue8478  closed by berker.peksag

#9372: pulldom.DOMEventStream.__getitem__ is broken
https://bugs.python.org/issue9372  closed by berker.peksag

#13837: test_shutil fails with symlinks enabled under Windows
https://bugs.python.org/issue13837  closed by berker.peksag

#15258: argparse documentation: Improve optparse section regarding all
https://bugs.python.org/issue15258  closed by berker.peksag

#24111: Valgrind suppression file should be updated
https://bugs.python.org/issue24111  closed by benjamin.peterson

#26818: trace CLI doesn't respect -s option
https://bugs.python.org/issue26818  closed by berker.peksag

#27026: async/await keywords are missing from reference docs
https://bugs.python.org/issue27026  closed by berker.peksag

#33451: Start pyc file lock the file
https://bugs.python.org/issue33451  closed by berker.peksag

#34151: use malloc() for better performance of some list operations
https://bugs.python.org/issue34151  closed by xiang.zhang

#34333: Path.with_suffix() raises TypeError when doing %-formatting
https://bugs.python.org/issue34333  closed by berker.peksag

#34356: Add support for args and kwargs in logging.conf
https://bugs.python.org/issue34356  closed by vinay.sajip

#34357: situation where urllib3 works, but urllib does not work
https://bugs.python.org/issue34357  closed by deivid

#34375: Subtests (unittest)
https://bugs.python.org/issue34375  closed by r.david.murray

#34376: Improve accuracy of math.hypot() and math.dist()
https://bugs.python.org/issue34376  closed by rhettinger

#34377: Update valgrind suppressions
https://bugs.python.org/issue34377  closed by benjamin.peterson

#34378: Documentation 3.5+
https://bugs.python.org/issue34378  closed by Mariatta

#34379: Move note about repeated calls to json.dump using the same fp 
https://bugs.python.org/issue34379  closed by inada.naoki

#34380: Relax Restrictions on await?
https://bugs.python.org/issue34380  closed by yselivanov

#34385: Failed to build these modules: _ctypes on Solaris 10
https://bugs.python.org/issue34385  closed by LarBob Doomer

#34386: Expose a dictionary of interned strings in sys module
https://bugs.python.org/issue34386  closed by serhiy.storchaka

#34387: Deletion of attributes in dataclass is buggy
https://bugs.python.org/issue34387  closed by eric.smith

#34390: arparse.ArgumentParser misparses list arguments followed by un
https://bugs.python.org/issue34390  closed by rhettinger

#34393: json.dumps - allow compression
https://bugs.python.org/issue34393  closed by eric.smith

#34399: [TLS] Update test keys to >= 2048bit
https://bugs.python.org/issue34399  closed by christian.heimes

#34400: Undefined behavior in Parser/parsetok.c
https://bugs.python.org/issue34400  closed by benjamin.peterson

#34414: Absolute imports conflict with local files
https://bugs.python.org/issue34414  closed by ronaldoussoren

#34418: Arguments missing from documentation for HTTPErrorProcessor.ht
https://bugs.python.org/issue34418  closed by berker.peksag

#34419: selectmodule.c does not compile on HP-UX due to bpo-31938/6dc5
https://bugs.python.org/issue34419  closed by taleinat

#34420: Complete your registration to Python tracker -- keysnSwaZe6Pti
https://bugs.python.org/issue34420  closed by berker.peksag

From arj.python at gmail.com  Mon Aug 20 02:41:28 2018
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Mon, 20 Aug 2018 10:41:28 +0400
Subject: [Python-Dev] unsolicited removal request
Message-ID: <CADrxXX=vTRydZpHxXw6zj+ySEmthGLu2HnVcC+LyJH_dcGENeg@mail.gmail.com>

just notifying that 199.103.2.101
was trying to remove me from this list. might try to remove others

yours,

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ
Mauritius
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/97115687/attachment.html>

From vstinner at redhat.com  Mon Aug 20 08:52:28 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 20 Aug 2018 14:52:28 +0200
Subject: [Python-Dev] [python-committers] Winding down 3.4
In-Reply-To: <55df745c-bf49-9849-0e64-b54b797ca0f1@hastings.org>
References: <55df745c-bf49-9849-0e64-b54b797ca0f1@hastings.org>
Message-ID: <CA+3bQGH0AbaWGyCNRHZ3E_TOS_mXGxWi0+SmT6-irZEAuaTqUg@mail.gmail.com>

> "shutil copy* unsafe on POSIX - they preserve setuid/setgit bits"
> https://bugs.python.org/issue17180

There is no fix. A fix may break the backward compatibility. Is it really
worth it for the last 3.4 release?

> "XML vulnerabilities in Python"
> https://bugs.python.org/issue17239

Bug inactive since 2015. I don't expect that anyone will step in next weeks
with a wonderful solution to all XML issues. I suggest to ignore this one
as well, this issue is as old as XML support in Python and I am not aware
of any victim of these issues.

Obviously, it would be "nice" to see a fix for these issues but it seems
like core devs are more interested to work on other topics and other
security issues.


> "fflush called on pointer to potentially closed file" (Windows only)
> https://bugs.python.org/issue19050

It seems like two core devs are opposed to fix this issue.

--

There are open security issues on the HTTP server and urllib. I am more
concerned by these issues, but it's hard to fix them, there is a risk of
introducing regressions.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/581125e3/attachment.html>

From rymg19 at gmail.com  Mon Aug 20 09:41:01 2018
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 20 Aug 2018 08:41:01 -0500
Subject: [Python-Dev] unsolicited removal request
In-Reply-To: <CADrxXX=vTRydZpHxXw6zj+ySEmthGLu2HnVcC+LyJH_dcGENeg@mail.gmail.com>
References: <CADrxXX=vTRydZpHxXw6zj+ySEmthGLu2HnVcC+LyJH_dcGENeg@mail.gmail.com>
Message-ID: <CAO41-mNSpCb8rPSUET2FiBaHop6EH5Fm0yXo_E-Q3wNtXLzmdw@mail.gmail.com>

I think you generally want to sent this to the list administrators
directly, but FWIW this has happened to me before. If it doesn't come up
again, you can probably ignore it.

On Mon, Aug 20, 2018, 1:43 AM Abdur-Rahmaan Janhangeer <arj.python at gmail.com>
wrote:

> just notifying that 199.103.2.101
> was trying to remove me from this list. might try to remove others
>
> yours,
>
> Abdur-Rahmaan Janhangeer
> https://github.com/Abdur-rahmaanJ
> Mauritius
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>
-- 

Ryan (????)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
https://refi64.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/c07a27b7/attachment.html>

From arj.python at gmail.com  Mon Aug 20 09:42:41 2018
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Mon, 20 Aug 2018 17:42:41 +0400
Subject: [Python-Dev] unsolicited removal request
In-Reply-To: <CAO41-mNSpCb8rPSUET2FiBaHop6EH5Fm0yXo_E-Q3wNtXLzmdw@mail.gmail.com>
References: <CADrxXX=vTRydZpHxXw6zj+ySEmthGLu2HnVcC+LyJH_dcGENeg@mail.gmail.com>
 <CAO41-mNSpCb8rPSUET2FiBaHop6EH5Fm0yXo_E-Q3wNtXLzmdw@mail.gmail.com>
Message-ID: <CADrxXXn1qQW4nEsXo+ZGYTETx3xA9_N6OKqLuKHx_GQYdyDO9A@mail.gmail.com>

ah yes, that's it. thank you !

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ
Mauritius

On Mon, 20 Aug 2018, 17:41 Ryan Gonzalez, <rymg19 at gmail.com> wrote:

> I think you generally want to sent this to the list administrators
> directly, but FWIW this has happened to me before. If it doesn't come up
> again, you can probably ignore it.
>
> On Mon, Aug 20, 2018, 1:43 AM Abdur-Rahmaan Janhangeer <
> arj.python at gmail.com> wrote:
>
>> just notifying that 199.103.2.101
>> was trying to remove me from this list. might try to remove others
>>
>> yours,
>>
>> Abdur-Rahmaan Janhangeer
>> https://github.com/Abdur-rahmaanJ
>> Mauritius
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>>
> --
>
> Ryan (????)
> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
> https://refi64.com/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/30cb88ca/attachment.html>

From redstone-cold at 163.com  Mon Aug 20 04:42:09 2018
From: redstone-cold at 163.com (Zhao Lee)
Date: Mon, 20 Aug 2018 16:42:09 +0800 (CST)
Subject: [Python-Dev] =?utf-8?q?=E2=80=8Bimprovement_on_shutil=2Emake=5Fa?=
 =?utf-8?q?rchive?=
Message-ID: <29f101a6.db.165567fa2af.Coremail.redstone-cold@163.com>

The current behavior of shutil.make_archive caused many issues , the problem is mainly on the extracted archive directory hierarchy. These are the proofs:

https://stackoverflow.com/questions/51914467/directory-hierarchy-issue-when-using-shutil-make-archive

https://stackoverflow.com/questions/32640053/compressing-directory-using-shutil-make-archive-while-preserving-directory-str

https://stackoverflow.com/questions/41624800/shutil-make-archive-issue-dont-want-directories-included-in-zip-file

https://stackoverflow.com/questions/50156657/unexpected-file-using-shutil-make-archive-to-compress-file
For example , if I want to create a zip archive of the pip package (path specified by pip.__path__[0])), and need a directory named pip to hold the  files and folders which originally reside in the pip package when unpacking the archive, then I need give root_dirparameter of shutil.make_archive the parent directory of the pippackage path (root_dir=Path(pip.__path__[0]).parent), and then the base_dir parameter the final path component of the pip package path(base_dir=Path(pip.__path__[0]).name) , so it is os.path.join(root_dir, base_dir) that specified the directory to archive , so weird !!!
I suggest to change shutil.make_archive(base_name, format[, root_dir[, base_dir]]) to shutil.make_archive(base_name, format[, archived_dir[, archive_prfix]]) where archived_dirdenotes the path to be archived and archive_prfix denotes the common prefix of all files and directories in the archive (it is just a path component and we shouldn?t assume the existence of it on the file system).

If the current behavior of shutil.make_archive won?t be changed , I?d suggest improve its doc, because  so many people couldn?t grasp the use of shutil.make_archive even consulting the doc , these are the proofs:
https://stackoverflow.com/questions/45245079/python-how-to-use-shutil-make-archive
https://stackoverflow.com/questions/30049201/how-to-compress-a-file-with-shutil-make-archive-in-python






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/8439eba7/attachment.html>

From p.f.moore at gmail.com  Mon Aug 20 11:18:57 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 20 Aug 2018 16:18:57 +0100
Subject: [Python-Dev] 
	=?utf-8?q?=E2=80=8Bimprovement_on_shutil=2Emake=5Fa?=
	=?utf-8?q?rchive?=
In-Reply-To: <29f101a6.db.165567fa2af.Coremail.redstone-cold@163.com>
References: <29f101a6.db.165567fa2af.Coremail.redstone-cold@163.com>
Message-ID: <CACac1F8+sF8jNmuTXLBg+r0UdX3iHrvOkNzv1cePj_r_b8zSMA@mail.gmail.com>

On Mon, 20 Aug 2018 at 15:58, Zhao Lee <redstone-cold at 163.com> wrote:
>
> The current behavior of shutil.make_archive caused many issues , the problem is mainly on the extracted archive directory hierarchy. These are the proofs:
>
> https://stackoverflow.com/questions/51914467/directory-hierarchy-issue-when-using-shutil-make-archive
>
> https://stackoverflow.com/questions/32640053/compressing-directory-using-shutil-make-archive-while-preserving-directory-str
>
> https://stackoverflow.com/questions/41624800/shutil-make-archive-issue-dont-want-directories-included-in-zip-file
>
> https://stackoverflow.com/questions/50156657/unexpected-file-using-shutil-make-archive-to-compress-file
> For example , if I want to create a zip archive of the pip package (path specified by pip.__path__[0])), and need a directory named pip to hold the  files and folders which originally reside in the pip package when unpacking the archive, then I need give root_dirparameter of shutil.make_archive the parent directory of the pippackage path (root_dir=Path(pip.__path__[0]).parent), and then the base_dir parameter the final path component of the pip package path(base_dir=Path(pip.__path__[0]).name) , so it is os.path.join(root_dir, base_dir) that specified the directory to archive , so weird !!!
> I suggest to change shutil.make_archive(base_name, format[, root_dir[, base_dir]]) to shutil.make_archive(base_name, format[, archived_dir[, archive_prfix]]) where archived_dirdenotes the path to be archived and archive_prfix denotes the common prefix of all files and directories in the archive (it is just a path component and we shouldn?t assume the existence of it on the file system).
>
> If the current behavior of shutil.make_archive won?t be changed , I?d suggest improve its doc, because  so many people couldn?t grasp the use of shutil.make_archive even consulting the doc , these are the proofs:
> https://stackoverflow.com/questions/45245079/python-how-to-use-shutil-make-archive
> https://stackoverflow.com/questions/30049201/how-to-compress-a-file-with-shutil-make-archive-in-python
>

Thanks for your interest, and for pointing out this problem.
Personally, I don't see that much of an issue with the current
documentation of base_dir and root_dir. But as a native English
speaker with a pretty long history of working with and reading
programming documentation, that's easy for me to say...

I expect that PR suggesting some improvements to the documentation
would be very welcome - in particular, the section would almost
certainly benefit from some examples. If that's something you'd feel
comfortable doing, that would be great.

Paul

From aixtools at felt.demon.nl  Mon Aug 20 11:34:29 2018
From: aixtools at felt.demon.nl (Michael)
Date: Mon, 20 Aug 2018 17:34:29 +0200
Subject: [Python-Dev] [python-committers] Winding down 3.4
In-Reply-To: <CA+3bQGH0AbaWGyCNRHZ3E_TOS_mXGxWi0+SmT6-irZEAuaTqUg@mail.gmail.com>
References: <55df745c-bf49-9849-0e64-b54b797ca0f1@hastings.org>
 <CA+3bQGH0AbaWGyCNRHZ3E_TOS_mXGxWi0+SmT6-irZEAuaTqUg@mail.gmail.com>
Message-ID: <c48d1697-711d-1d78-97f0-72940b18627e@felt.demon.nl>

On 20/08/2018 14:52, Victor Stinner wrote:
>> "shutil copy* unsafe on POSIX - they preserve setuid/setgit bits"
>> https://bugs.python.org/issue17180
> There is no fix. A fix may break the backward compatibility. Is it really
> worth it for the last 3.4 release?
>
My idea would be to focus on a "fix" for 3.8, and then decide if it can,
in one form or another, be backported. And also how far. IMHO - the
discussion about breakage is holding back even an attempt for a
resolution for 3.8.

Michael

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/84829a15/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/84829a15/attachment-0001.sig>

From arj.python at gmail.com  Mon Aug 20 12:32:34 2018
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Mon, 20 Aug 2018 20:32:34 +0400
Subject: [Python-Dev] unsolicited removal request
In-Reply-To: <OFAB5C3A5C.7C8AF52C-ON862582EF.005A7478-862582EF.005A96D3@prominic.net>
References: <CADrxXX=vTRydZpHxXw6zj+ySEmthGLu2HnVcC+LyJH_dcGENeg@mail.gmail.com>
 <OFAB5C3A5C.7C8AF52C-ON862582EF.005A7478-862582EF.005A96D3@prominic.net>
Message-ID: <CADrxXX=fFDMzXkRui4n6R5aA7UuZ_pd-7MAe5DivMqVNDMptUw@mail.gmail.com>

no idea, a mail popped up in my inbox ...

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ
Mauritius

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/206d80fa/attachment.html>

From AHettinger at Prominic.NET  Mon Aug 20 12:29:17 2018
From: AHettinger at Prominic.NET (Andrew M. Hettinger)
Date: Mon, 20 Aug 2018 11:29:17 -0500
Subject: [Python-Dev] unsolicited removal request
In-Reply-To: <CADrxXX=vTRydZpHxXw6zj+ySEmthGLu2HnVcC+LyJH_dcGENeg@mail.gmail.com>
References: <CADrxXX=vTRydZpHxXw6zj+ySEmthGLu2HnVcC+LyJH_dcGENeg@mail.gmail.com>
Message-ID: <OFAB5C3A5C.7C8AF52C-ON862582EF.005A7478-862582EF.005A96D3@prominic.net>


I apologize, that's my IP, and i was trying to remove myself from the list.

How that request got associated with you, I haven't the foggiest. I
certainly was not intended.

Andrew Hettinger
http://Prominic.NET | Skype: AndrewProminic
Tel: 866.339.3169 (toll free) -or- 1.217.356.2888 x. 110 (int'l)
Fax: 866.372.3356 (toll free) -or- 1.217.356.3356            (int'l)



From:	"Abdur-Rahmaan Janhangeer" <arj.python at gmail.com>
To:	"Python Dev" <python-dev at python.org>
Date:	08/20/2018 01:43
Subject:	[Python-Dev] unsolicited removal request
Sent by:	"Python-Dev" <python-dev-bounces
            +ahettinger=prominic.net at python.org>



just notifying that?199.103.2.101
was trying to remove me from this list. might try to remove others

yours,

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ
Mauritius_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ahettinger%40prominic.net

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/8c5310c7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/8c5310c7/attachment.gif>

From storchaka at gmail.com  Mon Aug 20 13:26:16 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 20 Aug 2018 20:26:16 +0300
Subject: [Python-Dev] 
 =?utf-8?q?=E2=80=8Bimprovement_on_shutil=2Emake=5Fa?=
 =?utf-8?q?rchive?=
In-Reply-To: <CACac1F8+sF8jNmuTXLBg+r0UdX3iHrvOkNzv1cePj_r_b8zSMA@mail.gmail.com>
References: <29f101a6.db.165567fa2af.Coremail.redstone-cold@163.com>
 <CACac1F8+sF8jNmuTXLBg+r0UdX3iHrvOkNzv1cePj_r_b8zSMA@mail.gmail.com>
Message-ID: <pletfl$ef7$1@blaine.gmane.org>

20.08.18 18:18, Paul Moore ????:
> I expect that PR suggesting some improvements to the documentation
> would be very welcome - in particular, the section would almost
> certainly benefit from some examples. If that's something you'd feel
> comfortable doing, that would be great.

There is an open documentation issue for shutil.make_archive():
https://bugs.python.org/issue22021


From brett at python.org  Mon Aug 20 13:39:40 2018
From: brett at python.org (Brett Cannon)
Date: Mon, 20 Aug 2018 10:39:40 -0700
Subject: [Python-Dev] unsolicited removal request
In-Reply-To: <CADrxXX=fFDMzXkRui4n6R5aA7UuZ_pd-7MAe5DivMqVNDMptUw@mail.gmail.com>
References: <CADrxXX=vTRydZpHxXw6zj+ySEmthGLu2HnVcC+LyJH_dcGENeg@mail.gmail.com>
 <OFAB5C3A5C.7C8AF52C-ON862582EF.005A7478-862582EF.005A96D3@prominic.net>
 <CADrxXX=fFDMzXkRui4n6R5aA7UuZ_pd-7MAe5DivMqVNDMptUw@mail.gmail.com>
Message-ID: <CAP1=2W63ATsjFixwpwJb1uqEmgyrPqAS2__pMH7BFK_3yO9nBg@mail.gmail.com>

Chances are, Andrew click the unsubscribe link from an email reply that
Abdur-Rahmaan made (this happens to me semi-regularly).

On Mon, 20 Aug 2018 at 09:34 Abdur-Rahmaan Janhangeer <arj.python at gmail.com>
wrote:

> no idea, a mail popped up in my inbox ...
>
>
> Abdur-Rahmaan Janhangeer
> https://github.com/Abdur-rahmaanJ
> Mauritius
>
>> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/f69c7a0b/attachment.html>

From mhroncok at redhat.com  Mon Aug 20 14:31:59 2018
From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=)
Date: Mon, 20 Aug 2018 20:31:59 +0200
Subject: [Python-Dev] Starting to use gcc-8 on upstream Python project CI
In-Reply-To: <CAHE_3Cj=CjMcXJQY3q95u4PRYeEZkNOupr-H80SO9nDnHgqzPw@mail.gmail.com>
References: <CAHE_3Cj=CjMcXJQY3q95u4PRYeEZkNOupr-H80SO9nDnHgqzPw@mail.gmail.com>
Message-ID: <e1fe445d-bc48-ad13-4d30-7e6982e9523c@redhat.com>

On 20.8.2018 20:02, Jun Aruga wrote:
> Dear Python sig.
> 
> Someone can you help to promote for the upstream Python project to use
> gcc-8 on the Travis CI test?
> Right now the project has 4 test cases [1] including defaut gcc
> version 4.8 cases on Travis CI.
> 
> However technically it is possible to use gcc-N (4.8, 5, 6, 7, 8, and
> etc) on Travis CI.
> I think that using the latest version gcc-8 on the upstream project is
> quite beneficial for us.
> Because maybe we Fedora people are working to fix new version gcc's issues.
> When the python project start to use gcc-8, it is easy to share the
> situation publicly outside of Fedora, and of course they can help to
> check the issues.
> As I checked the Python project's .travis.yml, I had no idea about how
> to add gcc-8 case. ;(
> 
> I can show you 2 cases to use the technique as an example. [2][3]
> 
> 
> [1] Python
>    https://travis-ci.org/python/cpython
>    https://github.com/python/cpython/blob/master/.travis.yml
> 
> [2] Ruby
>    https://travis-ci.org/junaruga/ruby/builds/418242410
>    https://github.com/junaruga/ruby/blob/feature/ci-new-gcc/.travis.yml
>    https://github.com/ruby/ruby/pull/1937
> 
> [3] A project I am working as a hobby.
>    https://travis-ci.org/trinityrnaseq/trinityrnaseq
>    https://github.com/trinityrnaseq/trinityrnaseq/blob/master/.travis.yml


I'm taking this to python-dev at python.org which is more appropriate place 
to discuss this. I think Victor is involved in the CIs, is that right?

-- 
Miro Hron?ok
--
Phone: +420777974800
IRC: mhroncok

From larry at hastings.org  Tue Aug 21 01:36:00 2018
From: larry at hastings.org (Larry Hastings)
Date: Mon, 20 Aug 2018 22:36:00 -0700
Subject: [Python-Dev] [python-committers] Winding down 3.4
In-Reply-To: <CA+3bQGH0AbaWGyCNRHZ3E_TOS_mXGxWi0+SmT6-irZEAuaTqUg@mail.gmail.com>
References: <55df745c-bf49-9849-0e64-b54b797ca0f1@hastings.org>
 <CA+3bQGH0AbaWGyCNRHZ3E_TOS_mXGxWi0+SmT6-irZEAuaTqUg@mail.gmail.com>
Message-ID: <c0fd7257-e2d3-c133-5f94-7b99e06cedce@hastings.org>


If they're really all wontfix, maybe we should mark them as wontfix, 
thus giving 3.4 a sendoff worthy of its heroic stature.

Godspeed, and may a flight of angels sing thee to thy rest,


//arry/

On 08/20/2018 05:52 AM, Victor Stinner wrote:
> > "shutil copy* unsafe on POSIX - they preserve setuid/setgit bits"
> > https://bugs.python.org/issue17180
>
> There is no fix. A fix may break the backward compatibility. Is it 
> really worth it for the last 3.4 release?
>
> > "XML vulnerabilities in Python"
> > https://bugs.python.org/issue17239
>
> Bug inactive since 2015. I don't expect that anyone will step in next 
> weeks with a wonderful solution to all XML issues. I suggest to ignore 
> this one as well, this issue is as old as XML support in Python and I 
> am not aware of any victim of these issues.
>
> Obviously, it would be "nice" to see a fix for these issues but it 
> seems like core devs are more interested to work on other topics and 
> other security issues.
>
>
> > "fflush called on pointer to potentially closed file" (Windows only)
> > https://bugs.python.org/issue19050
>
> It seems like two core devs are opposed to fix this issue.
>
> --
>
> There are open security issues on the HTTP server and urllib. I am 
> more concerned by these issues, but it's hard to fix them, there is a 
> risk of introducing regressions.
>
> Victor 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180820/8f19e58a/attachment.html>

From jaruga at redhat.com  Tue Aug 21 05:56:12 2018
From: jaruga at redhat.com (Jun Aruga)
Date: Tue, 21 Aug 2018 11:56:12 +0200
Subject: [Python-Dev] Starting to use gcc-8 on upstream Python project CI
In-Reply-To: <e1fe445d-bc48-ad13-4d30-7e6982e9523c@redhat.com>
References: <CAHE_3Cj=CjMcXJQY3q95u4PRYeEZkNOupr-H80SO9nDnHgqzPw@mail.gmail.com>
 <e1fe445d-bc48-ad13-4d30-7e6982e9523c@redhat.com>
Message-ID: <CAHE_3Ch8StuX5-jCQ_dQc2zxxieFBGm5Lsa3OwU5tHyOHWHozQ@mail.gmail.com>

Hi Miro,

Thanks for forwarding to python-dev.

Jun


On Mon, Aug 20, 2018 at 8:31 PM, Miro Hron?ok <mhroncok at redhat.com> wrote:
> On 20.8.2018 20:02, Jun Aruga wrote:
>>
>> Dear Python sig.
>>
>> Someone can you help to promote for the upstream Python project to use
>> gcc-8 on the Travis CI test?
>> Right now the project has 4 test cases [1] including defaut gcc
>> version 4.8 cases on Travis CI.
>>
>> However technically it is possible to use gcc-N (4.8, 5, 6, 7, 8, and
>> etc) on Travis CI.
>> I think that using the latest version gcc-8 on the upstream project is
>> quite beneficial for us.
>> Because maybe we Fedora people are working to fix new version gcc's
>> issues.
>> When the python project start to use gcc-8, it is easy to share the
>> situation publicly outside of Fedora, and of course they can help to
>> check the issues.
>> As I checked the Python project's .travis.yml, I had no idea about how
>> to add gcc-8 case. ;(
>>
>> I can show you 2 cases to use the technique as an example. [2][3]
>>
>>
>> [1] Python
>>    https://travis-ci.org/python/cpython
>>    https://github.com/python/cpython/blob/master/.travis.yml
>>
>> [2] Ruby
>>    https://travis-ci.org/junaruga/ruby/builds/418242410
>>    https://github.com/junaruga/ruby/blob/feature/ci-new-gcc/.travis.yml
>>    https://github.com/ruby/ruby/pull/1937
>>
>> [3] A project I am working as a hobby.
>>    https://travis-ci.org/trinityrnaseq/trinityrnaseq
>>    https://github.com/trinityrnaseq/trinityrnaseq/blob/master/.travis.yml
>
>
>
> I'm taking this to python-dev at python.org which is more appropriate place to
> discuss this. I think Victor is involved in the CIs, is that right?
>
> --
> Miro Hron?ok
> --
> Phone: +420777974800
> IRC: mhroncok



-- 
Jun Aruga jaruga at redhat.com
IRC: jaruga, Office: TPB(Technology Park Brno) Building C 1F, Brno,
Czech Republic

From arj.python at gmail.com  Tue Aug 21 16:06:42 2018
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Wed, 22 Aug 2018 00:06:42 +0400
Subject: [Python-Dev] Use of Cython
In-Reply-To: <CA+St6D1DNV3j9ieH-M9X0mkMH5mAXKYYOqBeDUWT952ZkCdNyA@mail.gmail.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol>
 <CADiSq7dKDSq7VsejP_FTDrSmvmeRcQmSD5OaVFO9qkwybtJqDg@mail.gmail.com>
 <17ebc01b-e0f2-1ebc-1229-b4ca84843f9c@python.org>
 <pk4e46$q92$1@blaine.gmane.org>
 <CADiSq7cKqgw=-AjDKCQ9QzGCN5yHOA+353cjkZvmdVvYN5qZpw@mail.gmail.com>
 <8CA25A41-D9F4-4634-9509-604F84B09E46@mac.com>
 <CADiSq7f0+CosMD5n2Ybbu90rQcJXq4Ska7KUn9smeaL=1XuMmA@mail.gmail.com>
 <2B17179F-29B4-40E2-824D-749359E33089@mac.com> <pk9oev$eh3$1@blaine.gmane.org>
 <57363ED3-4851-4A5D-A4B9-F9F0A9053F2C@mac.com>
 <CA+St6D1DNV3j9ieH-M9X0mkMH5mAXKYYOqBeDUWT952ZkCdNyA@mail.gmail.com>
Message-ID: <CADrxXXn2ekSvQoZurtjwMHrf=fPsBOAiwaHwjmpeZznkimyzyg@mail.gmail.com>

since when they started working on it?(mypyc)

Abdur-Rahmaan Janhangeer
https://github.com/Abdur-rahmaanJ
Mauritius
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180822/2eff7b69/attachment.html>

From 1520306395 at qq.com  Wed Aug 22 06:40:42 2018
From: 1520306395 at qq.com (=?gb18030?B?wqXP/rfl?=)
Date: Wed, 22 Aug 2018 18:40:42 +0800
Subject: [Python-Dev] Issue?
Message-ID: <tencent_6D5CCEF23A7CB9421BD734680CA6F0534D09@qq.com>

Why compare twice?


class Student(object):


    def __init__(self, name, score):
        self.name = name
        self.score = score


    def __str__(self):
        return '(%s: %s)' % (self.name, self.score)


    __repr__ = __str__


    def __lt__(self, s):
        #print(self, '--', s)
        if(self.score<s.score):
            print(self, '<', s)
            return True
        if(self.score>s.score):
            print(self, '>', s)
            return False
        if(self.score==s.score):
            if(self.name>s.name):
                print(self, '>', s)
                return False
            if(self.name<s.name):
                print(self, '<', s)
                return True
            if(self.name==s.name):
                print(self, '==', s)
                return False
                
    def __eq__(self, s):
        return (self.score == s.score) and (self.name == s.name)
    def __gt__(self, s):
        return not ((self == s) or (self < s))
    def __le__(self, s):
        return ((self == s) or (self < s))
    def __ge__(self, s):
        return ((self == s) or (self > s))
    def __nq__(self, s):
        return not (self == s)


L = [Student('Tim', 22), Student('Bob', 33), Student('Kevin', 11), Student('Alice', 11)]

print(sorted(L))


Output:
(Bob: 33) > (Tim: 22)
(Kevin: 11) < (Bob: 33)
(Kevin: 11) < (Bob: 33)
(Kevin: 11) < (Tim: 22)
(Alice: 11) < (Tim: 22)
(Alice: 11) < (Kevin: 11)
[(Alice: 11), (Kevin: 11), (Tim: 22), (Bob: 33)]


Best regards,
Xiaofeng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180822/708a349e/attachment.html>

From steve at pearwood.info  Wed Aug 22 12:50:15 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 23 Aug 2018 02:50:15 +1000
Subject: [Python-Dev] Issue?
In-Reply-To: <tencent_6D5CCEF23A7CB9421BD734680CA6F0534D09@qq.com>
References: <tencent_6D5CCEF23A7CB9421BD734680CA6F0534D09@qq.com>
Message-ID: <20180822165015.GL24160@ando.pearwood.info>

On Wed, Aug 22, 2018 at 06:40:42PM +0800, ??? wrote:

> Why compare twice?

This is not a mailing list for asking for help with your own code. You 
can try this list instead:

https://mail.python.org/mailman/listinfo/python-list



-- 
Steve

From chris.barker at noaa.gov  Wed Aug 22 12:55:36 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 22 Aug 2018 09:55:36 -0700
Subject: [Python-Dev] Issue?
In-Reply-To: <tencent_6D5CCEF23A7CB9421BD734680CA6F0534D09@qq.com>
References: <tencent_6D5CCEF23A7CB9421BD734680CA6F0534D09@qq.com>
Message-ID: <CALGmxELWx0HArWJ3s4yXixsS20qcEHWjD=BswX5syieCJraovA@mail.gmail.com>

python used the "timsort" sorting routine:

https://en.wikipedia.org/wiki/Timsort

So you can look at that and confirm that this is correct behaviour (I'm
betting it is :-)

But in general, sorting is O(n log(n)) -- there are going to be more than n
comparisons.

If comparing is slow, you want to use a key function, to reduce your
comparison to a simple and fast one:

sorted(L, key=lambda i: (i.name, i.score))

or something like that.

personally, I advocate adding a "key_fun" attribute to classes you want to
make efficiently sortable, so you'd have:

sorted(L, key=Student.key_fun)

There was a discussion on python-ideas about adding a __sort_key__ protocol
to python, but there were too many downsides.

-CHB




On Wed, Aug 22, 2018 at 3:40 AM, ??? <1520306395 at qq.com> wrote:

>
> *Why compare twice?*
>
> class Student(object):
>
>     def __init__(self, name, score):
>         self.name = name
>         self.score = score
>
>     def __str__(self):
>         return '(%s: %s)' % (self.name, self.score)
>
>     __repr__ = __str__
>
>     def __lt__(self, s):
>         #print(self, '--', s)
>         if(self.score<s.score):
>             print(self, '<', s)
>             return True
>         if(self.score>s.score):
>             print(self, '>', s)
>             return False
>         if(self.score==s.score):
>             if(self.name>s.name):
>                 print(self, '>', s)
>                 return False
>             if(self.name<s.name):
>                 print(self, '<', s)
>                 return True
>             if(self.name==s.name):
>                 print(self, '==', s)
>                 return False
>
>     def __eq__(self, s):
>         return (self.score == s.score) and (self.name == s.name)
>     def __gt__(self, s):
>         return not ((self == s) or (self < s))
>     def __le__(self, s):
>         return ((self == s) or (self < s))
>     def __ge__(self, s):
>         return ((self == s) or (self > s))
>     def __nq__(self, s):
>         return not (self == s)
>
> L = [Student('Tim', 22), Student('Bob', 33), Student('Kevin', 11),
> Student('Alice', 11)]
> print(sorted(L))
>
> Output:
> (Bob: 33) > (Tim: 22)
> *(Kevin: 11) < (Bob: 33)*
> *(Kevin: 11) < (Bob: 33)*
> (Kevin: 11) < (Tim: 22)
> (Alice: 11) < (Tim: 22)
> (Alice: 11) < (Kevin: 11)
> [(Alice: 11), (Kevin: 11), (Tim: 22), (Bob: 33)]
>
> *Best regards,*
> *Xiaofeng*
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180822/6807e711/attachment.html>

From brett at python.org  Wed Aug 22 13:02:35 2018
From: brett at python.org (Brett Cannon)
Date: Wed, 22 Aug 2018 10:02:35 -0700
Subject: [Python-Dev] Starting to use gcc-8 on upstream Python project CI
In-Reply-To: <CAHE_3Ch8StuX5-jCQ_dQc2zxxieFBGm5Lsa3OwU5tHyOHWHozQ@mail.gmail.com>
References: <CAHE_3Cj=CjMcXJQY3q95u4PRYeEZkNOupr-H80SO9nDnHgqzPw@mail.gmail.com>
 <e1fe445d-bc48-ad13-4d30-7e6982e9523c@redhat.com>
 <CAHE_3Ch8StuX5-jCQ_dQc2zxxieFBGm5Lsa3OwU5tHyOHWHozQ@mail.gmail.com>
Message-ID: <CAP1=2W5pRT5muNz=Y_oKUAxdbLGBahvq82Y49DmT=Ku2RZG1eg@mail.gmail.com>

So is the request simply to use gcc 8 instead of what Travis uses as the
default gcc? IOW change
https://github.com/python/cpython/blob/28853a249b1d0c890b7e9ca345290bb8c1756446/.travis.yml#L71
somehow to use gcc8 specifically? Since it's only used for the code
coverage build I don't think  anyone would mind if you changed it. Just
open an issue on bugs.python.org and then submit a PR to change it.

On Tue, 21 Aug 2018 at 08:35 Jun Aruga <jaruga at redhat.com> wrote:

> Hi Miro,
>
> Thanks for forwarding to python-dev.
>
> Jun
>
>
> On Mon, Aug 20, 2018 at 8:31 PM, Miro Hron?ok <mhroncok at redhat.com> wrote:
> > On 20.8.2018 20:02, Jun Aruga wrote:
> >>
> >> Dear Python sig.
> >>
> >> Someone can you help to promote for the upstream Python project to use
> >> gcc-8 on the Travis CI test?
> >> Right now the project has 4 test cases [1] including defaut gcc
> >> version 4.8 cases on Travis CI.
> >>
> >> However technically it is possible to use gcc-N (4.8, 5, 6, 7, 8, and
> >> etc) on Travis CI.
> >> I think that using the latest version gcc-8 on the upstream project is
> >> quite beneficial for us.
> >> Because maybe we Fedora people are working to fix new version gcc's
> >> issues.
> >> When the python project start to use gcc-8, it is easy to share the
> >> situation publicly outside of Fedora, and of course they can help to
> >> check the issues.
> >> As I checked the Python project's .travis.yml, I had no idea about how
> >> to add gcc-8 case. ;(
> >>
> >> I can show you 2 cases to use the technique as an example. [2][3]
> >>
> >>
> >> [1] Python
> >>    https://travis-ci.org/python/cpython
> >>    https://github.com/python/cpython/blob/master/.travis.yml
> >>
> >> [2] Ruby
> >>    https://travis-ci.org/junaruga/ruby/builds/418242410
> >>    https://github.com/junaruga/ruby/blob/feature/ci-new-gcc/.travis.yml
> >>    https://github.com/ruby/ruby/pull/1937
> >>
> >> [3] A project I am working as a hobby.
> >>    https://travis-ci.org/trinityrnaseq/trinityrnaseq
> >>
> https://github.com/trinityrnaseq/trinityrnaseq/blob/master/.travis.yml
> >
> >
> >
> > I'm taking this to python-dev at python.org which is more appropriate
> place to
> > discuss this. I think Victor is involved in the CIs, is that right?
> >
> > --
> > Miro Hron?ok
> > --
> > Phone: +420777974800 <+420%20777%20974%20800>
> > IRC: mhroncok
>
>
>
> --
> Jun Aruga jaruga at redhat.com
> IRC: jaruga, Office: TPB(Technology Park Brno) Building C 1F, Brno,
> Czech Republic
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180822/6581facd/attachment.html>

From happy at oh4u.net  Wed Aug 22 13:38:32 2018
From: happy at oh4u.net (=?UTF-8?B?5pyJ6LOA5rezL0p1biBBcnVnYQ==?=)
Date: Wed, 22 Aug 2018 19:38:32 +0200
Subject: [Python-Dev] Starting to use gcc-8 on upstream Python project CI
In-Reply-To: <CAP1=2W5pRT5muNz=Y_oKUAxdbLGBahvq82Y49DmT=Ku2RZG1eg@mail.gmail.com>
References: <CAHE_3Cj=CjMcXJQY3q95u4PRYeEZkNOupr-H80SO9nDnHgqzPw@mail.gmail.com>
 <e1fe445d-bc48-ad13-4d30-7e6982e9523c@redhat.com>
 <CAHE_3Ch8StuX5-jCQ_dQc2zxxieFBGm5Lsa3OwU5tHyOHWHozQ@mail.gmail.com>
 <CAP1=2W5pRT5muNz=Y_oKUAxdbLGBahvq82Y49DmT=Ku2RZG1eg@mail.gmail.com>
Message-ID: <CADfndDfWzTUW=LsmTpqZLOHdtXkV0WneAetiUW7RNgbyrSPjuw@mail.gmail.com>

Hi Brett,

> So is the request simply to use gcc 8 instead of what Travis uses as the default gcc?

Yes, you are right.

> IOW change https://github.com/python/cpython/blob/28853a249b1d0c890b7e9ca345290bb8c1756446/.travis.yml#L71 somehow to use gcc8 specifically?

If we just change only the test case of the gcc version from 4.8
(default) to 8, that's not difficult.
I wanted to know how we implement the case if we need it.

> Since it's only used for the code coverage build I don't think  anyone would mind if you changed it. Just open an issue on bugs.python.org and then submit a PR to change it.

Okay, as I have not join the Python development, I did not know how
the core developers are using the CI.

> Just open an issue on bugs.python.org and then submit a PR to change it.

Sure, later if anyone does not open the ticket, I will do it submitting the PR.

Thanks for the explanation.

Jun

On Wed, 22 Aug 2018 at 19:04, Brett Cannon <brett at python.org> wrote:
>
> So is the request simply to use gcc 8 instead of what Travis uses as the default gcc? IOW change https://github.com/python/cpython/blob/28853a249b1d0c890b7e9ca345290bb8c1756446/.travis.yml#L71 somehow to use gcc8 specifically? Since it's only used for the code coverage build I don't think  anyone would mind if you changed it. Just open an issue on bugs.python.org and then submit a PR to change it.
>
> On Tue, 21 Aug 2018 at 08:35 Jun Aruga <jaruga at redhat.com> wrote:
>>
>> Hi Miro,
>>
>> Thanks for forwarding to python-dev.
>>
>> Jun
>>
>>
>> On Mon, Aug 20, 2018 at 8:31 PM, Miro Hron?ok <mhroncok at redhat.com> wrote:
>> > On 20.8.2018 20:02, Jun Aruga wrote:
>> >>
>> >> Dear Python sig.
>> >>
>> >> Someone can you help to promote for the upstream Python project to use
>> >> gcc-8 on the Travis CI test?
>> >> Right now the project has 4 test cases [1] including defaut gcc
>> >> version 4.8 cases on Travis CI.
>> >>
>> >> However technically it is possible to use gcc-N (4.8, 5, 6, 7, 8, and
>> >> etc) on Travis CI.
>> >> I think that using the latest version gcc-8 on the upstream project is
>> >> quite beneficial for us.
>> >> Because maybe we Fedora people are working to fix new version gcc's
>> >> issues.
>> >> When the python project start to use gcc-8, it is easy to share the
>> >> situation publicly outside of Fedora, and of course they can help to
>> >> check the issues.
>> >> As I checked the Python project's .travis.yml, I had no idea about how
>> >> to add gcc-8 case. ;(
>> >>
>> >> I can show you 2 cases to use the technique as an example. [2][3]
>> >>
>> >>
>> >> [1] Python
>> >>    https://travis-ci.org/python/cpython
>> >>    https://github.com/python/cpython/blob/master/.travis.yml
>> >>
>> >> [2] Ruby
>> >>    https://travis-ci.org/junaruga/ruby/builds/418242410
>> >>    https://github.com/junaruga/ruby/blob/feature/ci-new-gcc/.travis.yml
>> >>    https://github.com/ruby/ruby/pull/1937
>> >>
>> >> [3] A project I am working as a hobby.
>> >>    https://travis-ci.org/trinityrnaseq/trinityrnaseq
>> >>    https://github.com/trinityrnaseq/trinityrnaseq/blob/master/.travis.yml
>> >
>> >
>> >
>> > I'm taking this to python-dev at python.org which is more appropriate place to
>> > discuss this. I think Victor is involved in the CIs, is that right?
>> >
>> > --
>> > Miro Hron?ok
>> > --
>> > Phone: +420777974800
>> > IRC: mhroncok
>>
>>
>>
>> --
>> Jun Aruga jaruga at redhat.com
>> IRC: jaruga, Office: TPB(Technology Park Brno) Building C 1F, Brno,
>> Czech Republic
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/happy%40oh4u.net



-- 
Jun Aruga

From tim.peters at gmail.com  Wed Aug 22 15:49:14 2018
From: tim.peters at gmail.com (Tim Peters)
Date: Wed, 22 Aug 2018 14:49:14 -0500
Subject: [Python-Dev] Issue?
In-Reply-To: <tencent_6D5CCEF23A7CB9421BD734680CA6F0534D09@qq.com>
References: <tencent_6D5CCEF23A7CB9421BD734680CA6F0534D09@qq.com>
Message-ID: <CAExdVN=rWi2qzho2a29Tt2Jxm-6__yp2FtjmkdR1ORxpgb=F0Q@mail.gmail.com>

I wrote Python's sort, so I may know something about it ;-)  To my
eyes, no, there's not "an issue" here, but a full explanation would be
very involved.

For some sorting algorithms, it's possible to guarantee a redundant
comparison is never made.  For example, a pure insertion sort.

But Python's sort is quite complex, using a number of distinct
strategies dynamically adapting to structure found in the data as it
goes along.  There's a possibility for a small bit of redundant work
whenever switching from one strategy to another.  That's typical of
"hybrid" strategies.

Your case involves a switch between Python's sort's two simplest
strategies.  Let's just look at what matters:

[22, 33, 11, 11]

The first step is searching for "a natural run", an initial
sub-sequence that's already sorted.  22 < 33 says the first two
elements are already in order.  Then 11 < 33 says that's the end of
the run.

That part is over now.

The next step is using a binary insertion sort to move 11 into the
already-sorted [22, 33] prefix.  A binary search starts looking "in
the middle" first, but a 2-element run is so short that 33 "is the
middle", so it first compares 11 to 33 _again_.  So it goes.  Code to
detect that special case could be added, but it would probably cost
more than it saves (it always costs time to test for it, but that time
is pure waste unless the tests succeed).

This is specific to an ascending natural run of length 2, so is quite
a special case.  For example, in

[22, 33, 44, 11, 11]

11 < 44 ends the natural run, and then the binary search first
compares 11 to 33 ("the middle" of the 3-element natural run), and no
redundant comparison gets made.  But in

[22, 33, 44. 43, 11]

43 gets compared to 44 again on the _second_ iteration of the binary
search loop.

In general, this particular strategy switch ends up doing at most one
redundant comparison only if the first element after an ascending
natural run belongs immediately before the last element of the run.
For technical reasons, this can happen at most

    min(len(the_list) / 32, number_of_natural_runs_in_the_list)

times, so it's generally trivial in an average-case O(N log N)
process.  It's so rare it would be minor in an O(N) process too,
unless N is very small.

A principled way to avoid this would be to skip the "find a run" step
if N is very small, leaving the whole thing to binary insertion sort.
Then a redundant comparison would never be done for small N.  But that
would end up doing more comparisons _overall_ in common cases where a
short list starts with a relatively (compared to N) sizable ascending
or descending run.

So I'm happy enough with the tradeoffs already in place.


On Wed, Aug 22, 2018 at 10:37 AM ??? <1520306395 at qq.com> wrote:
> Why compare twice?
>
> class Student(object):
>
>     def __init__(self, name, score):
>         self.name = name
>         self.score = score
>
>     def __str__(self):
>         return '(%s: %s)' % (self.name, self.score)
>
>     __repr__ = __str__
>
>     def __lt__(self, s):
>         #print(self, '--', s)
>         if(self.score<s.score):
>             print(self, '<', s)
>             return True
>         if(self.score>s.score):
>             print(self, '>', s)
>             return False
>         if(self.score==s.score):
>             if(self.name>s.name):
>                 print(self, '>', s)
>                 return False
>             if(self.name<s.name):
>                 print(self, '<', s)
>                 return True
>             if(self.name==s.name):
>                 print(self, '==', s)
>                 return False
>
>     def __eq__(self, s):
>         return (self.score == s.score) and (self.name == s.name)
>     def __gt__(self, s):
>         return not ((self == s) or (self < s))
>     def __le__(self, s):
>         return ((self == s) or (self < s))
>     def __ge__(self, s):
>         return ((self == s) or (self > s))
>     def __nq__(self, s):
>         return not (self == s)
>
> L = [Student('Tim', 22), Student('Bob', 33), Student('Kevin', 11), Student('Alice', 11)]
> print(sorted(L))
>
> Output:
> (Bob: 33) > (Tim: 22)
> (Kevin: 11) < (Bob: 33)
> (Kevin: 11) < (Bob: 33)
> (Kevin: 11) < (Tim: 22)
> (Alice: 11) < (Tim: 22)
> (Alice: 11) < (Kevin: 11)
> [(Alice: 11), (Kevin: 11), (Tim: 22), (Bob: 33)]
>
> Best regards,
> Xiaofeng
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tim.peters%40gmail.com

From nas-python at arctrix.com  Wed Aug 22 16:16:52 2018
From: nas-python at arctrix.com (Neil Schemenauer)
Date: Wed, 22 Aug 2018 14:16:52 -0600
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <CA+3bQGGjvgFjh5q=zWKuKsd4rihnJqiQUyRgxAXALRowdO8MrA@mail.gmail.com>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be>
 <CA+3bQGGjvgFjh5q=zWKuKsd4rihnJqiQUyRgxAXALRowdO8MrA@mail.gmail.com>
Message-ID: <20180822201652.awfclycjhgzgoi3f@python.ca>

On 2018-07-31, Victor Stinner wrote:
> I would be nice to be able to use something to "generate" C
> extensions, maybe even from pure Python code. But someone has to
> work on a full solution to implement that.

Perhaps a "argument clinic on steroids" would be the proper
approach.  So, extensions would mostly be written in C.  However, we
would have a pre-processor that does some "magic" to make using the
Python API cleaner.  Defining new types using static structures, for
example, is not the way to build a good API.  The other approach
would be something like mypyc + cffi.  I agree with others that
Cython is not the right tool.

Regards,

  Neil

From greg.ewing at canterbury.ac.nz  Wed Aug 22 21:34:02 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Aug 2018 13:34:02 +1200
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <20180822201652.awfclycjhgzgoi3f@python.ca>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be>
 <CA+3bQGGjvgFjh5q=zWKuKsd4rihnJqiQUyRgxAXALRowdO8MrA@mail.gmail.com>
 <20180822201652.awfclycjhgzgoi3f@python.ca>
Message-ID: <5B7E0F0A.3060202@canterbury.ac.nz>

Neil Schemenauer wrote:
> Perhaps a "argument clinic on steroids" would be the proper
> approach.  So, extensions would mostly be written in C.  However, we
> would have a pre-processor that does some "magic" to make using the
> Python API cleaner.

You seem to have started on the train of thought that
led me to create Pyrex (the precursor to Cython).

It went like this: I was getting frustrated with SWIG
because it wasn't powerful enough to do what I wanted.
I thought about something that would let me write functions
with Python headers and C bodies.

Then I realised that to really make the C API bearable it
would need to take care of refcounting and exception handling,
so the body couldn't just be plain C, it would need some Python
semantics as well. Given that, it seemed more sensible to base
all of the syntax on Python.

Also, I wanted to avoid the problem you get with all
preprocessors, that when something goes wrong you get
error messages in terms of the expanded code rather than
the original source. To fix that I would need to do a
full parsing and type analysis job -- essentially it
would need to be a full-blown compiler in its own right.

So, "argument clinic on steroids" is actually a pretty
good description of Pyrex.

-- 
Greg

From solipsis at pitrou.net  Thu Aug 23 01:36:53 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 23 Aug 2018 07:36:53 +0200
Subject: [Python-Dev] Let's change to C API!
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be>
 <CA+3bQGGjvgFjh5q=zWKuKsd4rihnJqiQUyRgxAXALRowdO8MrA@mail.gmail.com>
 <20180822201652.awfclycjhgzgoi3f@python.ca>
 <5B7E0F0A.3060202@canterbury.ac.nz>
Message-ID: <20180823073653.088ff0c7@fsol>

On Thu, 23 Aug 2018 13:34:02 +1200
Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Neil Schemenauer wrote:
> > Perhaps a "argument clinic on steroids" would be the proper
> > approach.  So, extensions would mostly be written in C.  However, we
> > would have a pre-processor that does some "magic" to make using the
> > Python API cleaner.  
> 
> You seem to have started on the train of thought that
> led me to create Pyrex (the precursor to Cython).
> 
> It went like this: I was getting frustrated with SWIG
> because it wasn't powerful enough to do what I wanted.
> I thought about something that would let me write functions
> with Python headers and C bodies.
> 
> Then I realised that to really make the C API bearable it
> would need to take care of refcounting and exception handling,
> so the body couldn't just be plain C, it would need some Python
> semantics as well. Given that, it seemed more sensible to base
> all of the syntax on Python.
> 
> Also, I wanted to avoid the problem you get with all
> preprocessors, that when something goes wrong you get
> error messages in terms of the expanded code rather than
> the original source. To fix that I would need to do a
> full parsing and type analysis job -- essentially it
> would need to be a full-blown compiler in its own right.
> 
> So, "argument clinic on steroids" is actually a pretty
> good description of Pyrex.

Agreed, at least abstractly, Cython is the best existing solutiong to
the problem.

The issues to solve for Cython to be usable for the CPython stdlib are
IMHO the following:

- the bootstrap problem (Cython self-compiles with CPython)
- the dependency / versioning problem (Cython is a large quick-evolving
  third-party package that we can't decently vendor)
- the maintenance problem (how do ensure we can change small things in
  the C API, especially semi-private ones, without having to submit PRs
  to Cython as well)
- the debugging problem (Cython's generated C code is unreadable,
  unlike Argument Clinic's, which can make debugging annoying)

Regards

Antoine.



From J.Demeyer at UGent.be  Thu Aug 23 02:07:08 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Thu, 23 Aug 2018 08:07:08 +0200
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <f708d94e0e0b4680834393a29334c320@xmail103.UGent.be>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be>
 <CA+3bQGGjvgFjh5q=zWKuKsd4rihnJqiQUyRgxAXALRowdO8MrA@mail.gmail.com>
 <20180822201652.awfclycjhgzgoi3f@python.ca>
 <5B7E0F0A.3060202@canterbury.ac.nz>
 <f708d94e0e0b4680834393a29334c320@xmail103.UGent.be>
Message-ID: <5B7E4F0C.9000809@UGent.be>

On 2018-08-23 07:36, Antoine Pitrou wrote:
> - the bootstrap problem (Cython self-compiles with CPython)

This is not a big problem: we can make sure that all stdlib dependencies 
of Cython either have PEP 399 pure Python implementations or we keep 
them in pure C.

> - the dependency / versioning problem (Cython is a large quick-evolving
>    third-party package that we can't decently vendor)

Is that a real problem? You're sort of doing the same thing with pip 
already.

> - the maintenance problem (how do ensure we can change small things in
>    the C API, especially semi-private ones, without having to submit PRs
>    to Cython as well)

Why don't you want to submit PRs to Cython? If you're saying "I don't 
want to wait for the next stable release of Cython", you could use 
development versions of Cython for development versions of CPython.

> - the debugging problem (Cython's generated C code is unreadable,
>    unlike Argument Clinic's, which can make debugging annoying)

Luckily, you don't need to read the C code most of the time. And it's 
also a matter of experience: I can read Cython-generated C code just fine.


Jeroen.


From solipsis at pitrou.net  Thu Aug 23 03:04:32 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 23 Aug 2018 09:04:32 +0200
Subject: [Python-Dev] Let's change to C API!
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be>
 <CA+3bQGGjvgFjh5q=zWKuKsd4rihnJqiQUyRgxAXALRowdO8MrA@mail.gmail.com>
 <20180822201652.awfclycjhgzgoi3f@python.ca>
 <5B7E0F0A.3060202@canterbury.ac.nz>
 <f708d94e0e0b4680834393a29334c320@xmail103.UGent.be>
 <5B7E4F0C.9000809@UGent.be>
Message-ID: <20180823090432.28bc1a16@fsol>

On Thu, 23 Aug 2018 08:07:08 +0200
Jeroen Demeyer <J.Demeyer at UGent.be> wrote:
> 
> > - the dependency / versioning problem (Cython is a large quick-evolving
> >    third-party package that we can't decently vendor)  
> 
> Is that a real problem? You're sort of doing the same thing with pip 
> already.

Building and developing CPython doesn't depend on pip at all.  The bundled
pip copy is just an inert pack of files, as far as we are concerned.

> > - the maintenance problem (how do ensure we can change small things in
> >    the C API, especially semi-private ones, without having to submit PRs
> >    to Cython as well)  
> 
> Why don't you want to submit PRs to Cython?

Because it forces a much longer cycle time when we want to introduce a
change in the C API: first prototype the C API change, then notice it
breaks Cython, then try to make a PR (which isn't trivial, given
Cython's size), then wait for the PR to be merged and a new Cython to
be released.

> If you're saying "I don't 
> want to wait for the next stable release of Cython", you could use 
> development versions of Cython for development versions of CPython.

But depending on the development version of a compiler isn't very
enticing, IMHO.

> > - the debugging problem (Cython's generated C code is unreadable,
> >    unlike Argument Clinic's, which can make debugging annoying)  
> 
> Luckily, you don't need to read the C code most of the time. And it's 
> also a matter of experience: I can read Cython-generated C code just fine.

Let's be serious here.  Regardless of the experience, nobody enjoys
reading / stepping through code like the following:

  __Pyx_TraceLine(206,0,__PYX_ERR(1, 206, __pyx_L1_error))
  __Pyx_XDECREF(__pyx_r);
  __pyx_t_2 = __Pyx_GetModuleGlobalName(__pyx_n_s_datetime); if (unlikely(!__pyx_t_2)) __PYX_ERR(1, 206, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_2);
  __pyx_t_3 = __Pyx_PyObject_GetAttrStr(__pyx_t_2, __pyx_n_s_datetime); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 206, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_3);
  __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
  __pyx_t_2 = __Pyx_PyObject_GetAttrStr(__pyx_t_3, __pyx_n_s_utcfromtimestamp); if (unlikely(!__pyx_t_2)) __PYX_ERR(1, 206, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_2);
  __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
  __pyx_t_3 = __Pyx_PyFloat_DivideObjC(__pyx_v_x, __pyx_float_1e3, 1e3, 0); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 206, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_3);
  __pyx_t_4 = NULL;
  if (CYTHON_UNPACK_METHODS && likely(PyMethod_Check(__pyx_t_2))) {
    __pyx_t_4 = PyMethod_GET_SELF(__pyx_t_2);
    if (likely(__pyx_t_4)) {
      PyObject* function = PyMethod_GET_FUNCTION(__pyx_t_2);
      __Pyx_INCREF(__pyx_t_4);
      __Pyx_INCREF(function);
      __Pyx_DECREF_SET(__pyx_t_2, function);
    }
  }
  if (!__pyx_t_4) {
    __pyx_t_1 = __Pyx_PyObject_CallOneArg(__pyx_t_2, __pyx_t_3); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 206, __pyx_L1_error)
    __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
    __Pyx_GOTREF(__pyx_t_1);
  } else {
    #if CYTHON_FAST_PYCALL
    if (PyFunction_Check(__pyx_t_2)) {
      PyObject *__pyx_temp[2] = {__pyx_t_4, __pyx_t_3};
      __pyx_t_1 = __Pyx_PyFunction_FastCall(__pyx_t_2, __pyx_temp+1-1, 1+1); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 206, __pyx_L1_error)
      __Pyx_XDECREF(__pyx_t_4); __pyx_t_4 = 0;
      __Pyx_GOTREF(__pyx_t_1);
      __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
    } else
    #endif
    #if CYTHON_FAST_PYCCALL
    if (__Pyx_PyFastCFunction_Check(__pyx_t_2)) {
      PyObject *__pyx_temp[2] = {__pyx_t_4, __pyx_t_3};
      __pyx_t_1 = __Pyx_PyCFunction_FastCall(__pyx_t_2, __pyx_temp+1-1, 1+1); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 206, __pyx_L1_error)
      __Pyx_XDECREF(__pyx_t_4); __pyx_t_4 = 0;
      __Pyx_GOTREF(__pyx_t_1);
      __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
    } else
    #endif
    {
      __pyx_t_5 = PyTuple_New(1+1); if (unlikely(!__pyx_t_5)) __PYX_ERR(1, 206, __pyx_L1_error)
      __Pyx_GOTREF(__pyx_t_5);
      __Pyx_GIVEREF(__pyx_t_4); PyTuple_SET_ITEM(__pyx_t_5, 0, __pyx_t_4); __pyx_t_4 = NULL;
      __Pyx_GIVEREF(__pyx_t_3);
      PyTuple_SET_ITEM(__pyx_t_5, 0+1, __pyx_t_3);
      __pyx_t_3 = 0;
      __pyx_t_1 = __Pyx_PyObject_Call(__pyx_t_2, __pyx_t_5, NULL); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 206, __pyx_L1_error)
      __Pyx_GOTREF(__pyx_t_1);
      __Pyx_DECREF(__pyx_t_5); __pyx_t_5 = 0;
    }
  }


Regards

Antoine.



From cmawebsite at gmail.com  Thu Aug 23 14:53:16 2018
From: cmawebsite at gmail.com (Collin Anderson)
Date: Thu, 23 Aug 2018 14:53:16 -0400
Subject: [Python-Dev] Python 2.7 EOL date
Message-ID: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>

Hi All,

Sorry if this has been mentioned before, but I noticed the Python 2.7 EOL
date was recently set to Jan 1st, 2020.

My understanding was Python releases get 5 years of support from their
initial release, and Python 2.7 was extended an additional 5 years.

Python 2.7 was originally released on 2010-07-03, and with an original EOL
of 2015-07-03. Extended 5 years, shouldn't the EOL be 2020-07-03?

Also, this statement is a little unclear to me:

> Specifically, 2.7 will receive bugfix support until January 1, 2020. All
2.7 development work will cease in 2020.

This statement makes it sound like bugfixes end on Jan 1st, but seems to
leave open the possibility that security fixes could continue through the
year.

Thanks!
Collin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180823/1be01499/attachment.html>

From mariatta.wijaya at gmail.com  Thu Aug 23 15:57:07 2018
From: mariatta.wijaya at gmail.com (Mariatta Wijaya)
Date: Thu, 23 Aug 2018 12:57:07 -0700
Subject: [Python-Dev] Python 2.7 EOL date
In-Reply-To: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
References: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
Message-ID: <CAGbohna_N1KdT215OJGDXDnFgitHS5A-0wrfpxQPw6zWeWGNiQ@mail.gmail.com>

No more security fixes after Jan 1, 2020.
It is the end of Python 2.7.



On Thu, Aug 23, 2018, 12:47 PM Collin Anderson <cmawebsite at gmail.com> wrote:

> Hi All,
>
> Sorry if this has been mentioned before, but I noticed the Python 2.7 EOL
> date was recently set to Jan 1st, 2020.
>
> My understanding was Python releases get 5 years of support from their
> initial release, and Python 2.7 was extended an additional 5 years.
>
> Python 2.7 was originally released on 2010-07-03, and with an original EOL
> of 2015-07-03. Extended 5 years, shouldn't the EOL be 2020-07-03?
>
> Also, this statement is a little unclear to me:
>
> > Specifically, 2.7 will receive bugfix support until January 1, 2020. All
> 2.7 development work will cease in 2020.
>
> This statement makes it sound like bugfixes end on Jan 1st, but seems to
> leave open the possibility that security fixes could continue through the
> year.
>
> Thanks!
> Collin
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mariatta.wijaya%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180823/8ac8cefb/attachment.html>

From stefan_ml at behnel.de  Thu Aug 23 16:15:35 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Aug 2018 22:15:35 +0200
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <20180823090432.28bc1a16@fsol>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be>
 <CA+3bQGGjvgFjh5q=zWKuKsd4rihnJqiQUyRgxAXALRowdO8MrA@mail.gmail.com>
 <20180822201652.awfclycjhgzgoi3f@python.ca>
 <5B7E0F0A.3060202@canterbury.ac.nz>
 <f708d94e0e0b4680834393a29334c320@xmail103.UGent.be>
 <5B7E4F0C.9000809@UGent.be> <20180823090432.28bc1a16@fsol>
Message-ID: <pln4h5$j4m$1@blaine.gmane.org>

Antoine Pitrou schrieb am 23.08.2018 um 09:04:
> On Thu, 23 Aug 2018 08:07:08 +0200
> Jeroen Demeyer wrote:
>>> - the maintenance problem (how do ensure we can change small things in
>>>    the C API, especially semi-private ones, without having to submit PRs
>>>    to Cython as well)  
>>
>> Why don't you want to submit PRs to Cython?
> 
> Because it forces a much longer cycle time when we want to introduce a
> change in the C API: first prototype the C API change, then notice it
> breaks Cython, then try to make a PR (which isn't trivial, given
> Cython's size), then wait for the PR to be merged and a new Cython to
> be released.

I think you can put that argument back into the attic. When CPython 3.6 and
3.7 came out, I swear I had already forgotten which new features they
provided, because we had implemented and released most of the major
features 6-12 months earlier in Cython (backported to Py2.6). And it has
happened more than once that we pushed out a point release within a few
days to fix a real need on user side.

What I would rather like to see instead is that both of our sides try to
jointly discuss ideas for C-API changes, so that we don't even run into the
problem that changes we made on one side surprisingly break the other.

Don't forget that the spark for this whole discussion was to make it easier
to change the C-API at all. Being able to change Cython in one place and
then adapt a whole bunch of real world extensions out there by simply
regenerating their C code with it is a really cool feature. Basically, it
passes the ability to do that back into your own hands.


>> If you're saying "I don't 
>> want to wait for the next stable release of Cython", you could use 
>> development versions of Cython for development versions of CPython.
> 
> But depending on the development version of a compiler isn't very
> enticing, IMHO.

In case that need arises, feel free to ask which git revision we recommend
for use in CPython. In the worst case, we can always create a stable branch
for you that makes sure we don't break your productivity while we're doing
our thing.


>>> - the debugging problem (Cython's generated C code is unreadable,
>>>    unlike Argument Clinic's, which can make debugging annoying)  
>>
>> Luckily, you don't need to read the C code most of the time. And it's 
>> also a matter of experience: I can read Cython-generated C code just fine.
> 
> Let's be serious here.  Regardless of the experience, nobody enjoys
> reading / stepping through code like the following:

Ok, you posted generated C code, let's read it together.


>   __Pyx_TraceLine(206,0,__PYX_ERR(1, 206, __pyx_L1_error))

This shows that you have enabled the generation of line tracing code with
the directive "linetrace=True", and Cython is translating line 206 of one
of your source modules here.


>   __Pyx_XDECREF(__pyx_r);
>   __pyx_t_2 = __Pyx_GetModuleGlobalName(__pyx_n_s_datetime); if (unlikely(!__pyx_t_2)) __PYX_ERR(1, 206, __pyx_L1_error)
>   __Pyx_GOTREF(__pyx_t_2);
>   __pyx_t_3 = __Pyx_PyObject_GetAttrStr(__pyx_t_2, __pyx_n_s_datetime); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 206, __pyx_L1_error)
>   __Pyx_GOTREF(__pyx_t_3);
>   __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
>   __pyx_t_2 = __Pyx_PyObject_GetAttrStr(__pyx_t_3, __pyx_n_s_utcfromtimestamp); if (unlikely(!__pyx_t_2)) __PYX_ERR(1, 206, __pyx_L1_error)
>   __Pyx_GOTREF(__pyx_t_2);
>   __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;

This implements "datetime.datetime.utcfromtimestamp", probably used in a
"return" statement.


>   __pyx_t_3 = __Pyx_PyFloat_DivideObjC(__pyx_v_x, __pyx_float_1e3, 1e3, 0); if (unlikely(!__pyx_t_3)) __PYX_ERR(1, 206, __pyx_L1_error)
>   __Pyx_GOTREF(__pyx_t_3);

This is "x/1e3", optimised for fast computation in the case that "x" turns
out to be a number, especially a float object.


>   __pyx_t_4 = NULL;
>   if (CYTHON_UNPACK_METHODS && likely(PyMethod_Check(__pyx_t_2))) {
>     __pyx_t_4 = PyMethod_GET_SELF(__pyx_t_2);
>     if (likely(__pyx_t_4)) {
>       PyObject* function = PyMethod_GET_FUNCTION(__pyx_t_2);
>       __Pyx_INCREF(__pyx_t_4);
>       __Pyx_INCREF(function);
>       __Pyx_DECREF_SET(__pyx_t_2, function);
>     }
>   }
>   if (!__pyx_t_4) {
>     __pyx_t_1 = __Pyx_PyObject_CallOneArg(__pyx_t_2, __pyx_t_3); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 206, __pyx_L1_error)
>     __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
>     __Pyx_GOTREF(__pyx_t_1);
>   } else {
>     #if CYTHON_FAST_PYCALL
>     if (PyFunction_Check(__pyx_t_2)) {
>       PyObject *__pyx_temp[2] = {__pyx_t_4, __pyx_t_3};
>       __pyx_t_1 = __Pyx_PyFunction_FastCall(__pyx_t_2, __pyx_temp+1-1, 1+1); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 206, __pyx_L1_error)
>       __Pyx_XDECREF(__pyx_t_4); __pyx_t_4 = 0;
>       __Pyx_GOTREF(__pyx_t_1);
>       __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
>     } else
>     #endif
>     #if CYTHON_FAST_PYCCALL
>     if (__Pyx_PyFastCFunction_Check(__pyx_t_2)) {
>       PyObject *__pyx_temp[2] = {__pyx_t_4, __pyx_t_3};
>       __pyx_t_1 = __Pyx_PyCFunction_FastCall(__pyx_t_2, __pyx_temp+1-1, 1+1); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 206, __pyx_L1_error)
>       __Pyx_XDECREF(__pyx_t_4); __pyx_t_4 = 0;
>       __Pyx_GOTREF(__pyx_t_1);
>       __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
>     } else
>     #endif
>     {
>       __pyx_t_5 = PyTuple_New(1+1); if (unlikely(!__pyx_t_5)) __PYX_ERR(1, 206, __pyx_L1_error)
>       __Pyx_GOTREF(__pyx_t_5);
>       __Pyx_GIVEREF(__pyx_t_4); PyTuple_SET_ITEM(__pyx_t_5, 0, __pyx_t_4); __pyx_t_4 = NULL;
>       __Pyx_GIVEREF(__pyx_t_3);
>       PyTuple_SET_ITEM(__pyx_t_5, 0+1, __pyx_t_3);
>       __pyx_t_3 = 0;
>       __pyx_t_1 = __Pyx_PyObject_Call(__pyx_t_2, __pyx_t_5, NULL); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 206, __pyx_L1_error)
>       __Pyx_GOTREF(__pyx_t_1);
>       __Pyx_DECREF(__pyx_t_5); __pyx_t_5 = 0;
>     }
>   }

And this is an optimised, inlined version of the method call "tmp2(tmp3)",
using the FASTCALL protocol if available, where tmp2 is the
"datetime.datetime.utcfromtimestamp" above and tmp3 is "x/1e3". You
probably have your reasons for calculating that. ;)

This code would probably be simplified if PEP 580 was accepted, although we
would want to keep it around for a while in order to avoid a performance
regression in older Python versions. The C compile time feature switches
like "CYTHON_FAST_PYCALL" or "CYTHON_UNPACK_METHODS" are one of the reasons
why Cython is so versatile and adaptive, also for end-users. If any of
these features break or if any CPython implementation detail isn't
available on a certain Python implementation (that tries to implement to
C-API), you can switch off the usage of that feature in the C code that
Cython has generated for you by simply defining a C macro at C compile
time. We do that automatically for PyPy and Pyston, but also for older
CPython versions that lack certain C-API features.

Sometimes there is an actual reason behind what looks like complexity at
first sight. :)

What you stripped in your example was the fact that Cython generates a C
code comment with the exact original source code line right above this C
code section, which helps a lot in understanding what goes on. And then
there's "cython -a", which gives you the whole thing nicely visualised as
an HTML file. And you get Cython source level profiling and code coverage
reporting, which is a huge raise in comfort compared to hand written C code.

Jeroen is right, in almost all cases, you really do not care what exactly
the C code is, even as an expert. It's perfectly enough to let Cython do
its thing. If you want to improve or otherwise modify the C code that
Cython generates, then yes, you certainly want to look at the code it
generates first. But in all other cases, you want to use Cython *because*
you then don't have to look at the C code. That's a huge feature, too.

Stefan


From stefan_ml at behnel.de  Thu Aug 23 16:18:46 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Aug 2018 22:18:46 +0200
Subject: [Python-Dev] Let's change to C API!
In-Reply-To: <5B7E0F0A.3060202@canterbury.ac.nz>
References: <CA+3bQGFzxdS=eBOyMWT-yHnP4pV=EukkbYKue6nPaAKk4713=w@mail.gmail.com>
 <20180730110120.6d03e6d8@fsol>
 <CA+3bQGGHjPJv6aL4t0T7iy9DGAc1xXSm+b_qxawEv=E2H_zoWw@mail.gmail.com>
 <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be>
 <5B600F47.3090503@UGent.be>
 <CA+3bQGGjvgFjh5q=zWKuKsd4rihnJqiQUyRgxAXALRowdO8MrA@mail.gmail.com>
 <20180822201652.awfclycjhgzgoi3f@python.ca>
 <5B7E0F0A.3060202@canterbury.ac.nz>
Message-ID: <pln4n3$1h1$1@blaine.gmane.org>

Greg Ewing schrieb am 23.08.2018 um 03:34:
> Neil Schemenauer wrote:
>> Perhaps a "argument clinic on steroids" would be the proper
>> approach.? So, extensions would mostly be written in C.? However, we
>> would have a pre-processor that does some "magic" to make using the
>> Python API cleaner.
> 
> You seem to have started on the train of thought that
> led me to create Pyrex (the precursor to Cython).

Greg, thank you so much for doing that. It's a great design that we and
hoards of Cython users out there continue to benefit from.

Stefan


From vstinner at redhat.com  Thu Aug 23 16:30:35 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 23 Aug 2018 22:30:35 +0200
Subject: [Python-Dev] Python 2.7 EOL date
In-Reply-To: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
References: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
Message-ID: <CA+3bQGGC6Z2wkV91BmePpFH77hW0U=N-CZKuP-ysYJAHm3AgEQ@mail.gmail.com>

Hi,

The reference is the PEP 373 "Python 2.7 Release Schedule". See the
"Update" section:
https://www.python.org/dev/peps/pep-0373/#update

Victor

2018-08-23 20:53 GMT+02:00 Collin Anderson <cmawebsite at gmail.com>:
> Hi All,
>
> Sorry if this has been mentioned before, but I noticed the Python 2.7 EOL
> date was recently set to Jan 1st, 2020.
>
> My understanding was Python releases get 5 years of support from their
> initial release, and Python 2.7 was extended an additional 5 years.
>
> Python 2.7 was originally released on 2010-07-03, and with an original EOL
> of 2015-07-03. Extended 5 years, shouldn't the EOL be 2020-07-03?
>
> Also, this statement is a little unclear to me:
>
>> Specifically, 2.7 will receive bugfix support until January 1, 2020. All
>> 2.7 development work will cease in 2020.
>
> This statement makes it sound like bugfixes end on Jan 1st, but seems to
> leave open the possibility that security fixes could continue through the
> year.
>
> Thanks!
> Collin
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
>

From eric at trueblade.com  Thu Aug 23 18:23:19 2018
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 23 Aug 2018 18:23:19 -0400
Subject: [Python-Dev] Python 2.7 EOL date
In-Reply-To: <CA+3bQGGC6Z2wkV91BmePpFH77hW0U=N-CZKuP-ysYJAHm3AgEQ@mail.gmail.com>
References: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
 <CA+3bQGGC6Z2wkV91BmePpFH77hW0U=N-CZKuP-ysYJAHm3AgEQ@mail.gmail.com>
Message-ID: <3e4fa912-7912-9163-22f1-e9cbbadee282@trueblade.com>

On 8/23/2018 4:30 PM, Victor Stinner wrote:
> Hi,
> 
> The reference is the PEP 373 "Python 2.7 Release Schedule". See the
> "Update" section:
> https://www.python.org/dev/peps/pep-0373/#update

We could probably make it more clear in this section and/or in 
https://www.python.org/dev/peps/pep-0373/#id4 that there will be no 2.7 
releases of any kind after 2020-01-01: no bugfix, no security, and no 
source releases after that date.

I'll whip something up if there's general agreement.

Eric

> 
> Victor
> 
> 2018-08-23 20:53 GMT+02:00 Collin Anderson <cmawebsite at gmail.com>:
>> Hi All,
>>
>> Sorry if this has been mentioned before, but I noticed the Python 2.7 EOL
>> date was recently set to Jan 1st, 2020.
>>
>> My understanding was Python releases get 5 years of support from their
>> initial release, and Python 2.7 was extended an additional 5 years.
>>
>> Python 2.7 was originally released on 2010-07-03, and with an original EOL
>> of 2015-07-03. Extended 5 years, shouldn't the EOL be 2020-07-03?
>>
>> Also, this statement is a little unclear to me:
>>
>>> Specifically, 2.7 will receive bugfix support until January 1, 2020. All
>>> 2.7 development work will cease in 2020.
>>
>> This statement makes it sound like bugfixes end on Jan 1st, but seems to
>> leave open the possibility that security fixes could continue through the
>> year.
>>
>> Thanks!
>> Collin
>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
>>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com
> 

From barry at python.org  Thu Aug 23 20:14:01 2018
From: barry at python.org (Barry Warsaw)
Date: Thu, 23 Aug 2018 17:14:01 -0700
Subject: [Python-Dev] Python 2.7 EOL date
In-Reply-To: <3e4fa912-7912-9163-22f1-e9cbbadee282@trueblade.com>
References: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
 <CA+3bQGGC6Z2wkV91BmePpFH77hW0U=N-CZKuP-ysYJAHm3AgEQ@mail.gmail.com>
 <3e4fa912-7912-9163-22f1-e9cbbadee282@trueblade.com>
Message-ID: <84AB3D60-FAC3-42C6-A34B-8B4423405C3B@python.org>

On Aug 23, 2018, at 15:23, Eric V. Smith <eric at trueblade.com> wrote:
> 
> On 8/23/2018 4:30 PM, Victor Stinner wrote:
>> Hi,
>> The reference is the PEP 373 "Python 2.7 Release Schedule". See the
>> "Update" section:
>> https://www.python.org/dev/peps/pep-0373/#update
> 
> We could probably make it more clear in this section and/or in https://www.python.org/dev/peps/pep-0373/#id4 that there will be no 2.7 releases of any kind after 2020-01-01: no bugfix, no security, and no source releases after that date.
> 
> I'll whip something up if there's general agreement.

+1 from me, but IIRC, Benjamin has said that the final 2.7 release may not happen *exactly* on January 1st (due to holidays and such), but that we should still consider that the ?hammer date? when all support stops.  We should get some confirmation from Benjamin (the 2.7 release manager) on the (tentative) exact final release date, and then codify that in the PEP.  I?d be very happy if that, or say December 31 2019 were the actual last release date.

Cheers,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180823/c2a9d059/attachment.sig>

From eric at trueblade.com  Thu Aug 23 20:48:37 2018
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 23 Aug 2018 20:48:37 -0400
Subject: [Python-Dev] Python 2.7 EOL date
In-Reply-To: <84AB3D60-FAC3-42C6-A34B-8B4423405C3B@python.org>
References: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
 <CA+3bQGGC6Z2wkV91BmePpFH77hW0U=N-CZKuP-ysYJAHm3AgEQ@mail.gmail.com>
 <3e4fa912-7912-9163-22f1-e9cbbadee282@trueblade.com>
 <84AB3D60-FAC3-42C6-A34B-8B4423405C3B@python.org>
Message-ID: <a9f7a8d1-a3a0-7a2c-8129-37a756c9c3da@trueblade.com>

On 8/23/2018 8:14 PM, Barry Warsaw wrote:
> On Aug 23, 2018, at 15:23, Eric V. Smith <eric at trueblade.com> wrote:
>>
>> On 8/23/2018 4:30 PM, Victor Stinner wrote:
>>> Hi,
>>> The reference is the PEP 373 "Python 2.7 Release Schedule". See the
>>> "Update" section:
>>> https://www.python.org/dev/peps/pep-0373/#update
>>
>> We could probably make it more clear in this section and/or in https://www.python.org/dev/peps/pep-0373/#id4 that there will be no 2.7 releases of any kind after 2020-01-01: no bugfix, no security, and no source releases after that date.
>>
>> I'll whip something up if there's general agreement.
> 
> +1 from me, but IIRC, Benjamin has said that the final 2.7 release may not happen *exactly* on January 1st (due to holidays and such), but that we should still consider that the ?hammer date? when all support stops.  We should get some confirmation from Benjamin (the 2.7 release manager) on the (tentative) exact final release date, and then codify that in the PEP.  I?d be very happy if that, or say December 31 2019 were the actual last release date.

Agreed on getting Benjamin's blessing. We can discuss it in a few weeks.

Eric


From tjreedy at udel.edu  Fri Aug 24 04:19:19 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 24 Aug 2018 04:19:19 -0400
Subject: [Python-Dev] Python 2.7 EOL date
In-Reply-To: <84AB3D60-FAC3-42C6-A34B-8B4423405C3B@python.org>
References: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
 <CA+3bQGGC6Z2wkV91BmePpFH77hW0U=N-CZKuP-ysYJAHm3AgEQ@mail.gmail.com>
 <3e4fa912-7912-9163-22f1-e9cbbadee282@trueblade.com>
 <84AB3D60-FAC3-42C6-A34B-8B4423405C3B@python.org>
Message-ID: <ploeu5$3nf$1@blaine.gmane.org>

On 8/23/2018 8:14 PM, Barry Warsaw wrote:
> On Aug 23, 2018, at 15:23, Eric V. Smith <eric at trueblade.com> wrote:
>>
>> On 8/23/2018 4:30 PM, Victor Stinner wrote:
>>> Hi,
>>> The reference is the PEP 373 "Python 2.7 Release Schedule". See the
>>> "Update" section:
>>> https://www.python.org/dev/peps/pep-0373/#update
>>
>> We could probably make it more clear in this section and/or in https://www.python.org/dev/peps/pep-0373/#id4 that there will be no 2.7 releases of any kind after 2020-01-01: no bugfix, no security, and no source releases after that date.
>>
>> I'll whip something up if there's general agreement.
> 
> +1 from me, but IIRC, Benjamin has said that the final 2.7 release may not happen *exactly* on January 1st (due to holidays and such), but that we should still consider that the ?hammer date? when all support stops.  We should get some confirmation from Benjamin (the 2.7 release manager) on the (tentative) exact final release date, and then codify that in the PEP.  I?d be very happy if that, or say December 31 2019 were the actual last release date.

Current: "Being the last of the 2.x series, 2.7 will have an extended 
period of maintenance. Specifically, 2.7 will receive bugfix support 
until January 1, 2020. All 2.7 development work will cease in 2020."

Proposed, based on what Benjamin said last discussion: "Being the last 
of the 2.x series, 2.7 will have an extended period of bugfix and 
security maintenance until January 1, 2020. The final 2.7 release will 
be on or soon after that date."

Anything more specific is up to Benjamin.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Fri Aug 24 04:39:19 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 24 Aug 2018 04:39:19 -0400
Subject: [Python-Dev] Python 2.7 EOL date
In-Reply-To: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
References: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
Message-ID: <plog3m$35c$1@blaine.gmane.org>

On 8/23/2018 2:53 PM, Collin Anderson wrote:
> Hi All,
> 
> Sorry if this has been mentioned before, but I noticed the Python 2.7 
> EOL date was recently set to Jan 1st, 2020.
> 
> My understanding was Python releases get 5 years of support from their 
> initial release,

'5' is a rounded, rather than exact, figure.

  and Python 2.7 was extended an additional 5 years.
> 
> Python 2.7 was originally released on?2010-07-03, and with an original 
> EOL of 2015-07-03. Extended 5 years, shouldn't the EOL be 2020-07-03?

Think of it this way.  In April 2015, after almost 5 years, support was 
extended for about 5 years, (primarily for build and security issues) to 
sometime in 2020, left intentionally vague.  Based on more information, 
we recently specified the first day in 2020.  Backporting of normal 
bugfixes is already very selective, and the selectivity may increase.

> Also, this statement is a little unclear to me:
> 
>  >?Specifically, 2.7 will receive bugfix support until January 1, 2020. 
> All 2.7 development work will cease in 2020.
> 
> This statement makes it sound like bugfixes end on Jan 1st, but seems to 
> leave open the possibility that security fixes could continue through 
> the year.

I agree that it is not clear that we actually meant 'final release on or 
soon after 2020 Jan 1".  In other words, no substantive patches of any 
kind after that date.


-- 
Terry Jan Reedy



From aixtools at felt.demon.nl  Fri Aug 24 06:20:01 2018
From: aixtools at felt.demon.nl (Michael)
Date: Fri, 24 Aug 2018 12:20:01 +0200
Subject: [Python-Dev] make patchcheck and git path
Message-ID: <5f9b58cd-224a-6851-0bd0-54a2731aa9e4@felt.demon.nl>

I am trying to be a 'good scout' and run "make patchcheck" more
regularly. However, I generally am not successful because I build and
test in separate directories.

There is access to git! just no ready reference in the build area.

So, not calling it a bug - but if someone else also experiences this,
and feels capable of makeing it more flexible - you will get thanks from
me, in any case!

Michael


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180824/50a38336/attachment.sig>

From status at bugs.python.org  Fri Aug 24 12:09:40 2018
From: status at bugs.python.org (Python tracker)
Date: Fri, 24 Aug 2018 18:09:40 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20180824160940.CA558116B3C@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2018-08-17 - 2018-08-24)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6807 (+31)
  closed 39431 (+36)
  total  46238 (+67)

Open issues with patches: 2707 


Issues opened (48)
==================

#34421: Cannot install package with unicode module names on Windows
https://bugs.python.org/issue34421  opened by julien.malard

#34422: __name__ not available for classes in typing module
https://bugs.python.org/issue34422  opened by Gabriel Tremblay

#34423: Overflow when casting from double to time_t, and_PyTime_t.
https://bugs.python.org/issue34423  opened by enedil

#34424: Unicode names break email header
https://bugs.python.org/issue34424  opened by _savage

#34426: "__lltrace__"  seems to be wrong , "__ltrace__" is correct in 
https://bugs.python.org/issue34426  opened by Nojima Takahide

#34427: calling MutableSequence.extend on self produces infinite loop
https://bugs.python.org/issue34427  opened by Naris R

#34428: tokenize
https://bugs.python.org/issue34428  opened by lkcl

#34429: On Windows tempfile.TemporaryFile behaves like NamedTemporaryF
https://bugs.python.org/issue34429  opened by dwich

#34430: Symmetrical chaining futures in asyncio.future.wrap_future
https://bugs.python.org/issue34430  opened by huji

#34431: Docs does not eval allows code object as argument
https://bugs.python.org/issue34431  opened by jfine2358

#34433: cancel all other pending child futures
https://bugs.python.org/issue34433  opened by tzongw

#34434: Removal of kwargs for built-in types not covered with "changed
https://bugs.python.org/issue34434  opened by andymaier

#34439: Expose venv --prompt value to an environment value
https://bugs.python.org/issue34439  opened by Tomer

#34442: zlib module not built on windows
https://bugs.python.org/issue34442  opened by steveire

#34443: enum repr should use __qualname__
https://bugs.python.org/issue34443  opened by doerwalter

#34444: Module's __file__ should be absolute always ("." in sys.path)
https://bugs.python.org/issue34444  opened by blueyed

#34445: asyncio support in doctests
https://bugs.python.org/issue34445  opened by Stefan Tjarks

#34446: ambiguous _max_size parameter in SpooledTemporaryFile
https://bugs.python.org/issue34446  opened by jcc2220

#34447: ttk.TreeView (and maybe other functions) is overzealous in con
https://bugs.python.org/issue34447  opened by abarnert

#34448: incomplete/missing message from configure.ac for usable wchar_
https://bugs.python.org/issue34448  opened by michael-o

#34449: HP aCC complains about invalid -fPIC on HP-UX
https://bugs.python.org/issue34449  opened by michael-o

#34450: improve  shutil.make_archive
https://bugs.python.org/issue34450  opened by redstone-cold

#34451: docs: tutorial/introduction doesn't mention toggle of prompts
https://bugs.python.org/issue34451  opened by jfine2358

#34453: ipaddress module accepts some strange IPv6 addresses that it s
https://bugs.python.org/issue34453  opened by lkcl

#34455: Tkinter crashing when pressing Command + ^  (OSX)
https://bugs.python.org/issue34455  opened by iPodClassic

#34458: No way to alternate options
https://bugs.python.org/issue34458  opened by porton

#34459: email.contentmanager should use IANA encoding
https://bugs.python.org/issue34459  opened by era

#34460: email.charset: common IANA labels missing
https://bugs.python.org/issue34460  opened by era

#34461: Availability of parsers in etree initializer
https://bugs.python.org/issue34461  opened by nilanjan roy

#34463: Discrepancy between traceback.print_exception and sys.__except
https://bugs.python.org/issue34463  opened by Timothy McCurrach

#34464: There are inconsitencies in the treatment of True, False, None
https://bugs.python.org/issue34464  opened by jfine2358

#34465: ipaddress should accept bytearray in addition to bytes
https://bugs.python.org/issue34465  opened by joernheissler

#34466: socket.settimeout working incorrectly for connect() method of 
https://bugs.python.org/issue34466  opened by Slon

#34467: No mechanism to abort created coroutine or suppress not-awaite
https://bugs.python.org/issue34467  opened by Sheng Zhong

#34470: windows msi in headless mode fails to install Script directory
https://bugs.python.org/issue34470  opened by cdknorow

#34472: zipfile: does not include optional descriptor signature
https://bugs.python.org/issue34472  opened by silas

#34475: functools.partial objects have no __qualname__ attribute
https://bugs.python.org/issue34475  opened by chris.jerdonek

#34476: asyncio.sleep(0) not documented
https://bugs.python.org/issue34476  opened by hniksic

#34478: Possibly misleading/wrong documentation about PyModuleDef.m_fr
https://bugs.python.org/issue34478  opened by Segev Finer

#34479: ArgumentParser subparser error display at the wrong level
https://bugs.python.org/issue34479  opened by siming85

#34480: _markupbase.py fails with UnboundLocalError on invalid keyword
https://bugs.python.org/issue34480  opened by kodial

#34481: Different behavior of C and Python impls of datetime.strftime 
https://bugs.python.org/issue34481  opened by izbyshev

#34482: datetime: Tests for potential crashes due to non-UTF-8-encodab
https://bugs.python.org/issue34482  opened by izbyshev

#34483: eval() raises NameError: name '...' is not defined
https://bugs.python.org/issue34483  opened by alexb

#34484: Unicode HOWTO incorrectly refers to Private Use Area for surro
https://bugs.python.org/issue34484  opened by mark.dickinson

#34485: _PyCoreConfig: add stdio_encoding and stdio_errors
https://bugs.python.org/issue34485  opened by vstinner

#34486: "RuntimeError: release unlocked lock" when starting a thread
https://bugs.python.org/issue34486  opened by Stanis??aw Skonieczny (Uosiu)

#34487: enum _sunder_ names mix metaclass and enum class attributes
https://bugs.python.org/issue34487  opened by Gerrit.Holl



Most recent 15 issues with no replies (15)
==========================================

#34487: enum _sunder_ names mix metaclass and enum class attributes
https://bugs.python.org/issue34487

#34485: _PyCoreConfig: add stdio_encoding and stdio_errors
https://bugs.python.org/issue34485

#34482: datetime: Tests for potential crashes due to non-UTF-8-encodab
https://bugs.python.org/issue34482

#34481: Different behavior of C and Python impls of datetime.strftime 
https://bugs.python.org/issue34481

#34479: ArgumentParser subparser error display at the wrong level
https://bugs.python.org/issue34479

#34478: Possibly misleading/wrong documentation about PyModuleDef.m_fr
https://bugs.python.org/issue34478

#34475: functools.partial objects have no __qualname__ attribute
https://bugs.python.org/issue34475

#34472: zipfile: does not include optional descriptor signature
https://bugs.python.org/issue34472

#34463: Discrepancy between traceback.print_exception and sys.__except
https://bugs.python.org/issue34463

#34461: Availability of parsers in etree initializer
https://bugs.python.org/issue34461

#34460: email.charset: common IANA labels missing
https://bugs.python.org/issue34460

#34451: docs: tutorial/introduction doesn't mention toggle of prompts
https://bugs.python.org/issue34451

#34449: HP aCC complains about invalid -fPIC on HP-UX
https://bugs.python.org/issue34449

#34447: ttk.TreeView (and maybe other functions) is overzealous in con
https://bugs.python.org/issue34447

#34445: asyncio support in doctests
https://bugs.python.org/issue34445



Most recent 15 issues waiting for review (15)
=============================================

#34485: _PyCoreConfig: add stdio_encoding and stdio_errors
https://bugs.python.org/issue34485

#34482: datetime: Tests for potential crashes due to non-UTF-8-encodab
https://bugs.python.org/issue34482

#34472: zipfile: does not include optional descriptor signature
https://bugs.python.org/issue34472

#34461: Availability of parsers in etree initializer
https://bugs.python.org/issue34461

#34449: HP aCC complains about invalid -fPIC on HP-UX
https://bugs.python.org/issue34449

#34448: incomplete/missing message from configure.ac for usable wchar_
https://bugs.python.org/issue34448

#34434: Removal of kwargs for built-in types not covered with "changed
https://bugs.python.org/issue34434

#34433: cancel all other pending child futures
https://bugs.python.org/issue34433

#34430: Symmetrical chaining futures in asyncio.future.wrap_future
https://bugs.python.org/issue34430

#34427: calling MutableSequence.extend on self produces infinite loop
https://bugs.python.org/issue34427

#34426: "__lltrace__"  seems to be wrong , "__ltrace__" is correct in 
https://bugs.python.org/issue34426

#34424: Unicode names break email header
https://bugs.python.org/issue34424

#34423: Overflow when casting from double to time_t, and_PyTime_t.
https://bugs.python.org/issue34423

#34421: Cannot install package with unicode module names on Windows
https://bugs.python.org/issue34421

#34408: possible null pointer dereference in pystate.c
https://bugs.python.org/issue34408



Top 10 most discussed issues (10)
=================================

#34207: test_cmd_line test_utf8_mode test_warnings fail in all FreeBSD
https://bugs.python.org/issue34207   9 msgs

#34455: Tkinter crashing when pressing Command + ^  (OSX)
https://bugs.python.org/issue34455   9 msgs

#26544: platform.libc_ver() returns incorrect version number
https://bugs.python.org/issue26544   7 msgs

#34428: tokenize
https://bugs.python.org/issue34428   6 msgs

#34434: Removal of kwargs for built-in types not covered with "changed
https://bugs.python.org/issue34434   6 msgs

#34347: AIX: test_utf8_mode.test_cmd_line fails
https://bugs.python.org/issue34347   5 msgs

#34410: itertools.tee not thread-safe; can segfault interpreter when w
https://bugs.python.org/issue34410   5 msgs

#34217: windows: cross compilation fails due to headers with uppercase
https://bugs.python.org/issue34217   4 msgs

#34431: Docs does not eval allows code object as argument
https://bugs.python.org/issue34431   4 msgs

#6700: inspect.getsource() returns incorrect source lines at the modu
https://bugs.python.org/issue6700   3 msgs



Issues closed (35)
==================

#2122: mmap.flush does not check for errors on windows
https://bugs.python.org/issue2122  closed by berker.peksag

#5999: compile error on HP-UX 11.22 ia64 - 'mbstate_t' is used	as a t
https://bugs.python.org/issue5999  closed by vstinner

#18925: select.poll.modify is not documented
https://bugs.python.org/issue18925  closed by berker.peksag

#22057: The doc say all globals are copied on eval(), but only __built
https://bugs.python.org/issue22057  closed by berker.peksag

#22602: UTF-7 codec decodes ill-formed sequences starting with "+"
https://bugs.python.org/issue22602  closed by serhiy.storchaka

#26363: __builtins__ propagation is misleading described in exec and e
https://bugs.python.org/issue26363  closed by berker.peksag

#27088: doc: select: epoll.poll: incorrect timeout units, missing maxe
https://bugs.python.org/issue27088  closed by berker.peksag

#30411: git doesn't support "-C" args under 1.8.5 occurs in configure.
https://bugs.python.org/issue30411  closed by xiang.zhang

#34170: Py_Initialize(): computing path configuration must not have si
https://bugs.python.org/issue34170  closed by vstinner

#34318: Convert deprecated behavior of assertRaises() etc into errors
https://bugs.python.org/issue34318  closed by serhiy.storchaka

#34381: Make tests with outbound connection optional
https://bugs.python.org/issue34381  closed by terry.reedy

#34412: strsignal(3) does not exist on HP-UX
https://bugs.python.org/issue34412  closed by berker.peksag

#34413: Porting section of 3.5 "What's New" doc does not mention bytec
https://bugs.python.org/issue34413  closed by eric.snow

#34415: Typo in logging.Formatter docstring
https://bugs.python.org/issue34415  closed by vinay.sajip

#34416: PyCapsule_Import seems to release the GIL without acquiring it
https://bugs.python.org/issue34416  closed by brett.cannon

#34417: imp.find_module reacts badly to iterator
https://bugs.python.org/issue34417  closed by eric.snow

#34425: :s formatting broken for objects without __format__
https://bugs.python.org/issue34425  closed by martin.panter

#34432: doc Mention complex and decimal.Decimal on str.format note abo
https://bugs.python.org/issue34432  closed by eric.smith

#34435: Missing NULL check in unicode_encode_ucs1()
https://bugs.python.org/issue34435  closed by serhiy.storchaka

#34436: Overallocation is never disabled in _PyBytes_FormatEx()
https://bugs.python.org/issue34436  closed by izbyshev

#34437: print statement using \x results in improper and extra bytes
https://bugs.python.org/issue34437  closed by steven.daprano

#34438: do_handshake stuck in ssl.py
https://bugs.python.org/issue34438  closed by christian.heimes

#34440: Certificate verify failed (works fine in 3.6)
https://bugs.python.org/issue34440  closed by christian.heimes

#34441: NULL dereference when issubclass() is called on a class with b
https://bugs.python.org/issue34441  closed by izbyshev

#34452: docs: help text for [>>>] toggle always in English
https://bugs.python.org/issue34452  closed by Mariatta

#34454: datetime: NULL dereference in fromisoformat() on PyUnicode_AsU
https://bugs.python.org/issue34454  closed by taleinat

#34456: pickle: Missing NULL check in save_global()
https://bugs.python.org/issue34456  closed by izbyshev

#34457: Missing NULL check in alias_for_import_name() from Python/ast.
https://bugs.python.org/issue34457  closed by serhiy.storchaka

#34462: _xxsubinterpreters: Wrong NULL check in _copy_raw_string()
https://bugs.python.org/issue34462  closed by berker.peksag

#34468: An always-false condition in range_repr() from Objects/rangeob
https://bugs.python.org/issue34468  closed by benjamin.peterson

#34469: windows msi in headless mode fails to install Script directory
https://bugs.python.org/issue34469  closed by cdknorow

#34471: _datetime: Missing NULL check in tzinfo_from_isoformat_results
https://bugs.python.org/issue34471  closed by benjamin.peterson

#34473: pip is unusable on Windows 10 1803 - SHA-1 certificates are un
https://bugs.python.org/issue34473  closed by mtuttle

#34474: Python/bltinmodule.c: Missing NULL check in builtin_sum_impl()
https://bugs.python.org/issue34474  closed by benjamin.peterson

#34477: Objects/typeobject.c: Missing NULL check in type_init()
https://bugs.python.org/issue34477  closed by benjamin.peterson

From mariatta.wijaya at gmail.com  Fri Aug 24 13:37:21 2018
From: mariatta.wijaya at gmail.com (Mariatta Wijaya)
Date: Fri, 24 Aug 2018 10:37:21 -0700
Subject: [Python-Dev] make patchcheck and git path
In-Reply-To: <5f9b58cd-224a-6851-0bd0-54a2731aa9e4@felt.demon.nl>
References: <5f9b58cd-224a-6851-0bd0-54a2731aa9e4@felt.demon.nl>
Message-ID: <CAGbohnZXazy1PxA39mxzzxrRhp9n_8ouUpDz7CZr_f2VXzCKMw@mail.gmail.com>

I don't quite understand the problem you're facing with git and make
patchcheck?

Also, perhaps this is more for core-workflow instead of python-dev.

Mariatta

?

On Fri, Aug 24, 2018 at 3:20 AM Michael <aixtools at felt.demon.nl> wrote:

> I am trying to be a 'good scout' and run "make patchcheck" more
> regularly. However, I generally am not successful because I build and
> test in separate directories.
>
> There is access to git! just no ready reference in the build area.
>
> So, not calling it a bug - but if someone else also experiences this,
> and feels capable of makeing it more flexible - you will get thanks from
> me, in any case!
>
> Michael
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mariatta.wijaya%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180824/c2408cf6/attachment.html>

From berker.peksag at gmail.com  Fri Aug 24 14:08:09 2018
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Fri, 24 Aug 2018 21:08:09 +0300
Subject: [Python-Dev] make patchcheck and git path
In-Reply-To: <5f9b58cd-224a-6851-0bd0-54a2731aa9e4@felt.demon.nl>
References: <5f9b58cd-224a-6851-0bd0-54a2731aa9e4@felt.demon.nl>
Message-ID: <CAF4280J2YM8_jw0fO-v3rgCrOML8U6NnuaU5Re8SUY3MCty99g@mail.gmail.com>

On Fri, Aug 24, 2018 at 1:22 PM Michael <aixtools at felt.demon.nl> wrote:
>
> I am trying to be a 'good scout' and run "make patchcheck" more
> regularly. However, I generally am not successful because I build and
> test in separate directories.

There is an open issue about supporting out-of-tree builds:
https://bugs.python.org/issue32256

--Berker

From benjamin at python.org  Sat Aug 25 14:50:33 2018
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 25 Aug 2018 11:50:33 -0700
Subject: [Python-Dev] Python 2.7 EOL date
In-Reply-To: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
References: <CAHp18wVsuAo_ev3PQtO13a6FAfXgxXmwE6wOmLxDsogT-k_BSw@mail.gmail.com>
Message-ID: <1535223033.3211797.1486041560.6B4E95EE@webmail.messagingengine.com>

I was operating under the optimistic assumption whatever the precise time of 2.7's official demise would only be an amusing piece of trivia for a world of happy Python 3 users.

It's still to early to promise exact release dates; that will depend on the day-to-day schedules of the release manager and binary builders circa January 2020. A conservative assumption is that no 2.7 changes that land after December 31 2019 will ever be released.

We could make the last release of 2.7 in July 2020. But what does that buy anyone?

On Thu, Aug 23, 2018, at 11:53, Collin Anderson wrote:
> Hi All,
> 
> Sorry if this has been mentioned before, but I noticed the Python 2.7 EOL
> date was recently set to Jan 1st, 2020.
> 
> My understanding was Python releases get 5 years of support from their
> initial release, and Python 2.7 was extended an additional 5 years.
> 
> Python 2.7 was originally released on 2010-07-03, and with an original EOL
> of 2015-07-03. Extended 5 years, shouldn't the EOL be 2020-07-03?
> 
> Also, this statement is a little unclear to me:
> 
> > Specifically, 2.7 will receive bugfix support until January 1, 2020. All
> 2.7 development work will cease in 2020.
> 
> This statement makes it sound like bugfixes end on Jan 1st, but seems to
> leave open the possibility that security fixes could continue through the
> year.
> 
> Thanks!
> Collin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org

From jeremy.kloth at gmail.com  Sat Aug 25 16:47:31 2018
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Sat, 25 Aug 2018 14:47:31 -0600
Subject: [Python-Dev] [Python-checkins] bpo-34171: Prevent creating
 Lib/trace.cover when run the trace module. (GH-8841)
In-Reply-To: <41y8rt4ZQVzFrcd@mail.python.org>
References: <41y8rt4ZQVzFrcd@mail.python.org>
Message-ID: <CAGvrs3LsCmerfJuedWShYjhwSfF_bmW-QC2cxfjQaA+2Di1NZA@mail.gmail.com>

On Sat, Aug 25, 2018 at 1:28 AM Serhiy Storchaka
<webhook-mailer at python.org> wrote:
>
> https://github.com/python/cpython/commit/c406d5cd74002964a64c3eb7d9e2445a7fd3a03f
> commit: c406d5cd74002964a64c3eb7d9e2445a7fd3a03f
> branch: master
> author: Serhiy Storchaka <storchaka at gmail.com>
> committer: GitHub <noreply at github.com>
> date: 2018-08-25T10:27:55+03:00
> summary:
>
> bpo-34171: Prevent creating Lib/trace.cover when run the trace module. (GH-8841)
>
> files:
> A Misc/NEWS.d/next/Library/2018-08-21-00-29-01.bpo-34171.6LkWav.rst
> M Lib/test/test_trace.py
> M Lib/trace.py

This change seems to have caused most buildbots to go red.

From xiang.zhu at outlook.com  Sat Aug 25 19:09:10 2018
From: xiang.zhu at outlook.com (ZHU Xiang)
Date: Sat, 25 Aug 2018 23:09:10 +0000
Subject: [Python-Dev] Python REPL doesn't work on Windows over remote
 powershell session (winrm)
Message-ID: <SN6PR03MB3535A0670D3477274BC1E404EC350@SN6PR03MB3535.namprd03.prod.outlook.com>

Hello dears Python devs,

I'm taking the initiative of writing to you for a question on Python REPL over Windows remote powershell session (winrm).

As we?ve all known, Python REPL works well on local Linux, local Windows, and remote SSH session.

But for the remote Windows powershell session the REPL doesn?t work, when I type ??python? on the remote session, there?s nothing happened.

===================================
Steps to reproduce

# 1/ pre-install python on server1 (server 1 is a windows os)
# 2/ from a powershell console on server0, type below 2 commands:
enter-pssession server1
python

Expected behavior
# The python >>> prompt appears

Actual behavior
# Nothing, it is still the powershell prompt
===================================

The problem impacts all the python versions and all the windows versions.

This make me (and other windows guys) unable to use Python remotely, especially for the Windows Server Core version, which is a headless version (no GUI, so no remote desktop connection), the only way to connect to them is by the remote powershell session.

You can imagine the panic if Python REPL doesn?t work over SSH for Linux.

Could you please kindly have a look, and tell at least why it doesn?t work ? Thanks.

FYI, I?ve also opened a issue on Microsoft Powershell GitHub :
https://github.com/PowerShell/PowerShell/issues/7581

Regards,

Xiang ZHU


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180825/ebd2c0c1/attachment.html>

From rosuav at gmail.com  Sat Aug 25 21:03:18 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 26 Aug 2018 11:03:18 +1000
Subject: [Python-Dev] Python REPL doesn't work on Windows over remote
 powershell session (winrm)
In-Reply-To: <SN6PR03MB3535A0670D3477274BC1E404EC350@SN6PR03MB3535.namprd03.prod.outlook.com>
References: <SN6PR03MB3535A0670D3477274BC1E404EC350@SN6PR03MB3535.namprd03.prod.outlook.com>
Message-ID: <CAPTjJmqY8vfF0UFu_1-dWmJsoB=s2GUDUkTuiJJY5Qr1FGE-RA@mail.gmail.com>

On Sun, Aug 26, 2018 at 9:09 AM, ZHU Xiang <xiang.zhu at outlook.com> wrote:
> But for the remote Windows powershell session the REPL doesn?t work, when I
> type ??python? on the remote session, there?s nothing happened.
>
> # 1/ pre-install python on server1 (server 1 is a windows os)
> # 2/ from a powershell console on server0, type below 2 commands:
> enter-pssession server1
> python
>

Quick check: With the exact same servers, after you enter-possession,
can you run a Python *script* successfully? And/or can you run:

python -c "print(1+2)"

? If so, it's definitely just a REPL problem.

> The problem impacts all the python versions and all the windows versions.

Which versions were tested?

ChrisA

From db3l.net at gmail.com  Sat Aug 25 22:12:48 2018
From: db3l.net at gmail.com (David Bolen)
Date: Sat, 25 Aug 2018 22:12:48 -0400
Subject: [Python-Dev] Python REPL doesn't work on Windows over remote
 powershell session (winrm)
References: <SN6PR03MB3535A0670D3477274BC1E404EC350@SN6PR03MB3535.namprd03.prod.outlook.com>
Message-ID: <m27ekddgf3.fsf@valheru.db3l.homeip.net>

ZHU Xiang <xiang.zhu at outlook.com> writes:

> ===================================
> Steps to reproduce
>
> # 1/ pre-install python on server1 (server 1 is a windows os)
> # 2/ from a powershell console on server0, type below 2 commands:
> enter-pssession server1
> python
>
> Expected behavior
> # The python >>> prompt appears
>
> Actual behavior
> # Nothing, it is still the powershell prompt
> ===================================

Still the powershell prompt or nothing at all?  If the latter, try using
"python -i" instead.  The "-i" will force interactive mode if stdin
isn't otherwise detected as interactive (under the covers, isatty() is
false for stdin), which is where I believe the issue is.

I've used that under Windows ssh sessions (though with cygwin rather
than powershell as well as some of my own remoting tools) for as long as
I can remember (certainly back to XP and Python 2.x - maybe 1.x) for an
interactive prompt when operating without a local windows console.

I'm not sure if there's any better way for Python to detect a remote
shell as being interactive under Windows that would cover such cases.
Perhaps some of the newer pty changes I read Microsoft is making might
help, assuming it flows through to the isatty() test.

-- David


From steve at pearwood.info  Sun Aug 26 06:34:26 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 26 Aug 2018 20:34:26 +1000
Subject: [Python-Dev] Python REPL doesn't work on Windows over remote
 powershell session (winrm)
In-Reply-To: <SN6PR03MB3535A0670D3477274BC1E404EC350@SN6PR03MB3535.namprd03.prod.outlook.com>
References: <SN6PR03MB3535A0670D3477274BC1E404EC350@SN6PR03MB3535.namprd03.prod.outlook.com>
Message-ID: <20180826103426.GZ24160@ando.pearwood.info>

On Sat, Aug 25, 2018 at 11:09:10PM +0000, ZHU Xiang wrote:

> But for the remote Windows powershell session the REPL doesn?t work, 
> when I type ??python? on the remote session, there?s nothing happened.
[...]
> FYI, I?ve also opened a issue on Microsoft Powershell GitHub :
> https://github.com/PowerShell/PowerShell/issues/7581

If I have understood this correctly:

https://github.com/PowerShell/PowerShell/issues/7295

and the explanation here:

https://stackoverflow.com/questions/21466372/execution-of-interactive-command-in-remote-powershell-session-not-working/21466434#21466434

I think this is a limitation of Powershell, not Python.

But I am not a Windows admin and don't really understand Powershell, so 
I could be wrong.



-- 
Steve

From aixtools at felt.demon.nl  Sun Aug 26 17:15:18 2018
From: aixtools at felt.demon.nl (Michael Felt (aixtools))
Date: Sun, 26 Aug 2018 23:15:18 +0200
Subject: [Python-Dev] make patchcheck and git path
In-Reply-To: <CAGbohnZXazy1PxA39mxzzxrRhp9n_8ouUpDz7CZr_f2VXzCKMw@mail.gmail.com>
References: <5f9b58cd-224a-6851-0bd0-54a2731aa9e4@felt.demon.nl>
 <CAGbohnZXazy1PxA39mxzzxrRhp9n_8ouUpDz7CZr_f2VXzCKMw@mail.gmail.com>
Message-ID: <F8F7381B-EAC5-47EA-8153-7F45512A8ECB@felt.demon.nl>

When building out of tree there is no .git reference. If I understand the process it uses git to see what files have changed, and does further processing on those. 

As to workflow, that may be better, but other than the name, unknown to me. 

Sent from my iPhone

> On 24 Aug 2018, at 19:37, Mariatta Wijaya <mariatta.wijaya at gmail.com> wrote:
> 
> I don't quite understand the problem you're facing with git and make patchcheck?
> 
> Also, perhaps this is more for core-workflow instead of python-dev.
> 
> Mariatta
> 
> ?
> 
>> On Fri, Aug 24, 2018 at 3:20 AM Michael <aixtools at felt.demon.nl> wrote:
>> I am trying to be a 'good scout' and run "make patchcheck" more
>> regularly. However, I generally am not successful because I build and
>> test in separate directories.
>> 
>> There is access to git! just no ready reference in the build area.
>> 
>> So, not calling it a bug - but if someone else also experiences this,
>> and feels capable of makeing it more flexible - you will get thanks from
>> me, in any case!
>> 
>> Michael
>> 
>> 
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mariatta.wijaya%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180826/848e7be0/attachment.html>

From aixtools at felt.demon.nl  Mon Aug 27 06:34:29 2018
From: aixtools at felt.demon.nl (Michael)
Date: Mon, 27 Aug 2018 12:34:29 +0200
Subject: [Python-Dev] AIX, test_ssl in particular,
 but also AIX failed tests in general - getting to the 'STABLE' side
Message-ID: <0ee0ac2c-99a6-0ac1-5359-1c7a70874eee@felt.demon.nl>

Dear all,

Last week I experimented with test_ssl. My expectation was that test
failures was caused by the openssl.base provided by IBM for AIX not
having a default certificate file (CApath). However, that is not the case.

The tests that fail are similar to:
self.assertRaisesRegex(ssl.SSLError, "PEM lib")

I started out by testing with something as:
if not AIX:
??? with self.assertRaisesRegex(ssl.SSLError, "PEM lib"):
??????? ctx.load_cert_chain(BADCERT)
else:
??? with self.assertRaises(ssl.SSLError):
??????? ctx.load_cert_chain(BADCERT)

This is after an analysis where I saw that calls too SSL were returning
an non-success status (!= 1) while ERR_peek_last_error() regularly
returned 0. Hence, the frequent 'AssertionError: "PEM lib" does not
match "unknown error ...' with "unknown error" the string Python provides.

While above might remove the 'fail messages' it did not satisfy me. So,
I downloaded openssl (1.0.2p) and compiled - with no optimization! And
now, even from Python3.6 I see:

test_ssl passed in 1 min 23 sec

== Tests result: SUCCESS ==

1 test OK.

In short, the failures of test_ssl may be ignored - as far as raising an
exception goes.

a) I am running a bot for Python, and once the argument
"-with-openssl=/opt/aixtools" is added my bot will stop showing these
errors. I mention this so that it is clear why they suddenly disappear
on my bot (but not elsewhere). Also to alert Python-Dev that the AIX
platform, regarding ssl.py, _ssl.c and test_ssl.py functions 'stable'
but is not as friendly when it comes to saying why WHEN (my guess) a
heavily optimized (I am thinking -O3 to -O5) library is used.

b) With this feedback - MAYBE - the team from IBM might review the way
they package openssl and make sure the messages are visible via
ERR_peek_last_error() et al. Ideally, IBM will notice and work on it
without prompting. One can dream :)

c) In the meantime - I am curious to know what this 'proof' means to
Python-Dev.

I have a simple goal - work through the tests that AIX has been failing
historically and figure out why they fail and fix the tests. To that end
I have submitted several PR's - starting back In January, then nothing
as noone ever seemed to notice, and the last weeks several additional
ones. Victor has been kind enough to say he will look at the tests as he
has time (and back from vacation). But we are all, or most, working on
our time. My goal, rephrased, is to see AIX in the 'stable' column so
that when a test fails it is because there is a regression that needs
addressing - either in the test or in the proposed code change. So I
would be grateful if others were also looking.

I am not trying to re-invent the wheel and will not be surprised if my
'test fix' is not done in the 'Python' way. I'll learn over time - but
this calls for instructive (and critical) comments. "bij voorbaat dank"
aka Thanks in Advance.

So, hoping this helps - I'll continue as I can. Time and resources are
limited. And, I am very curious re: point c) above.

Great Days! everyone,

Michael


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180827/e3363d6f/attachment.sig>

From vstinner at redhat.com  Mon Aug 27 09:26:09 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 27 Aug 2018 15:26:09 +0200
Subject: [Python-Dev] [Python-checkins] bpo-34171: Prevent creating
 Lib/trace.cover when run the trace module. (GH-8841)
In-Reply-To: <CAGvrs3LsCmerfJuedWShYjhwSfF_bmW-QC2cxfjQaA+2Di1NZA@mail.gmail.com>
References: <41y8rt4ZQVzFrcd@mail.python.org>
 <CAGvrs3LsCmerfJuedWShYjhwSfF_bmW-QC2cxfjQaA+2Di1NZA@mail.gmail.com>
Message-ID: <CA+3bQGGK1d8U52nYt7ZRyBAhVVATGvNDhqQRKV_AbPD6ZA5J9Q@mail.gmail.com>

Jemery: would you mind to revert this commit since nobody fixed the
buildbot since 2 days?
https://pythondev.readthedocs.io/ci.html#revert-on-fail

Victor
Le sam. 25 ao?t 2018 ? 22:50, Jeremy Kloth <jeremy.kloth at gmail.com> a ?crit :
>
> On Sat, Aug 25, 2018 at 1:28 AM Serhiy Storchaka
> <webhook-mailer at python.org> wrote:
> >
> > https://github.com/python/cpython/commit/c406d5cd74002964a64c3eb7d9e2445a7fd3a03f
> > commit: c406d5cd74002964a64c3eb7d9e2445a7fd3a03f
> > branch: master
> > author: Serhiy Storchaka <storchaka at gmail.com>
> > committer: GitHub <noreply at github.com>
> > date: 2018-08-25T10:27:55+03:00
> > summary:
> >
> > bpo-34171: Prevent creating Lib/trace.cover when run the trace module. (GH-8841)
> >
> > files:
> > A Misc/NEWS.d/next/Library/2018-08-21-00-29-01.bpo-34171.6LkWav.rst
> > M Lib/test/test_trace.py
> > M Lib/trace.py
>
> This change seems to have caused most buildbots to go red.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com

From jeremy.kloth at gmail.com  Mon Aug 27 13:30:32 2018
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Mon, 27 Aug 2018 11:30:32 -0600
Subject: [Python-Dev] [Python-checkins] bpo-34171: Prevent creating
 Lib/trace.cover when run the trace module. (GH-8841)
In-Reply-To: <CA+3bQGGK1d8U52nYt7ZRyBAhVVATGvNDhqQRKV_AbPD6ZA5J9Q@mail.gmail.com>
References: <41y8rt4ZQVzFrcd@mail.python.org>
 <CAGvrs3LsCmerfJuedWShYjhwSfF_bmW-QC2cxfjQaA+2Di1NZA@mail.gmail.com>
 <CA+3bQGGK1d8U52nYt7ZRyBAhVVATGvNDhqQRKV_AbPD6ZA5J9Q@mail.gmail.com>
Message-ID: <CAGvrs3+EiOpY-8KNVZqXrkLAVJ52rJsGHg0bxZEOghDuwP22=A@mail.gmail.com>

On Mon, Aug 27, 2018 at 7:26 AM Victor Stinner <vstinner at redhat.com> wrote:
>
> Jemery: would you mind to revert this commit since nobody fixed the
> buildbot since 2 days?
> https://pythondev.readthedocs.io/ci.html#revert-on-fail

I think you meant Serhiy :)  Anyway, a commit has finally landed that
addresses the buildbot failures, at least for master.

-- 
Jeremy Kloth

From pierre.quentel at gmail.com  Mon Aug 27 15:40:42 2018
From: pierre.quentel at gmail.com (Pierre Quentel)
Date: Mon, 27 Aug 2018 21:40:42 +0200
Subject: [Python-Dev] Review of Pull Request #2078
Message-ID: <CA+gUBLSAv7qbvRAbHr_Oq1NHAv_XfsTxTx8P2YTJS+4CYbYhqQ@mail.gmail.com>

 Hi,

A few months ago I have submitted the Pull Request #2078 named "bpo-30576 :
Add HTTP compression support to http.server".

After a couple of iterations, discussions on the PR page (
https://github.com/python/cpython/pull/2078) and on the bug tracker (
https://bugs.python.org/issue30576), it seems to me that there are no
objection to adding this feature (the Github page even has recent messages
favorable to its inclusion) and that the implementation is not a complete
mess ;-)

It has been in the "waiting core review" state for many months now. As
indicated on the contributing page (
https://github.com/python/cpython/blob/master/.github/CONTRIBUTING.rst) I
send this message to python-dev to see if one of the core developers could
take a look at the Pull Request.

Thanks for your time,
Pierre
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180827/ed828bbd/attachment.html>

From turnbull.stephen.fw at u.tsukuba.ac.jp  Tue Aug 28 03:57:17 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Tue, 28 Aug 2018 16:57:17 +0900
Subject: [Python-Dev] make patchcheck and git path
In-Reply-To: <F8F7381B-EAC5-47EA-8153-7F45512A8ECB@felt.demon.nl>
References: <5f9b58cd-224a-6851-0bd0-54a2731aa9e4@felt.demon.nl>
 <CAGbohnZXazy1PxA39mxzzxrRhp9n_8ouUpDz7CZr_f2VXzCKMw@mail.gmail.com>
 <F8F7381B-EAC5-47EA-8153-7F45512A8ECB@felt.demon.nl>
Message-ID: <23429.93.348012.547338@turnbull.sk.tsukuba.ac.jp>

Michael Felt (aixtools) writes:

 > When building out of tree there is no .git reference. If I
 > understand the process it uses git to see what files have changed,
 > and does further processing on those.

Just guessing based on generic git knowledge here:

If you build in a sibling directory of the .git directory, git should
"see" the GITDIR, and it should work.  Where is your build directory
relative to the GITDIR?

I suspect you could also set GITDIR=/path/to/python/source/.git in
make's process environment, and do "make patchcheck" outside of the
Python source tree successfully.

Regards,


From vstinner at redhat.com  Tue Aug 28 08:59:09 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 28 Aug 2018 14:59:09 +0200
Subject: [Python-Dev] UTF-8 Mode now also enabled by the POSIX locale
Message-ID: <CA+3bQGEUz-PATkcBkPUoxKKLe64YjQFukn8yB+zkHqOWm=QBUQ@mail.gmail.com>

Hi,

While working on test_utf8_mode on AIX (bpo-34347) and HP-UX
(bpo-34403), I noticed that FreeBSD doesn't work properly with the
POSIX locale (bpo-34527). I also noticed that my implementation of my
PEP 540 "UTF-8 Mode" doesn't respect the PEP: the UTF-8 Mode should be
enabled by the POSIX locale, not only by the C locale.

I just modified Python 3.7 and master (future 3.8) to enable UTF-8
Mode if the LC_CTYPE locale is "POSIX":
https://bugs.python.org/issue34527

I also fixed FreeBSD to support the "POSIX" locale as well (3.6, 3.7
and master branches).

Note: The C locale coercion (PEP 538) is only enabled if the LC_CTYPE
locale is "C".

To finish my list of recent Unicode issues, I'm also working on
bpo-34523: select the Python filesystem encoding before
Py_Initialize(). This change will allow embedders to force an encoding
instead of letting Python choose one for them.

Victor

From eelizondo at fb.com  Tue Aug 28 16:09:04 2018
From: eelizondo at fb.com (Eddie Elizondo)
Date: Tue, 28 Aug 2018 20:09:04 +0000
Subject: [Python-Dev] A Subtle Bug in Class Initializations
In-Reply-To: <CAOTD34b_w1f46FR4hJf7P1kU7TVWOHy-9qV0sLu3VkELCK09rw@mail.gmail.com>
References: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
 <CAOTD34ZperVL_ZVRkoU55RRrEmCxp0AKSPEYa9jbqArzdaF6bA@mail.gmail.com>
 <1422780e-f11c-5a21-88b8-db17c1e78aec@python.org>
 <CAOTD34Zwu=LU4b9L-We4X2hkds=UQ5Xs53vjK6_7m9TQFcWD=w@mail.gmail.com>
 <61704013-0bfd-1af9-b87b-42ab6a30d11e@python.org>
 <CAOTD34b_w1f46FR4hJf7P1kU7TVWOHy-9qV0sLu3VkELCK09rw@mail.gmail.com>
Message-ID: <F74B2A02-BAFB-4D8D-B33F-09B7B4E2C804@fb.com>

Hi everyone, 

Sorry for the delay - I finally sent out a patch: https://bugs.python.org/issue34522.

Also, I'm +1 for modifying all extension modules to use PyType_FromSpec! I might lend a hand to start moving them :)

- Eddie

?On 8/13/18, 6:02 AM, "Erik Bray" <erik.m.bray at gmail.com> wrote:

    On Fri, Aug 10, 2018 at 6:49 PM Steve Dower <steve.dower at python.org> wrote:
    >
    > On 10Aug2018 0354, Erik Bray wrote:
    > > Thanks!  I'm not sure what you mean by "on other OS's" though.  Do you
    > > mean other OS's that happen to use Windows-style PE/COFF binaries?
    > > Because other than Windows I'm not sure what we care about there.
    > >
    > > For ELF binaries, at least on Linux (and probably elsewhere) it the
    > > runtime loader can perform more sophisticated relocations when loading
    > > a binary into memory, including relocating pointers in the binary's
    > > .data section.  This allows it to initialize data in one executable
    > > "A" with pointers to data in another library "B" *before* "A" is
    > > considered fully loaded and executable.
    > >
    > > So this problem never arises, at least on Linux.
    >
    > That's exactly what I meant. I simply didn't know how/whether other
    > loaders handled this case :) I recognise it's nothing to do with the
    > binary format and everything to do with whether the loader knows what to
    > do or not.
    
    Ah, that's not exactly what *I* meant, but you are also right: In
    principle it shouldn't have anything to do with the binary formation.
    You could stuff a more sophisticated dynamic relocation section into a
    PE/COFF binary too but Windows wouldn't know what to do with it.
    
    So you're right that this kind of problem could affect other OSes, I
    just have no idea if it does.
    
    > >>> So I'm +1 for requiring passing NULL to PyVarObject_HEAD_INIT,
    > >>> requiring PyType_Ready with an explicit base type argument, and maybe
    > >>> (eventually) making PyVarObject_HEAD_INIT argumentless.
    > >>
    > >> Since PyVarObject_HEAD_INIT currently requires PyType_Ready() in
    > >> extension modules already, then don't we just need to fix the built-in
    > >> types?
    > >>
    > >> As far as the "eventually" case, I'd hope that eventually extension
    > >> modules are all using PyType_FromSpec() :)
    > >
    > > +1 :)
    >
    > Is that just a +1 for PyType_FromSpec(), or are you agreeing that we
    > only need to fix the built-in types?
    
    Both! I think we should fix the built-in types, but it would be nice
    if more extension modules used PyType_FromSpec, if nothing else
    because I find it much more readable, and (I'm guessing) it's better
    from the perspective of Victor's C-API work.  But I admit I'm not
    fully versed in the downsides (if there are any).
    


From ronaldoussoren at mac.com  Wed Aug 29 15:34:31 2018
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 29 Aug 2018 21:34:31 +0200
Subject: [Python-Dev] A Subtle Bug in Class Initializations
In-Reply-To: <F74B2A02-BAFB-4D8D-B33F-09B7B4E2C804@fb.com>
References: <D4EE57F4-EBAC-42FF-BB11-28A2945A6144@fb.com>
 <CAOTD34ZperVL_ZVRkoU55RRrEmCxp0AKSPEYa9jbqArzdaF6bA@mail.gmail.com>
 <1422780e-f11c-5a21-88b8-db17c1e78aec@python.org>
 <CAOTD34Zwu=LU4b9L-We4X2hkds=UQ5Xs53vjK6_7m9TQFcWD=w@mail.gmail.com>
 <61704013-0bfd-1af9-b87b-42ab6a30d11e@python.org>
 <CAOTD34b_w1f46FR4hJf7P1kU7TVWOHy-9qV0sLu3VkELCK09rw@mail.gmail.com>
 <F74B2A02-BAFB-4D8D-B33F-09B7B4E2C804@fb.com>
Message-ID: <1C2E18E9-5FB4-4B1F-99DB-DA9AD8609153@mac.com>



> On 28 Aug 2018, at 22:09, Eddie Elizondo <eelizondo at fb.com> wrote:
> 
> Hi everyone, 
> 
> Sorry for the delay - I finally sent out a patch: https://bugs.python.org/issue34522.

As I mentioned on the tracker this patch is incorrect because it redefines PyVarObject_HEAD_INIT in a way that changes the semantics of existing code. This will break C extensions (I?m the author of one, and there may be others). 

> 
> Also, I'm +1 for modifying all extension modules to use PyType_FromSpec! I might lend a hand to start moving them :)

Somewhat off-topic, but I personally don?t like PyType_FromSpec and prefer the older PyTypeObject construction but using C99 initialisers. That gives short and readable code where the compiler can (and does) warn when initialising slots to values that don?t match the declared type of those slots.  The only advantage of using PyType_FromSpec is that this can result in C extensions that are binary compatible between python versions, which IMHO is less useful than it used to be in the past thanks to CI/CD systems. 

PyType_FromSpec also cannot deal with some esoteric use cases, such as defining types with a custom metatype in C code. 

Ronald

From status at bugs.python.org  Fri Aug 31 12:10:03 2018
From: status at bugs.python.org (Python tracker)
Date: Fri, 31 Aug 2018 18:10:03 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20180831161003.7114357536@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2018-08-24 - 2018-08-31)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6828 (+21)
  closed 39479 (+48)
  total  46307 (+69)

Open issues with patches: 2719 


Issues opened (49)
==================

#34171: Lib/trace.cover not removed by the clean target
https://bugs.python.org/issue34171  reopened by pablogsal

#34488: improve performance of BytesIO.writelines() by avoiding creati
https://bugs.python.org/issue34488  opened by sir-sigurd

#34489: subprocess: execution of batch-files (.cmd/.bat) is vulnerable
https://bugs.python.org/issue34489  opened by sebres

#34490: transport.get_extra_info('sockname') of test_asyncio fails on 
https://bugs.python.org/issue34490  opened by Michael.Felt

#34496: Argparse library: parse --set type
https://bugs.python.org/issue34496  opened by fralau

#34498: Python 3.7 breaks on singledispatch_function.register(pseudo_t
https://bugs.python.org/issue34498  opened by Dutcho

#34499: Extend registering of single-dispatch functions to parametrize
https://bugs.python.org/issue34499  opened by Dutcho

#34500: Fix ResourceWarning in difflib.py
https://bugs.python.org/issue34500  opened by Tiger-222

#34505: urllib2 fails for proxy credentials that contains a '/' charac
https://bugs.python.org/issue34505  opened by Balibax5445

#34506: Traceback logged when SSL handshake fails
https://bugs.python.org/issue34506  opened by hniksic

#34507: Add path expansion interpolation in pyvenv.cfg home key
https://bugs.python.org/issue34507  opened by chnlior

#34508: return of non-parenthesized star-unpacking expression a Syntax
https://bugs.python.org/issue34508  opened by mark.dickinson

#34509: Starting to use gcc-8 on CI
https://bugs.python.org/issue34509  opened by junaruga

#34510: add HTTPConnection.settimeout()
https://bugs.python.org/issue34510  opened by collinanderson

#34512: Document platform-specific strftime() behavior for non-ASCII f
https://bugs.python.org/issue34512  opened by izbyshev

#34513: test_multiprocessing_spawn fails on x86 Windows7 3.7 buildbot
https://bugs.python.org/issue34513  opened by pablogsal

#34514: assertEqual doesn't use maxDiff when comparing dictionaries
https://bugs.python.org/issue34514  opened by Victor Engmark2

#34515: lib2to3: support non-ASCII identifiers
https://bugs.python.org/issue34515  opened by monson

#34516: httplib sets unbefitting "Host" in request header when request
https://bugs.python.org/issue34516  opened by visionwun

#34518: Documentation for coroutine objects
https://bugs.python.org/issue34518  opened by hniksic

#34519: Add additional aliases for HP Roman 8
https://bugs.python.org/issue34519  opened by michael-o

#34520: test_asyncio leaked [2, 2, 2] references, sum=6 in AMD64 Windo
https://bugs.python.org/issue34520  opened by pablogsal

#34521: test_socket.RecvmsgIntoSCMRightsStreamTest fails on AMD64 Free
https://bugs.python.org/issue34521  opened by pablogsal

#34522: PyTypeObject's tp_base initialization bug
https://bugs.python.org/issue34522  opened by eelizondo

#34525: smtplib's authobject return value wrongly documented
https://bugs.python.org/issue34525  opened by srittau

#34526: Path.relative_to() taking multiple arguments could be better d
https://bugs.python.org/issue34526  opened by Antony.Lee

#34529: add the option for json.dumps to return newline delimited json
https://bugs.python.org/issue34529  opened by ronron

#34530: distutils: find_executable() fails if the PATH environment var
https://bugs.python.org/issue34530  opened by vstinner

#34532: Python launcher on Windows exits with error when requesting li
https://bugs.python.org/issue34532  opened by bgerrity

#34533: Apply PEP384 to _csv module
https://bugs.python.org/issue34533  opened by eelizondo

#34535: queue.Queue(timeout=0.001) avg delay Windows:14.5ms, Ubuntu: 0
https://bugs.python.org/issue34535  opened by Gammaguy

#34536: Enum._missing_ doesn't raise TypeError when a non-Enum object 
https://bugs.python.org/issue34536  opened by Paul Pinterits

#34537: test_gdb fails with LC_ALL=C
https://bugs.python.org/issue34537  opened by vstinner

#34538: Remove encouragement to author a base class for all Exception 
https://bugs.python.org/issue34538  opened by Nathaniel Manista

#34539: namedtuple's exec() throws segmentation fault
https://bugs.python.org/issue34539  opened by turicas

#34541: pathlib.Path.iterdir doesn't throw an exception until you star
https://bugs.python.org/issue34541  opened by Paul Pinterits

#34542: [TLS] Update test certs to future proof settings
https://bugs.python.org/issue34542  opened by christian.heimes

#34543: _struct.Struct: calling functions without calling __init__ res
https://bugs.python.org/issue34543  opened by DeKrain

#34544: FreeBSD: Fatal Python error: get_locale_encoding: failed to ge
https://bugs.python.org/issue34544  opened by vstinner

#34546: Zipfile encryption function
https://bugs.python.org/issue34546  opened by ????????????

#34547: Wsgiref server does not handle closed connections gracefully
https://bugs.python.org/issue34547  opened by Petter S

#34548: IDLE: Make TextView use the configured theme colors
https://bugs.python.org/issue34548  opened by taleinat

#34549: unittest docs could use another header
https://bugs.python.org/issue34549  opened by napsterinblue

#34550: UnicodeDecodeError when invoke method configure() of Menu inst
https://bugs.python.org/issue34550  opened by sbellct

#34551: Redundant store can be removed from _PyFunction_FastCallDict
https://bugs.python.org/issue34551  opened by Eric Lippert

#34552: Clarify built-in types comparisons
https://bugs.python.org/issue34552  opened by Windson Yang

#34553: Python Crashes when trying to access any date related fields i
https://bugs.python.org/issue34553  opened by vijay

#34555: AF_VSOCK not unset because of wrong nesting
https://bugs.python.org/issue34555  opened by mcduke

#34556: Add --upgrade to venv module
https://bugs.python.org/issue34556  opened by cooperlees



Most recent 15 issues with no replies (15)
==========================================

#34555: AF_VSOCK not unset because of wrong nesting
https://bugs.python.org/issue34555

#34552: Clarify built-in types comparisons
https://bugs.python.org/issue34552

#34550: UnicodeDecodeError when invoke method configure() of Menu inst
https://bugs.python.org/issue34550

#34549: unittest docs could use another header
https://bugs.python.org/issue34549

#34548: IDLE: Make TextView use the configured theme colors
https://bugs.python.org/issue34548

#34547: Wsgiref server does not handle closed connections gracefully
https://bugs.python.org/issue34547

#34546: Zipfile encryption function
https://bugs.python.org/issue34546

#34537: test_gdb fails with LC_ALL=C
https://bugs.python.org/issue34537

#34536: Enum._missing_ doesn't raise TypeError when a non-Enum object 
https://bugs.python.org/issue34536

#34533: Apply PEP384 to _csv module
https://bugs.python.org/issue34533

#34532: Python launcher on Windows exits with error when requesting li
https://bugs.python.org/issue34532

#34526: Path.relative_to() taking multiple arguments could be better d
https://bugs.python.org/issue34526

#34525: smtplib's authobject return value wrongly documented
https://bugs.python.org/issue34525

#34521: test_socket.RecvmsgIntoSCMRightsStreamTest fails on AMD64 Free
https://bugs.python.org/issue34521

#34515: lib2to3: support non-ASCII identifiers
https://bugs.python.org/issue34515



Most recent 15 issues waiting for review (15)
=============================================

#34555: AF_VSOCK not unset because of wrong nesting
https://bugs.python.org/issue34555

#34551: Redundant store can be removed from _PyFunction_FastCallDict
https://bugs.python.org/issue34551

#34548: IDLE: Make TextView use the configured theme colors
https://bugs.python.org/issue34548

#34542: [TLS] Update test certs to future proof settings
https://bugs.python.org/issue34542

#34541: pathlib.Path.iterdir doesn't throw an exception until you star
https://bugs.python.org/issue34541

#34533: Apply PEP384 to _csv module
https://bugs.python.org/issue34533

#34530: distutils: find_executable() fails if the PATH environment var
https://bugs.python.org/issue34530

#34525: smtplib's authobject return value wrongly documented
https://bugs.python.org/issue34525

#34522: PyTypeObject's tp_base initialization bug
https://bugs.python.org/issue34522

#34519: Add additional aliases for HP Roman 8
https://bugs.python.org/issue34519

#34515: lib2to3: support non-ASCII identifiers
https://bugs.python.org/issue34515

#34512: Document platform-specific strftime() behavior for non-ASCII f
https://bugs.python.org/issue34512

#34509: Starting to use gcc-8 on CI
https://bugs.python.org/issue34509

#34508: return of non-parenthesized star-unpacking expression a Syntax
https://bugs.python.org/issue34508

#34500: Fix ResourceWarning in difflib.py
https://bugs.python.org/issue34500



Top 10 most discussed issues (10)
=================================

#34171: Lib/trace.cover not removed by the clean target
https://bugs.python.org/issue34171  12 msgs

#34535: queue.Queue(timeout=0.001) avg delay Windows:14.5ms, Ubuntu: 0
https://bugs.python.org/issue34535   9 msgs

#34538: Remove encouragement to author a base class for all Exception 
https://bugs.python.org/issue34538   9 msgs

#34529: add the option for json.dumps to return newline delimited json
https://bugs.python.org/issue34529   8 msgs

#34508: return of non-parenthesized star-unpacking expression a Syntax
https://bugs.python.org/issue34508   7 msgs

#34522: PyTypeObject's tp_base initialization bug
https://bugs.python.org/issue34522   7 msgs

#34453: ipaddress module accepts some strange IPv6 addresses that it s
https://bugs.python.org/issue34453   6 msgs

#34481: Different behavior of C and Python impls of datetime.strftime 
https://bugs.python.org/issue34481   6 msgs

#34007: test_gdb fails in s390x SLES buildbots
https://bugs.python.org/issue34007   5 msgs

#34200: importlib: python -m test test_pkg -m test_7 fails randomly
https://bugs.python.org/issue34200   5 msgs



Issues closed (47)
==================

#6700: inspect.getsource() returns incorrect source lines at the modu
https://bugs.python.org/issue6700  closed by taleinat

#11190: test_locale error on AIX
https://bugs.python.org/issue11190  closed by vstinner

#11193: test_subprocess test_undecodable_env error on AIX
https://bugs.python.org/issue11193  closed by gregory.p.smith

#13312: test_time fails: strftime('%Y', y) for negative year
https://bugs.python.org/issue13312  closed by gregory.p.smith

#17102: tarfile extract can write files outside the destination path
https://bugs.python.org/issue17102  closed by gregory.p.smith

#21145: Add the @cached_property decorator
https://bugs.python.org/issue21145  closed by ncoghlan

#24482: itertools.tee causes segfault in a multithreading environment,
https://bugs.python.org/issue24482  closed by xiang.zhang

#26694: Disasembler fall with Key Error while disassemble obfuscated c
https://bugs.python.org/issue26694  closed by serhiy.storchaka

#32836: Symbol table for comprehensions (list, dict, set) still includ
https://bugs.python.org/issue32836  closed by serhiy.storchaka

#32968: Fraction modulo infinity should behave consistently with other
https://bugs.python.org/issue32968  closed by mark.dickinson

#33550: Sigpipe handling issue should be documented
https://bugs.python.org/issue33550  closed by Mariatta

#34062: Python launcher on Windows does not work with --list or --list
https://bugs.python.org/issue34062  closed by steve.dower

#34207: test_cmd_line test_utf8_mode test_warnings fail in all FreeBSD
https://bugs.python.org/issue34207  closed by vstinner

#34306: minidom: wrong processing of xmlns attributes
https://bugs.python.org/issue34306  closed by porton

#34319: Clarify pathlib.Path("filepath").read_text()
https://bugs.python.org/issue34319  closed by CuriousLearner

#34347: AIX: test_utf8_mode.test_cmd_line fails
https://bugs.python.org/issue34347  closed by vstinner

#34355: SIGSEGV (Address boundary error)
https://bugs.python.org/issue34355  closed by inada.naoki

#34395: memory leaks in error handling in csv and pickle modules
https://bugs.python.org/issue34395  closed by serhiy.storchaka

#34403: test_utf8_mode.test_cmd_line() fails on HP-UX due to false ass
https://bugs.python.org/issue34403  closed by vstinner

#34426: "__lltrace__"  seems to be wrong , "__ltrace__" is correct in 
https://bugs.python.org/issue34426  closed by Mariatta

#34427: calling MutableSequence.extend on self produces infinite loop
https://bugs.python.org/issue34427  closed by rhettinger

#34428: tokenize
https://bugs.python.org/issue34428  closed by terry.reedy

#34434: Removal of kwargs for built-in types not covered with "changed
https://bugs.python.org/issue34434  closed by xiang.zhang

#34448: incomplete/missing message from configure.ac for usable wchar_
https://bugs.python.org/issue34448  closed by berker.peksag

#34483: eval() raises NameError: name '...' is not defined
https://bugs.python.org/issue34483  closed by eric.smith

#34485: _PyCoreConfig: add stdio_encoding and stdio_errors
https://bugs.python.org/issue34485  closed by vstinner

#34491: _bsddb.c: Missing Py_DECREF() in DB_join()
https://bugs.python.org/issue34491  closed by xiang.zhang

#34492: Python/coreconfig.c: Fix _Py_wstrlist_copy()
https://bugs.python.org/issue34492  closed by vstinner

#34493: Objects/genobject.c: Missing NULL check in compute_cr_origin()
https://bugs.python.org/issue34493  closed by benjamin.peterson

#34494: simple "sequence" class ignoring __len__
https://bugs.python.org/issue34494  closed by rhettinger

#34495: excess_args asserts if args == nullptr
https://bugs.python.org/issue34495  closed by benjamin.peterson

#34497: Remove needless set operator restriction
https://bugs.python.org/issue34497  closed by rhettinger

#34501: PyType_FromSpecWithBases: spec->name is dereferenced before ch
https://bugs.python.org/issue34501  closed by benjamin.peterson

#34502: Docs of sys.exit() mention utf8_mode for an unclear reason
https://bugs.python.org/issue34502  closed by benjamin.peterson

#34503: Reference leak in PyErr_SetObject()
https://bugs.python.org/issue34503  closed by xiang.zhang

#34504: PySequence_Check: argument is dereferenced and then checked fo
https://bugs.python.org/issue34504  closed by benjamin.peterson

#34511: I suggest to add documentation about "method" parameter of url
https://bugs.python.org/issue34511  closed by Mariatta

#34517: Error referencing local variables in dict comprehensions insid
https://bugs.python.org/issue34517  closed by serhiy.storchaka

#34523: Choose the filesystem encoding before Python initialization (a
https://bugs.python.org/issue34523  closed by vstinner

#34524: Format conversion documentation example don't match comment
https://bugs.python.org/issue34524  closed by Richard Evans

#34527: On FreeBSD, Python 3 doesn't support support the POSIX locale
https://bugs.python.org/issue34527  closed by vstinner

#34528: Windows installer: Failed to run untrusted mode.
https://bugs.python.org/issue34528  closed by steve.dower

#34531: doc Move comment about list sorting behavour outside impl-deta
https://bugs.python.org/issue34531  closed by rhettinger

#34534: importlib.resources does not work with packages that have no _
https://bugs.python.org/issue34534  closed by barry

#34540: shutil._call_external_zip should use subprocess
https://bugs.python.org/issue34540  closed by benjamin.peterson

#34545: error in the repl due to indentation
https://bugs.python.org/issue34545  closed by eryksun

#34554: Add match built-in function
https://bugs.python.org/issue34554  closed by eric.snow