From vstinner at redhat.com  Mon Oct  1 07:19:39 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 1 Oct 2018 13:19:39 +0200
Subject: [Python-Dev] Communication channels
Message-ID: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>

Hi,

Last months, new communication channels appear. This is just a
reminder that they exist:

* Zulip: https://python.zulipchat.com/ (exist since 1 year?)
* Discourse: http://discuss.python.org/ (I'm not sure if it's fully
official yet ;-))
* IRC: #python-dev on FreeNode, only for development *of* Python,
mostly to discuss bugs and pull requests
* Mailing lists: python-ideas, python-dev, etc.

Some core developers are also active on Twitter. Some ideas were first
discussed on Twitter. You may want to follow some of them. Incomplete
list of core devs that I follow:

* Barry Warsaw: https://twitter.com/pumpichank
* Brett Cannon: https://twitter.com/brettsky
* Guido van Rossum: https://twitter.com/gvanrossum
* ?ukasz Langa: https://twitter.com/llanga
* Mariatta: https://twitter.com/Mariatta
* Serhiy Storchaka: https://twitter.com/SerhiyStorchaka
* Yury Selivanov: https://twitter.com/1st1
* ... the full list is very long, and I'm too lazy to complete it :-)
Maybe someone has already a list more complete than mine.

I hope that I didn't miss an important communication channel :-)

Victor

From chris.jerdonek at gmail.com  Mon Oct  1 08:13:57 2018
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 1 Oct 2018 05:13:57 -0700
Subject: [Python-Dev] Communication channels
In-Reply-To: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>
References: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>
Message-ID: <CAOTb1wf=-VO4bRKY_PkiDFyuN-PdfEgLZH+G_sgsjC4dMQ64EA@mail.gmail.com>

Another one is GitHub (and the bug tracker, for that matter). For
example, I believe here is where the discussion took place that led to
the initial draft of PEP 582 re: recognizing a local __packages__
directory:
https://github.com/kushaldas/peps/pull/1

The PEP was posted here:
https://github.com/python/peps/pull/776
https://www.python.org/dev/peps/pep-0582/

To my knowledge it hasn't been discussed on python-dev yet.

Also, if you are trying to be complete, another communication channel
is in-person events and conferences, etc.

--Chris


On Mon, Oct 1, 2018 at 4:21 AM Victor Stinner <vstinner at redhat.com> wrote:
>
> Hi,
>
> Last months, new communication channels appear. This is just a
> reminder that they exist:
>
> * Zulip: https://python.zulipchat.com/ (exist since 1 year?)
> * Discourse: http://discuss.python.org/ (I'm not sure if it's fully
> official yet ;-))
> * IRC: #python-dev on FreeNode, only for development *of* Python,
> mostly to discuss bugs and pull requests
> * Mailing lists: python-ideas, python-dev, etc.
>
> Some core developers are also active on Twitter. Some ideas were first
> discussed on Twitter. You may want to follow some of them. Incomplete
> list of core devs that I follow:
>
> * Barry Warsaw: https://twitter.com/pumpichank
> * Brett Cannon: https://twitter.com/brettsky
> * Guido van Rossum: https://twitter.com/gvanrossum
> * ?ukasz Langa: https://twitter.com/llanga
> * Mariatta: https://twitter.com/Mariatta
> * Serhiy Storchaka: https://twitter.com/SerhiyStorchaka
> * Yury Selivanov: https://twitter.com/1st1
> * ... the full list is very long, and I'm too lazy to complete it :-)
> Maybe someone has already a list more complete than mine.
>
> I hope that I didn't miss an important communication channel :-)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com

From vstinner at redhat.com  Mon Oct  1 08:48:05 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 1 Oct 2018 14:48:05 +0200
Subject: [Python-Dev] Communication channels
In-Reply-To: <CAOTb1wf=-VO4bRKY_PkiDFyuN-PdfEgLZH+G_sgsjC4dMQ64EA@mail.gmail.com>
References: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>
 <CAOTb1wf=-VO4bRKY_PkiDFyuN-PdfEgLZH+G_sgsjC4dMQ64EA@mail.gmail.com>
Message-ID: <CA+3bQGH7NmJAWfedDFtqpJH_qryiYToyQrmDuqt78SjydJx7=g@mail.gmail.com>

Le lun. 1 oct. 2018 ? 14:14, Chris Jerdonek <chris.jerdonek at gmail.com> a ?crit :
> Also, if you are trying to be complete, another communication channel
> is in-person events and conferences, etc.

Right, there are two main events for CPython core developers which are
only in the US:

* Language Summit during Pycon US (one day)
* Sprint in September (one week)

Victor

From brett at python.org  Mon Oct  1 13:51:07 2018
From: brett at python.org (Brett Cannon)
Date: Mon, 1 Oct 2018 10:51:07 -0700
Subject: [Python-Dev] Communication channels
In-Reply-To: <CAOTb1wf=-VO4bRKY_PkiDFyuN-PdfEgLZH+G_sgsjC4dMQ64EA@mail.gmail.com>
References: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>
 <CAOTb1wf=-VO4bRKY_PkiDFyuN-PdfEgLZH+G_sgsjC4dMQ64EA@mail.gmail.com>
Message-ID: <CAP1=2W50pKEcPf1GP1gkZR=rEvqMZTN74iCVigxCtd8zvO1rHA@mail.gmail.com>

On Mon, 1 Oct 2018 at 05:15, Chris Jerdonek <chris.jerdonek at gmail.com>
wrote:

> Another one is GitHub (and the bug tracker, for that matter). For
> example, I believe here is where the discussion took place that led to
> the initial draft of PEP 582 re: recognizing a local __packages__
> directory:
> https://github.com/kushaldas/peps/pull/1


It started on GitHub and then continued in person at the dev sprints.


>
>
> The PEP was posted here:
> https://github.com/python/peps/pull/776
> https://www.python.org/dev/peps/pep-0582/
>
> To my knowledge it hasn't been discussed on python-dev yet.
>

Nope, I assume because there is no one to actually approve it so there's no
point it discussing it yet. :)

-Brett


>
> Also, if you are trying to be complete, another communication channel
> is in-person events and conferences, etc.
>
> --Chris
>
>
> On Mon, Oct 1, 2018 at 4:21 AM Victor Stinner <vstinner at redhat.com> wrote:
> >
> > Hi,
> >
> > Last months, new communication channels appear. This is just a
> > reminder that they exist:
> >
> > * Zulip: https://python.zulipchat.com/ (exist since 1 year?)
> > * Discourse: http://discuss.python.org/ (I'm not sure if it's fully
> > official yet ;-))
> > * IRC: #python-dev on FreeNode, only for development *of* Python,
> > mostly to discuss bugs and pull requests
> > * Mailing lists: python-ideas, python-dev, etc.
> >
> > Some core developers are also active on Twitter. Some ideas were first
> > discussed on Twitter. You may want to follow some of them. Incomplete
> > list of core devs that I follow:
> >
> > * Barry Warsaw: https://twitter.com/pumpichank
> > * Brett Cannon: https://twitter.com/brettsky
> > * Guido van Rossum: https://twitter.com/gvanrossum
> > * ?ukasz Langa: https://twitter.com/llanga
> > * Mariatta: https://twitter.com/Mariatta
> > * Serhiy Storchaka: https://twitter.com/SerhiyStorchaka
> > * Yury Selivanov: https://twitter.com/1st1
> > * ... the full list is very long, and I'm too lazy to complete it :-)
> > Maybe someone has already a list more complete than mine.
> >
> > I hope that I didn't miss an important communication channel :-)
> >
> > Victor
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181001/1e5d4adc/attachment.html>

From aixtools at felt.demon.nl  Mon Oct  1 15:12:35 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Mon, 1 Oct 2018 21:12:35 +0200
Subject: [Python-Dev] LDLAST variable in configure.ac
Message-ID: <a295bfb3-a102-947d-4c14-cdfa3259ff3a@felt.demon.nl>

Hi all,

Before I submit a patch to increase the default MAXDATA setting for AIX
when in 32-bit mode - I want to know if I can put this LDFLAG setting in
LDLAST, or if I should introduce a new AC_SUBST() variable (e.g.,
LDMAXDATA).

I have not looked yet, but I was thinking that MAYBE! LDLAST is intended
as a last resort variable that can be modified in Makefile.

Thanks!

Michael


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181001/e8498d91/attachment.sig>

From tseaver at palladion.com  Mon Oct  1 16:29:10 2018
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 1 Oct 2018 16:29:10 -0400
Subject: [Python-Dev] Communication channels
In-Reply-To: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>
References: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>
Message-ID: <potvuj$kob$1@blaine.gmane.org>

On 10/01/2018 07:19 AM, Victor Stinner wrote:

> Some core developers are also active on Twitter. Some ideas were first
> discussed on Twitter. You may want to follow some of them. Incomplete
> list of core devs that I follow:

I'm pretty strongly -1 on the notion that folks who subscribe python-dev,
BPO, and the github repositories should need to *also* follow an
arbitrarily-growing set of Twitter accounts:  how would one know if a new
one popped into being?  How likely is it that everything a given Python
developer tweets is relevant for the Python development community?


Tres.
-- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com


From ethan at stoneleaf.us  Mon Oct  1 16:36:39 2018
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 1 Oct 2018 13:36:39 -0700
Subject: [Python-Dev] Communication channels
In-Reply-To: <potvuj$kob$1@blaine.gmane.org>
References: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>
 <potvuj$kob$1@blaine.gmane.org>
Message-ID: <a21382f3-b236-ac3b-0e81-94aa4139846b@stoneleaf.us>

On 10/01/2018 01:29 PM, Tres Seaver wrote:

> I'm pretty strongly -1 on the notion that folks who subscribe python-dev,
> BPO, and the github repositories should need to *also* follow an
> arbitrarily-growing set of Twitter accounts:  how would one know if a new
> one popped into being?  How likely is it that everything a given Python
> developer tweets is relevant for the Python development community?

+1

Anything more than an initial idea really should occur on an ML, 
tracker, or repo.

--
~Ethan~

From vstinner at redhat.com  Mon Oct  1 16:37:59 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 1 Oct 2018 22:37:59 +0200
Subject: [Python-Dev] Communication channels
In-Reply-To: <potvuj$kob$1@blaine.gmane.org>
References: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>
 <potvuj$kob$1@blaine.gmane.org>
Message-ID: <CA+3bQGG9FEMNUVVSyNoV7aMbnkAXBWWOcQb+0TJF31C-XvRCKg@mail.gmail.com>

Le lun. 1 oct. 2018 ? 22:32, Tres Seaver <tseaver at palladion.com> a ?crit :
> I'm pretty strongly -1 on the notion that folks who subscribe python-dev,
> BPO, and the github repositories should need to *also* follow an
> arbitrarily-growing set of Twitter accounts:  how would one know if a new
> one popped into being?  How likely is it that everything a given Python
> developer tweets is relevant for the Python development community?

My intent is not to ask you to subscribe to everything. I just wanted
to list all communication channels and let you make your own choices.
In my case, I'm unable to follow Zulip chat. There are too many
communication channels for me :-)

Victor

From aixtools at felt.demon.nl  Mon Oct  1 16:45:19 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Mon, 1 Oct 2018 22:45:19 +0200
Subject: [Python-Dev] Change in Python 3's "round" behavior
In-Reply-To: <20180930121703.GK19437@ando.pearwood.info>
References: <1519658635.56.0.467229070634.issue32956@psf.upfronthosting.co.za>
 <1537960170.25.0.545547206417.issue32956@psf.upfronthosting.co.za>
 <CANwREeUjUDTFmZ=hkMwsG6vxUvwYYEiuSaY4zF8SEL6J74FvoQ@mail.gmail.com>
 <5BAC70BB.2040707@canterbury.ac.nz>
 <20180927135327.GE19437@ando.pearwood.info>
 <234e01d4585e$829acc20$87d06460$@sdamon.com>
 <20180930121703.GK19437@ando.pearwood.info>
Message-ID: <dc0c48a7-3fd5-f6e2-3b9e-a47760fa808b@felt.demon.nl>


On 9/30/2018 2:17 PM, Steven D'Aprano wrote:
>  (It's also called Dutch Rounding.)

Ah - as to why - and from school! (as so-called intuitive! rather desired!).

A test score goes from 5.5 to 6.0 - which becomes passing.

Oh, do I recall my children's frustrations when they had a X.4Y score -
that became X.0. Tears!


From bussonniermatthias at gmail.com  Mon Oct  1 17:47:14 2018
From: bussonniermatthias at gmail.com (Matthias Bussonnier)
Date: Mon, 1 Oct 2018 14:47:14 -0700
Subject: [Python-Dev] Communication channels
In-Reply-To: <CA+3bQGG9FEMNUVVSyNoV7aMbnkAXBWWOcQb+0TJF31C-XvRCKg@mail.gmail.com>
References: <CA+3bQGEj9voO4qGZNsr9p=QfSd4oZ0OEUsv0+3q9j-HKz1B9NQ@mail.gmail.com>
 <potvuj$kob$1@blaine.gmane.org>
 <CA+3bQGG9FEMNUVVSyNoV7aMbnkAXBWWOcQb+0TJF31C-XvRCKg@mail.gmail.com>
Message-ID: <CANJQusVbsKp3qUp9L0Zx_VSXGaqBuR8GGHOVj7HY6M+y34X=RQ@mail.gmail.com>

On Mon, 1 Oct 2018 at 13:38, Victor Stinner <vstinner at redhat.com> wrote:

> Le lun. 1 oct. 2018 ? 22:32, Tres Seaver <tseaver at palladion.com> a ?crit :
> > I'm pretty strongly -1 on the notion that folks who subscribe python-dev,
> > BPO, and the github repositories should need to *also* follow an
> > arbitrarily-growing set of Twitter accounts:  how would one know if a new
> > one popped into being?  How likely is it that everything a given Python
> > developer tweets is relevant for the Python development community?
>
> My intent is not to ask you to subscribe to everything. I just wanted

to list all communication channels and let you make your own choices.


And too modest to add self to list of twitter accounts...
https://twitter.com/VictorStinner
Or did he deemed himself not worthy of beeing followed ?

I was tempted to do a joke based on "takes 0 positional arguments but 1 was
given"...

Much love for the time you take to list all those channels and reach out to
the community.
-- 
M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181001/fd09ef11/attachment.html>

From aixtools at felt.demon.nl  Mon Oct  1 18:41:56 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Tue, 2 Oct 2018 00:41:56 +0200
Subject: [Python-Dev] dear core-devs
Message-ID: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>

Dear core-devs,

I have some bad characteristics.

I can be extremely enthusiastic - and write too much. I have been trying
to not write - anything - worried that my enthusiasm is not matched by
yours, or worse was a reason to ignore my work to get AIX passing all tests.

FYI: since the end of July I have dedicated 16 to 24 hours of my free
time to get this done. All for Python; all in my freetime. My employer
does not care - I do, or did.

I am grateful to Martin Panter - who helped me graciously when I knew
absolutely nothing when I first got started; Victor was kind enough to
answer some emails and help me along but also clear that he has zero
interest in AIX and my questions were taking too much of his time.
Regretfully for me.

Again - Victor - thank you for your time. I appreciated the assistance
and feedback.

(Others have helped from time to time, my apologies for not naming you
specifically.)

I am, to put it lightly, extremely frustrated, at this point.

I am sorry, for myself obviously - but also for Python. Obviously, I am
doing it all wrong - as I see lots of other issues being picked up
immediately.

All volunteers need some level of recognition to keep moving on.

And, while you may not give a damn about anything other than Windows,
macos and/or Linux - there are other platforms that would like a stable
Python.

Sincerely,

Michael


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181002/2912a2d0/attachment.sig>

From benjamin at python.org  Mon Oct  1 19:07:20 2018
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 01 Oct 2018 16:07:20 -0700
Subject: [Python-Dev] LDLAST variable in configure.ac
In-Reply-To: <a295bfb3-a102-947d-4c14-cdfa3259ff3a@felt.demon.nl>
References: <a295bfb3-a102-947d-4c14-cdfa3259ff3a@felt.demon.nl>
Message-ID: <1538435240.4040554.1527177920.188EFF58@webmail.messagingengine.com>



On Mon, Oct 1, 2018, at 12:12, Michael Felt wrote:
> Hi all,
> 
> Before I submit a patch to increase the default MAXDATA setting for AIX
> when in 32-bit mode - I want to know if I can put this LDFLAG setting in
> LDLAST, or if I should introduce a new AC_SUBST() variable (e.g.,
> LDMAXDATA).

I think you should just put it in LDFLAGS.

> 
> I have not looked yet, but I was thinking that MAYBE! LDLAST is intended
> as a last resort variable that can be modified in Makefile.

LDLAST looks vestigial from OSF/1 support and should probably be removed.

From steve at pearwood.info  Mon Oct  1 19:50:42 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 2 Oct 2018 09:50:42 +1000
Subject: [Python-Dev] dear core-devs
In-Reply-To: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
Message-ID: <20181001235042.GU19437@ando.pearwood.info>

Hi Michael, and welcome,

On Tue, Oct 02, 2018 at 12:41:56AM +0200, Michael Felt wrote:

[...]
> FYI: since the end of July I have dedicated 16 to 24 hours of my free
> time to get this done. All for Python; all in my freetime. My employer
> does not care - I do, or did.

[...]
> I am, to put it lightly, extremely frustrated, at this point.

Okay, but what are you frustrated *about*?


> I am sorry, for myself obviously - but also for Python. Obviously, I am
> doing it all wrong - as I see lots of other issues being picked up
> immediately.

Doing what wrong?


> All volunteers need some level of recognition to keep moving on.
> 
> And, while you may not give a damn about anything other than Windows,
> macos and/or Linux - there are other platforms that would like a stable
> Python.

???

Have the core devs decided to drop support for AIX? Have your patches 
been rejected or something? Please explain what the actual problem is.



-- 
Steve

From tseaver at palladion.com  Mon Oct  1 21:51:39 2018
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 1 Oct 2018 21:51:39 -0400
Subject: [Python-Dev] dear core-devs
In-Reply-To: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
Message-ID: <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/01/2018 06:41 PM, Michael Felt wrote:

> And, while you may not give a damn about anything other than Windows, 
> macos and/or Linux - there are other platforms that would like a
> stable Python.

Michael,

I can understand the frustration you feel:  you have been putting effort
into a labor of love geting Python support on AIX (back?) into shape, and
feel that your efforts are unappreciated, or worse, that they will be waste
d.

The key thing to realize about the various core developers (and the
broader Python and open source communities) is that their attention is a
heavily over-committed resource:  it isn't that folks here aren't
benevolent toward your efforts, but rather that each of them (us?) makes
decisions every day juggling which projects / tasks to give the minutes /
hours we have available.  In the common case, the "triage" involves
scrathing an itch:  this bug affects me / my work, that feature would
make my life / my employment simpler, etc.  Even where there are minutes
available, the "is reviewing this feasible for me?" question kicks in.

Because AIX is relatively narrow in the scope of folks it impacts, the
average, overcommitted developer is likely to see a bug report, or even a
pull request, which makes stuff build on AIX and say, "Hmm, I don't know
enough to evalute that one, I'll leave it to folks who do know (and by
implication, who have some skin in the game)."  Even for more
consumer-focused platforms, it has historically been harder to get
attention for bugs / patches which affect only a single platform (Windows
file locking semantics, or the Mac installer, etc.)

One key way to get past that hurdle is to slice the size of each "thing"
down as fine as possible:  e.g., a pull request adding a single "#ifdef
AIX" block to one file.  Anything more than a screenful of diff is likely
to trigger the "let someone else review it" pattern, whereas being able
to scan the patch at a glance lets even a non-itchy reviewer decide,
"well, at least it can't hurt anything, give it a shot."

Once you've gotten a number of those small patches merged, you will find
that you've built a relationship with the folks who have been reviewing
them, and that they are more likely to pass them, and to review larger
ones, at least in part because *you* will have learned more about what is
needed in terms of code style, documentation, test coverage, etc., and
*they* will have learned to trust your judgement.

I'm sorry it isn't easier,


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAluyzyYACgkQFXKVXuSL+CMAHQCfXxFKpKyBXQg3dBSPY8MYOwh1
djsAnitN3SjTt+xwDdnT2NvTs965wEjR
=Bl8Z
-----END PGP SIGNATURE-----


From aixtools at felt.demon.nl  Tue Oct  2 06:29:33 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Tue, 2 Oct 2018 12:29:33 +0200
Subject: [Python-Dev] LDLAST variable in configure.ac
In-Reply-To: <1538435240.4040554.1527177920.188EFF58@webmail.messagingengine.com>
References: <a295bfb3-a102-947d-4c14-cdfa3259ff3a@felt.demon.nl>
 <1538435240.4040554.1527177920.188EFF58@webmail.messagingengine.com>
Message-ID: <B23E11B7-37BB-4B24-98B6-BFB7ECABD38A@felt.demon.nl>

A non-text attachment was scrubbed...
Name: mime-attachment
Type: application/pgp-encrypted
Size: 11 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181002/0064f3f8/attachment.pgp>
-------------- next part --------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: encrypted.asc
Type: application/octet-stream
Size: 3238 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181002/0064f3f8/attachment.obj>

From lukasz at langa.pl  Tue Oct  2 08:51:52 2018
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Tue, 2 Oct 2018 14:51:52 +0200
Subject: [Python-Dev] LDLAST variable in configure.ac
In-Reply-To: <B23E11B7-37BB-4B24-98B6-BFB7ECABD38A@felt.demon.nl>
References: <a295bfb3-a102-947d-4c14-cdfa3259ff3a@felt.demon.nl>
 <1538435240.4040554.1527177920.188EFF58@webmail.messagingengine.com>
 <B23E11B7-37BB-4B24-98B6-BFB7ECABD38A@felt.demon.nl>
Message-ID: <DD7CAA94-87F6-4637-93E3-089FF62A6C5C@langa.pl>


> On 2 Oct 2018, at 12:29, Michael Felt <aixtools at felt.demon.nl> wrote:
> 
> <mime-attachment><encrypted.asc>

Michael, this message looks encrypted on my end. For people without your public key, it's impossible to read. This was probably unintentional on your end. In either case I'd avoid encrypting messages that go to public mailing lists.

- ?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181002/e548cad0/attachment.sig>

From erik.m.bray at gmail.com  Tue Oct  2 10:45:27 2018
From: erik.m.bray at gmail.com (Erik Bray)
Date: Tue, 2 Oct 2018 16:45:27 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
Message-ID: <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>

On Tue, Oct 2, 2018 at 3:53 AM Tres Seaver <tseaver at palladion.com> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 10/01/2018 06:41 PM, Michael Felt wrote:
>
> > And, while you may not give a damn about anything other than Windows,
> > macos and/or Linux - there are other platforms that would like a
> > stable Python.
>
> Michael,
>
> I can understand the frustration you feel:  you have been putting effort
> into a labor of love geting Python support on AIX (back?) into shape, and
> feel that your efforts are unappreciated, or worse, that they will be waste
> d.
>
> The key thing to realize about the various core developers (and the
> broader Python and open source communities) is that their attention is a
> heavily over-committed resource:  it isn't that folks here aren't
> benevolent toward your efforts, but rather that each of them (us?) makes
> decisions every day juggling which projects / tasks to give the minutes /
> hours we have available.  In the common case, the "triage" involves
> scrathing an itch:  this bug affects me / my work, that feature would
> make my life / my employment simpler, etc.  Even where there are minutes
> available, the "is reviewing this feasible for me?" question kicks in.
>
> Because AIX is relatively narrow in the scope of folks it impacts, the
> average, overcommitted developer is likely to see a bug report, or even a
> pull request, which makes stuff build on AIX and say, "Hmm, I don't know
> enough to evalute that one, I'll leave it to folks who do know (and by
> implication, who have some skin in the game)."  Even for more
> consumer-focused platforms, it has historically been harder to get
> attention for bugs / patches which affect only a single platform (Windows
> file locking semantics, or the Mac installer, etc.)
>
> One key way to get past that hurdle is to slice the size of each "thing"
> down as fine as possible:  e.g., a pull request adding a single "#ifdef
> AIX" block to one file.  Anything more than a screenful of diff is likely
> to trigger the "let someone else review it" pattern, whereas being able
> to scan the patch at a glance lets even a non-itchy reviewer decide,
> "well, at least it can't hurt anything, give it a shot."
>
> Once you've gotten a number of those small patches merged, you will find
> that you've built a relationship with the folks who have been reviewing
> them, and that they are more likely to pass them, and to review larger
> ones, at least in part because *you* will have learned more about what is
> needed in terms of code style, documentation, test coverage, etc., and
> *they* will have learned to trust your judgement.
>
> I'm sorry it isn't easier,

I have thought of writing an almost verbatim post w.r.t. my efforts to
get Cygwin re-supported (which was never officially un-supported
either).  Victor asked me to set up a buildbot for Cygwin as a
prerequisite to much else, which I have done [1].  But it has been
turning out broken build after broken build and is all but useless
since, even at the time of setting it up, I pointed out that there are
two major blocker issues [2] [3] that prevent an even
partially-working build.  Naoki Inada provided some review of the
first one a while ago, and while we had some (I think valid)
disagreement on how to proceed, I updated the issue description with a
checklist of issues he raised that need some clear direction on how
they should be resolved (since at least on some of them we disagreed).
I'd be happy to do pretty much whatever so long as I knew it was
meeting a core dev's requirements while also meeting my own
requirements.

Obviously I'm sympathetic to the limited time and attention of core
devs--I am a maintainer on several projects myself and I know how it
goes, and I have tried not to make too much of a fuss about it.  But
there's a chicken-egg problem in the specific area of platform
support, which feels a little different from just "I need my pet bug
fixed", for someone who is not already a core developer: In order to
make any progress on the issue we need at least one core dev who is
interested in the same platform.  But if we're the only ones willing
to do the work who know or care anything about that platform, how are
we supposed to progress in the first place?

I, like Michael Felt, have a number of fixes waiting in the wings but
can't really progress until a little bit of bare minimum ground work
is at least done to get us up and running.

Michael, if there are any PRs you want to point me to that I might be
able to help review please do.  I don't know anything about AIX either
and am not a core dev so I can't have a final say.  But I've been
hacking on CPython for a long time anyways, and might be able to help
at least with some initial review.


[1] https://buildbot.python.org/all/#/builders/164
[2] https://github.com/python/cpython/pull/4348
[3] https://github.com/python/cpython/pull/8712

From van.lindberg at gmail.com  Tue Oct  2 10:47:24 2018
From: van.lindberg at gmail.com (VanL)
Date: Tue, 2 Oct 2018 09:47:24 -0500
Subject: [Python-Dev] Maintaining civility - the core of the Python community
Message-ID: <CAFQvZEP9198jVULau3V3S+FnJZFdOXTdbrLCdS68dpAOOXo7WQ@mail.gmail.com>

Hello everyone,

You all have probably noted that there have been some contentious threads
recently, ultimately ending in a few people being given a time-out from
posting on these lists.

I don't normally get into things on this list, but it has been generally
discouraging to see a bunch of generally nice and reasonable people get
sidetracked into unproductive, sarcastic, and unhelpful remarks. There are
some efforts underway to formalize what is in and out of bounds - but I
would suggest that we are losing our way when we need to get to explicit
lists of things not to do.

Unfortunately, we are getting there.

I would like to reemphasize that we are bound together by the things that
we share. We love working on Python. We love being in the community and
seeing it grow. We can make our community stronger and more pleasant by
choosing to emphasize the things that we have in common and by ignoring or
avoiding topics that are more likely to generate unproductive discussion.

We can and should also try to remember that not everyone is coming from the
same place, and so we should actively assume the best of others and
interpret what they say in the most charitable way possible. Think of it as
Postel's law [1] as applied to people.

I'd also suggest that generally, 1) use of profanity, 2) use of sexual
terms and imagery, and 3) use of specifically denigrating terms to refer to
a person [2][3][4] are completely out of bounds for the professional
environment we want to maintain. It is ok to attack arguments, and ideas,
but ad hominem arguments - those attack a person, rather than the person's
argument - are also inappropriate. Use of sarcasm should be strongly
moderated, as it is not correctly interpreted by many people.

No reply is needed to this email. Instead, I'd prefer to see a continuation
of solid technical discussion, rather than meta-discussion.

Thanks,
Van


[1] https://en.wikipedia.org/wiki/Robustness_principle
[2] https://en.wikipedia.org/wiki/List_of_ethnic_slurs
[3] https://en.wikipedia.org/wiki/List_of_religious_slurs
[4]
https://en.wikipedia.org/wiki/List_of_disability-related_terms_with_negative_connotations
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181002/0769fd7c/attachment.html>

From will at worrbase.com  Tue Oct  2 02:42:59 2018
From: will at worrbase.com (William Orr)
Date: Mon, 01 Oct 2018 23:42:59 -0700
Subject: [Python-Dev] [PATCH] BSD extended attributes
Message-ID: <87o9cc6e8c.fsf@locke.worr.haus>

Hey,

Can I get a review for GH-1690[1]? It fixes bpo-12978 and
has been sitting for a handful of years now. This adds
support for os.*xattr on DragonflyBSD, FreeBSD and NetBSD.

Thanks!

[1] https://github.com/python/cpython/pull/1690

From hodgestar+pythondev at gmail.com  Tue Oct  2 12:41:15 2018
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Tue, 2 Oct 2018 18:41:15 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
Message-ID: <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>

Are there any core devs that Michael or Erik could collaborate with?
Rather than rely on adhoc patch review from random core developers.

Michael and Eric: Question -- are you interested in becoming core
developers at least for the purposes of maintaining these platforms in
future?

From eelizondo at fb.com  Tue Oct  2 13:09:09 2018
From: eelizondo at fb.com (Eddie Elizondo)
Date: Tue, 2 Oct 2018 17:09:09 +0000
Subject: [Python-Dev] Heap-allocated StructSequences
In-Reply-To: <20180914063459.sqid2n6bera5or6l@python.ca>
References: <EA6CC3C2-A371-4F24-AF5C-4401B57ED487@fb.com>
 <20180914063459.sqid2n6bera5or6l@python.ca>
Message-ID: <CAECB1F3-BC0F-4565-8C29-E75B60BA1FD1@fb.com>

> We have to assess how 3rd party extension modules would be affected
> by this change.

This change is fully compatible with 3rd party extensions.
The current change to InitType2 is only refactoring, there is no logic change there so that API remains unchanged.
Also, there should not be any instances of PyStructSequence_NewType in use. Any usage of this API causes a crash. A quick Google and Github search show that this is true. Thus, modifying this function should have no conflicts.

A more interesting question would be: "Is the migration of PyStructSequence_InitType2 to PyStructSequence_NewType backwards-compatible?" The answer is yes!

Using gevent as an example (https://github.com/gevent/gevent). This library has a dependency on StatResultType from cpython/Modules/posixmodule.c. This type gets initialized with PyStructSequence_InitType2. After modifying posixmodule.c to use NewType instead of InitType2 gevent still builds and passes all tests. Example: https://github.com/python/cpython/pull/9665

Thus, this change is backwards-compatible by itself and even after migrating to the NewType C-API.

> Converting things to use PyType_FromSpec 
> falls in there. As long as the old API still works, these changes should 
> go in (but they might need a PEP).

I agree that this change is standalone and should go in by itself. Yet, I'm open to whatever people thing might be the right approach to get this in. i.e: Have more people look at it, writing a PEP, etc.

- Eddie


From aixtools at felt.demon.nl  Tue Oct  2 14:41:57 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Tue, 2 Oct 2018 20:41:57 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
Message-ID: <400a6409-74bb-c7dd-345f-4a0c542c5401@felt.demon.nl>

I am willing to assist as best I can with AIX - I seem to have the core
requirements re: time available: (i.e., over-comitted at work, but
'work' evenings and weekends on OSS :p)


On 10/2/2018 6:41 PM, Simon Cross wrote:
> Are there any core devs that Michael or Erik could collaborate with?
> Rather than rely on adhoc patch review from random core developers.
>
> Michael and Eric: Question -- are you interested in becoming core
> developers at least for the purposes of maintaining these platforms in
> future?
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/aixtools%40felt.demon.nl


From aixtools at felt.demon.nl  Tue Oct  2 14:52:25 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Tue, 2 Oct 2018 20:52:25 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
Message-ID: <7c67a954-9e1f-2187-f027-2695109986b4@felt.demon.nl>



On 10/2/2018 4:45 PM, Erik Bray wrote:
> Michael, if there are any PRs you want to point me to that I might be
> able to help review please do.
A little trick I learned:
https://github.com/python/cpython/pulls?q=is%3Aopen+is%3Apr+author%3Aaixtools+sort%3Aupdated-desc
lists them all.

What "flipped my switch" yesterday was discovering a PR that I was
gifted (by an ex? core-dev) and put in the system back in January is now
broken by a patch merged about two weeks ago. Worse, pieces of
test_ctypes(bitfields) that previously worked when using __xlc__ seem to
be broken. Which highlighted the "time pressure" of getting tests to
pass so that regressions can be seen.

If you let me know what info you would need (I gave lots of debug info
two years ago to get that initial fix).

And, I guess the other "larger" change re: test_distutils. Also, some
issues specific to xlc being different from gcc.

Those two do not show on the gccfarm buildbot.

Many thanks for the offer! I'll try to not take more than the hand offered!
>   I don't know anything about AIX either
> and am not a core dev so I can't have a final say.  But I've been
> hacking on CPython for a long time anyways, and might be able to help
> at least with some initial review.


From aixtools at felt.demon.nl  Tue Oct  2 16:05:11 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Tue, 2 Oct 2018 22:05:11 +0200
Subject: [Python-Dev] LDLAST variable in configure.ac
In-Reply-To: <DD7CAA94-87F6-4637-93E3-089FF62A6C5C@langa.pl>
References: <a295bfb3-a102-947d-4c14-cdfa3259ff3a@felt.demon.nl>
 <1538435240.4040554.1527177920.188EFF58@webmail.messagingengine.com>
 <B23E11B7-37BB-4B24-98B6-BFB7ECABD38A@felt.demon.nl>
 <DD7CAA94-87F6-4637-93E3-089FF62A6C5C@langa.pl>
Message-ID: <c09c6cde-54b3-9a08-f8f8-bb28f1d09a3d@felt.demon.nl>

Yes, unintended. It was only supposed to be signed, but "Send Later"?
encrypts it.

Unpacked version:



On 10/2/2018 1:07 AM, Benjamin Peterson wrote:
> On Mon, Oct 1, 2018, at 12:12, Michael Felt wrote:
>> Hi all,
>>
>> Before I submit a patch to increase the default MAXDATA setting for AIX
>> when in 32-bit mode - I want to know if I can put this LDFLAG setting in
>> LDLAST, or if I should introduce a new AC_SUBST() variable (e.g.,
>> LDMAXDATA).
> I think you should just put it in LDFLAGS.
I was wanting to avoid that, as LDFLAGS is an environmental variable.

At the surface, it appears Python is using PY_LDFLAGS (with
CONFIGURE_LDFLAGS coming from LDFLAGS during the ./configure moment.

A reason for a separate variable is that this particular option is only
relevant for the python EXE, and not for shared libraries and "other
things". IMHO, a reason for LDMAXDATA is because LDLAST is actually
already too widely used:

root at x066:[/data/prj/python/git/cpython-master]grep LDFLAGS *.in
Makefile.pre.in:CONFIGURE_LDFLAGS=????? @LDFLAGS@
Makefile.pre.in:# Avoid assigning CFLAGS, LDFLAGS, etc. so users can use
them on the
Makefile.pre.in:# Both CPPFLAGS and LDFLAGS need to contain the shell's
value for setup.py to
Makefile.pre.in:PY_LDFLAGS=???? $(CONFIGURE_LDFLAGS) $(LDFLAGS)
Makefile.pre.in:LDSHARED=?????? @LDSHARED@ $(PY_LDFLAGS)
Makefile.pre.in:BLDSHARED=????? @BLDSHARED@ $(PY_LDFLAGS)
Makefile.pre.in:OPENSSL_LDFLAGS=@OPENSSL_LDFLAGS@
Makefile.pre.in:??????? $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS)
$(PGO_PROF_GEN_FLAG)" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG)"
LIBS="$(LIBS)"
Makefile.pre.in:??????? $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS)
$(PGO_PROF_USE_FLAG)" LDFLAGS="$(LDFLAGS)"
Makefile.pre.in:??????? $(LINKCC) $(PY_LDFLAGS) $(LINKFORSHARED) -o $@
Programs/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)
Makefile.pre.in:???????? $(CC) -dynamiclib -Wl,-single_module
$(PY_LDFLAGS) -undefined dynamic_lookup
-Wl,-install_name,$(prefix)/lib/libpython$(LDVERSION).dylib
-Wl,-compatibility_version,$(VERSION) -Wl,-current_version,$(VERSION) -o
$@ $(LIBRARY_OBJS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
Makefile.pre.in:??????? $(CC) -o $(LDLIBRARY) $(PY_LDFLAGS) -dynamiclib \
Makefile.pre.in:??????? $(LINKCC) $(PY_LDFLAGS) $(LINKFORSHARED) -o $@
Programs/_testembed.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)
Makefile.pre.in:??????? $(LINKCC) $(PY_LDFLAGS) -o $@
Programs/_freeze_importlib.o $(LIBRARY_OBJS_OMIT_FROZEN) $(LIBS)
$(MODLIBS) $(SYSLIBS) $(LDLAST)
Makefile.pre.in:??????????????? $(CC) $(OPT) $(PY_LDFLAGS) $(PGENOBJS)
$(LIBS) -o $(PGEN)

The ONLY line that needs $LDMAXDATA is:

Makefile.pre.in:??????? $(LINKCC) $(PY_LDFLAGS) -o $@
Programs/_freeze_importlib.o $(LIBRARY_OBJS_OMIT_FROZEN) $(LIBS)
$(MODLIBS) $(SYSLIBS) $(LDLAST) $(LDMAXDATA)

or set $(LDLAST) at the end rather than append $(LDMAXDATA)
>> I have not looked yet, but I was thinking that MAYBE! LDLAST is intended
>> as a last resort variable that can be modified in Makefile.
> LDLAST looks vestigial from OSF/1 support and should probably be removed.


On 10/2/2018 2:51 PM, ?ukasz Langa wrote:
>> On 2 Oct 2018, at 12:29, Michael Felt <aixtools at felt.demon.nl> wrote:
>>
>> <mime-attachment><encrypted.asc>
> Michael, this message looks encrypted on my end. For people without your public key, it's impossible to read. This was probably unintentional on your end. In either case I'd avoid encrypting messages that go to public mailing lists.
>
> - ?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181002/af67fca3/attachment.html>

From breamoreboy at gmail.com  Tue Oct  2 17:15:24 2018
From: breamoreboy at gmail.com (Mark Lawrence)
Date: Tue, 2 Oct 2018 22:15:24 +0100
Subject: [Python-Dev] Change in Python 3's "round" behavior
In-Reply-To: <dc0c48a7-3fd5-f6e2-3b9e-a47760fa808b@felt.demon.nl>
References: <1519658635.56.0.467229070634.issue32956@psf.upfronthosting.co.za>
 <1537960170.25.0.545547206417.issue32956@psf.upfronthosting.co.za>
 <CANwREeUjUDTFmZ=hkMwsG6vxUvwYYEiuSaY4zF8SEL6J74FvoQ@mail.gmail.com>
 <5BAC70BB.2040707@canterbury.ac.nz>
 <20180927135327.GE19437@ando.pearwood.info>
 <234e01d4585e$829acc20$87d06460$@sdamon.com>
 <20180930121703.GK19437@ando.pearwood.info>
 <dc0c48a7-3fd5-f6e2-3b9e-a47760fa808b@felt.demon.nl>
Message-ID: <pp0n1a$1h4$1@blaine.gmane.org>

On 01/10/18 21:45, Michael Felt wrote:
> 
> On 9/30/2018 2:17 PM, Steven D'Aprano wrote:
>>   (It's also called Dutch Rounding.)
> 
> Ah - as to why - and from school! (as so-called intuitive! rather desired!).
> 
> A test score goes from 5.5 to 6.0 - which becomes passing.
> 
> Oh, do I recall my children's frustrations when they had a X.4Y score -
> that became X.0. Tears!
> 

Please do not reply to any message from Steven D'Aprano as you are also 
likely to get banned by the incompetent moderators.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From tjreedy at udel.edu  Tue Oct  2 17:34:43 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 2 Oct 2018 17:34:43 -0400
Subject: [Python-Dev] dear core-devs
In-Reply-To: <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
Message-ID: <pp0o5i$e2a$1@blaine.gmane.org>

On 10/2/2018 12:41 PM, Simon Cross wrote:
> Are there any core devs that Michael or Erik could collaborate with?
> Rather than rely on adhoc patch review from random core developers.

You two might collaborate with each other to the extent of reviewing 
some of each other's PRs.  That still leaves the issue of merging.

> Michael and Eric: Question -- are you interested in becoming core
> developers at least for the purposes of maintaining these platforms in
> future?

Since adhoc is not working to get merges, I had this same suggestion. 
Michael and Erik, I presume you have gotten some guidelines on what 
modifications to C code might be accepted, and what concerns people have.

I think for tests, a separate test_aix.py might be a good idea for 
aix-only tests, while modification of other tests might be limited to 
adding skips.  The idea would be to make it easy to remove aix stuff in 
the future if it again became unsupported.  Ditto for other specialized 
platforms.




-- 
Terry Jan Reedy


From aixtools at felt.demon.nl  Tue Oct  2 19:16:46 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Wed, 3 Oct 2018 01:16:46 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <pp0o5i$e2a$1@blaine.gmane.org>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
 <pp0o5i$e2a$1@blaine.gmane.org>
Message-ID: <d1a07423-d09a-626d-3f49-4e382d83e802@felt.demon.nl>



On 10/2/2018 11:34 PM, Terry Reedy wrote:
> On 10/2/2018 12:41 PM, Simon Cross wrote:
>> Are there any core devs that Michael or Erik could collaborate with?
>> Rather than rely on adhoc patch review from random core developers.
>
> You two might collaborate with each other to the extent of reviewing
> some of each other's PRs. 
Might be difficult. We both, or at least I, claim ignorance of the
others platform. I still have a lot of PEP to learn, and my idea of a
bug-fix (for Python2) was seen by core-dev as a feature change. I would
not feel comfortable trying to mentor someone in things PEP, etc..
> That still leaves the issue of merging.
How much confidence is there in all the "CI" tests? Does that not offer
sufficient confidence for a core-dev to press merge.
How about "master" continuing to be what it is, but insert a new
"pre-master" branch that the buildbots actually test on (e.g., what is
now the 3.X) and have a 3.8 buildbot - for what is now the "master".

PR would still be done based on master, but an "initial" merge would be
via the pre-master aka 3.X buildbot tests.

How "friendly" git is - that it not become such a workload to keep it
clean - I cannot say. Still learning to use git. Better, but still do
not want to assume it would be easy.

My hope is that it would make it easier to consider a "merge" step that
gets all the buildbots involved for even broader CI tests.

>
>> Michael and Eric: Question -- are you interested in becoming core
>> developers at least for the purposes of maintaining these platforms in
>> future?
>
> Since adhoc is not working to get merges, I had this same suggestion.
> Michael and Erik, I presume you have gotten some guidelines on what
> modifications to C code might be accepted, and what concerns people have.
imho: guidelines - paraphrased - as little as possible :)

I have many assumptions, and one of those is that my assumptions are
probably incorrect.
Goal: have AIX recognized as a Stable platform, even if not in the
highest supported category.
And that implies, support as far as I am able, to keep it "Stable".
>
> I think for tests, a separate test_aix.py might be a good idea for
> aix-only tests
Unclear to me how this would work. Too young in Python I guess (or just
a very old dog), but what test would be needed for AIX, or any other
platform, that would not need to be tested in some fashion for the
'other' platforms. At a hunch, where there are many platform.system()
dependencies expected (e.g., test_posix, maybe doing something in the
class definition (is there a "Root" Object/Class that all inherit from.
Maybe a (read-only) "root" attribute (or is property better?) could be
the value of platform.system(), and iirc, might be used by as @property
in unittest. (so, if not in "root" class, then in something like
unittest/__init__.py.

I hope to be "close" in "Python thinking" - enough that someone who
actually knows how the pieces fit together could come with a better, and
more appropriate guideline/implementation.

> , while modification of other tests might be limited to adding skips.?
> The idea would be to make it easy to remove aix stuff in the future if
> it again became unsupported.
IMHO: IBM and AIX do not mention it, but for openstack cloudmanagement
(very specifically cloud-init) AIX needs a recognized stable Python
implementation. I am "surprised" in the level of communication of IBM
with Python community.

Personally, I do not see AIX as a specialized platform. Feels more like
the "last-standing" fully supported (commercial OEM) 'POSIX-UNIX'. Of
course my focus is narrow - so maybe there is a lot of support for
commercial platforms such as HPUX, Solaris, and other mainstream UNIXes.
Feel free to correct me!!
> Ditto for other specialized platforms.
>
>
>
>


From nas-python at arctrix.com  Tue Oct  2 19:46:26 2018
From: nas-python at arctrix.com (Neil Schemenauer)
Date: Tue, 2 Oct 2018 17:46:26 -0600
Subject: [Python-Dev] dear core-devs
In-Reply-To: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
Message-ID: <20181002234626.uaz5qisonzseywiz@python.ca>

On 2018-10-02, Michael Felt wrote:
> I am sorry, for myself obviously - but also for Python. Obviously, I am
> doing it all wrong - as I see lots of other issues being picked up
> immediately.

I'm not sure that's the case.  There are a lot of PRs or bugs that
sit there without getting reviews.  The problem is that few (or no)
core developers get paid to work on Python.  So, the time they spend
is motivated by their specific "itch".  Getting reviews on any PR is
difficult, even for core developers.  In their case, they have to
option of forcing the issue, I guess.

This is a problem we should try to deal with somehow.  Turning off
valuable contributors like you is bad.  I'm not sure how to do it
though.  At the core Python sprint in September there was some talk
about how CPython developers might get funding.  Maybe that could
help deal with the backlog of reviews required.

> And, while you may not give a damn about anything other than Windows,
> macos and/or Linux - there are other platforms that would like a stable
> Python.

There is probably some truth in not caring about other platforms.
The problem from the reviewer perspective is the question of "what
is the potential downsides of this PR vs what are the benefits?".
The safest thing is to not approve the PR.  No core developer wants
to be the person who broke CPython.  You must admit, AIX is an
extremely niche platform at this point.  I bet if you picked 1000
software developers at random, it would be likely that zero of them
have ever used AIX.  So, it's not that we don't care at all about
AIX but that the cost/benefit equation makes accepting AIX specific
changes more difficult.

One specific suggestion I have about your PR is to try to make your
changes not AIX specific.  Or at least, make the AIX checking as
localized as possible.  So, as an example, in test_uuid you have:

    _notAIX = not sys.platform.startswith("aix")

then later in the module you check that flag.  While that is the
most direct approach to fixing the issue and making the test pass,
it is not good for the long term maintainability of the code.  You
end up with boolean flags like _notAIX spread about the logic.  Over
time, code like that becomes a nightmare to maintain.

Instead, I would suggest test_uuid is making platform specific
assumptions that are not true on AIX and possibly other platforms.
So, do something like:

    
    _IS_AIX = sys.platform.startswith("aix")

    _HAVE_MACADDR = (os.name == 'posix' and not _IS_AIX)

    @unittest.skipUnless(_HAVE_MACADDR, 'requires Posix with macaddr')
    def test_arp_getnode(self):
        ...

The _HAVE_MACADDR test is relatively simple and clear, does this
platform have this capability.  Later in the code, a check for
_HAVE_MACADDR is also quite clear.  If someone comes along with
another platform that doesn't support macaddr, they only have to
change one line of code.

This kind of capability checking is similar to what happened with
web browsers.  In that case, people discovered that checking the
User Agent header was a bad idea.  Instead, you should probe for
specific functionality and not assume based on browser IDs.  For the
macaddr case, is there some way to you probe the arp command to see
if supports macaddr?  That way your test doesn't have to include any
AIX specific check at all.  Further, it would have some hope of
working on platforms other than AIX that also don't support macaddr
but are POSIX and have 'arp'.  The code could be something like:

_HAVE_MACADDR = False
if os.name == 'posix':
    if <check arp to see if it supports macaddr>:
        _HAVE_MACADDR = True

Hope that is helpful.

  Neil

From tjreedy at udel.edu  Tue Oct  2 20:48:11 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 2 Oct 2018 20:48:11 -0400
Subject: [Python-Dev] dear core-devs
In-Reply-To: <d1a07423-d09a-626d-3f49-4e382d83e802@felt.demon.nl>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
 <pp0o5i$e2a$1@blaine.gmane.org>
 <d1a07423-d09a-626d-3f49-4e382d83e802@felt.demon.nl>
Message-ID: <pp13ga$i2i$1@blaine.gmane.org>

On 10/2/2018 7:16 PM, Michael Felt wrote:
> 
> 
> On 10/2/2018 11:34 PM, Terry Reedy wrote:
>> On 10/2/2018 12:41 PM, Simon Cross wrote:
>>> Are there any core devs that Michael or Erik could collaborate with?
>>> Rather than rely on adhoc patch review from random core developers.
>>
>> You two might collaborate with each other to the extent of reviewing
>> some of each other's PRs.

> Might be difficult. We both, or at least I, claim ignorance of the
> others platform.

Partial reviews, short of accept/change are better than no review and 
can make a merge decision easier for a core dev.  You should each be or 
become familiar with PEP 7 and somewhat familiar with local C idioms. 
Do names follow local standards.  Do C-API calls make sense.

 >>  I still have a lot of PEP to learn, and my idea of a
 >> bug-fix (for Python2) was seen by core-dev as a feature change.

Failures of current tests would seem to me to be bugs.  However, some 
bug fixes require a feature change.  It is an awkward situation.  We are 
increasingly reluctant to patch 2.7.

>> That still leaves the issue of merging.
> How much confidence is there in all the "CI" tests? Does that not offer
> sufficient confidence for a core-dev to press merge.

Code for new features or bugs that escaped the tests should have new 
tests.  AIX-specific code should (as in must ;-) be tested before being 
submitted, since it will not be properly tested by CI.  With CI now 
covering Windows twice, Linux twice, and Mac, I believe it has become 
rarer for buildbots to fail after CI passes.  Victor would know.

I  believe that you are initially dealing with bugs that do not pass 
current tests.

> How about "master" continuing to be what it is, but insert a new
> "pre-master" branch that the buildbots actually test on (e.g., what is
> now the 3.X) and have a 3.8 buildbot - for what is now the "master".
> 
> PR would still be done based on master, but an "initial" merge would be
> via the pre-master aka 3.X buildbot tests.
> 
> How "friendly" git is - that it not become such a workload to keep it
> clean - I cannot say. Still learning to use git. Better, but still do
> not want to assume it would be easy.

Too complicated.

> My hope is that it would make it easier to consider a "merge" step that
> gets all the buildbots involved for even broader CI tests.

I considered the wider buildbot fleet to be post-merge CI ;-).

>> I think for tests, a separate test_aix.py might be a good idea for
>> aix-only tests

I may be wrong on this.


From erik.m.bray at gmail.com  Wed Oct  3 05:55:51 2018
From: erik.m.bray at gmail.com (Erik Bray)
Date: Wed, 3 Oct 2018 11:55:51 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
Message-ID: <CAOTD34ZR_86jKv5QiMX=FfTpih1xZpg7sOAhweMa+sX9fyCXgA@mail.gmail.com>

On Tue, Oct 2, 2018 at 6:41 PM Simon Cross
<hodgestar+pythondev at gmail.com> wrote:
>
> Are there any core devs that Michael or Erik could collaborate with?
> Rather than rely on adhoc patch review from random core developers.
>
> Michael and Eric: Question -- are you interested in becoming core
> developers at least for the purposes of maintaining these platforms in
> future?

I would be for the purposes of said platform maintenance.  I believe I
already have some maintainer permissions on bpo for exactly this
reason.

That said, while I'm sure it would help, I'm not exactly sure what it
would solve either.  I believe strongly in code review, and just
having a "core developer" status does not necessarily free one from
responsibility for obtaining code review.

It also partly depends on the issue.  If it's a change that touches
other parts of the code in ways that could impact it beyond the narrow
scope of platform support, I believe it definitely should get a second
pair of eyes.  Unfortunately many of the outstanding patches I have
for review fall in that category.  Though in the future there will be
fewer like that.  The majority of work needed for Cygwin, at least, is
tweaking some areas of the tests that make assumptions that don't
necessarily hold on that platform.*

Thanks,
E


* For example, there are some tests that assume there is a user with
UID 0.  While UID 0 is reserved for a "superuser", I don't know that
there's any requirement that such a user *must* exist (on Cygwin it
does not :)

From erik.m.bray at gmail.com  Wed Oct  3 05:58:33 2018
From: erik.m.bray at gmail.com (Erik Bray)
Date: Wed, 3 Oct 2018 11:58:33 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <7c67a954-9e1f-2187-f027-2695109986b4@felt.demon.nl>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <7c67a954-9e1f-2187-f027-2695109986b4@felt.demon.nl>
Message-ID: <CAOTD34bLkVKioZOdcQKZoT=nn22ZGs1b2PA1TaVg1AsZJVMi-Q@mail.gmail.com>

On Tue, Oct 2, 2018 at 8:54 PM Michael Felt <aixtools at felt.demon.nl> wrote:
>
>
>
> On 10/2/2018 4:45 PM, Erik Bray wrote:
> > Michael, if there are any PRs you want to point me to that I might be
> > able to help review please do.
> A little trick I learned:
> https://github.com/python/cpython/pulls?q=is%3Aopen+is%3Apr+author%3Aaixtools+sort%3Aupdated-desc
> lists them all.

Cool, I'll have a look.

> What "flipped my switch" yesterday was discovering a PR that I was
> gifted (by an ex? core-dev) and put in the system back in January is now
> broken by a patch merged about two weeks ago. Worse, pieces of
> test_ctypes(bitfields) that previously worked when using __xlc__ seem to
> be broken. Which highlighted the "time pressure" of getting tests to
> pass so that regressions can be seen.

Yes, that can certainly happen.  I have many PRs floating around on
different projects that, you know, get stalled for months and months
and inevitably break.  It's extremely frustrating, but we've all been
there :)

> If you let me know what info you would need (I gave lots of debug info
> two years ago to get that initial fix).
>
> And, I guess the other "larger" change re: test_distutils. Also, some
> issues specific to xlc being different from gcc.
>
> Those two do not show on the gccfarm buildbot.
>
> Many thanks for the offer! I'll try to not take more than the hand offered!
> >   I don't know anything about AIX either
> > and am not a core dev so I can't have a final say.  But I've been
> > hacking on CPython for a long time anyways, and might be able to help
> > at least with some initial review.

I also about to ask if you have a buildbot for AIX, and I see now you
have several. So step in the right direction!

From J.Demeyer at UGent.be  Wed Oct  3 08:12:13 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Wed, 3 Oct 2018 14:12:13 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
Message-ID: <5BB4B21D.2070302@UGent.be>

Hello,

I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, 
titled "The C call protocol". He has co-authored several PEPs (PEP 394, 
PEP 489, PEP 534, PEP 547, PEP 573), several of which involve extension 
modules.

Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also 
Antoine Pitrou, INADA Naoki and Nick Coghlan have approved Petr being 
BDFL-Delegate.

I am well aware of the current governance issues, but several people 
have mentioned that the BDFL-Delegate process can still continue for 
now. I created a PR for the peps repository at 
https://github.com/python/peps/pull/797


Cheers,
Jeroen Demeyer.

From wes.turner at gmail.com  Wed Oct  3 09:29:16 2018
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 3 Oct 2018 09:29:16 -0400
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <5BB4B21D.2070302@UGent.be>
References: <5BB4B21D.2070302@UGent.be>
Message-ID: <CACfEFw_yaBuS1G6F7rG4qyfsHiRkFgErtswQDCpQap+zB90AAg@mail.gmail.com>

On Wednesday, October 3, 2018, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> Hello,
>
> I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, titled
> "The C call protocol". He has co-authored several PEPs (PEP 394, PEP 489,
> PEP 534, PEP 547, PEP 573), several of which involve extension modules.
>
> Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also Antoine
> Pitrou, INADA Naoki and Nick Coghlan have approved Petr being BDFL-Delegate.
>
> I am well aware of the current governance issues, but several people have
> mentioned that the BDFL-Delegate process can still continue for now. I
> created a PR for the peps repository at https://github.com/python/peps
> /pull/797


+1. Are we doing upvotes on the mailing list or on the GitHub PR and/or
issue now?

"[Python-ideas] PEPs: Theory of operation"
https://markmail.org/thread/zr4o6l7ivnj4irtp

"""
Process suggestions that could minimize non-BDFL's BDFL legwork:

[...]

* Use GitHub reactions for voting on BDFL delegates, PEP final approval,
and PEP sub issues?
  * Specify a voting deadline?
  * How to make a quorum call?
  * Add '@core/team' as reviewers for every PEP?


* Link to the mailing list thread(s) at the top of the PR
  * [ ] Add unique message URLs to footers with mailman3


* What type of communications are better suited for mailing lists over PEP
pull-requests and PEP code reviews?
[The original thread is probably a better place to discuss PEP process
going forward]
"""


>
>
> Cheers,
> Jeroen Demeyer.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.
> turner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/460a8863/attachment.html>

From songofacandy at gmail.com  Wed Oct  3 09:51:24 2018
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 3 Oct 2018 22:51:24 +0900
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <5BB4B21D.2070302@UGent.be>
References: <5BB4B21D.2070302@UGent.be>
Message-ID: <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>

2018?10?3?(?) 21:24 Jeroen Demeyer <J.Demeyer at ugent.be>:

> Hello,
>
>
> I am well aware of the current governance issues, but several people
> have mentioned that the BDFL-Delegate process can still continue for
> now.


Really?
I don't know process to assign BDFL-delegate without BDFL.


This PEP is mainly for third party tools.
I want to get much feedback from them before new APIs become stable (e.g.
3.8b1)

So I want this PEP is approved (or
Provisionally Accepted) and reference implementation is merged as fast as
possible.

Regards,

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/03b43b3a/attachment.html>

From lukasz at langa.pl  Wed Oct  3 11:06:53 2018
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Wed, 3 Oct 2018 17:06:53 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
Message-ID: <08061C62-97C5-4245-81FA-DC624F4D8730@langa.pl>


> On 3 Oct 2018, at 15:51, INADA Naoki <songofacandy at gmail.com> wrote:
> 
> 
> 2018?10?3?(?) 21:24 Jeroen Demeyer <J.Demeyer at ugent.be <mailto:J.Demeyer at ugent.be>>:
> Hello,
> 
> 
> I am well aware of the current governance issues, but several people
> have mentioned that the BDFL-Delegate process can still continue for
> now.
> 
> Really?
> I don't know process to assign BDFL-delegate without BDFL.

My understand is that accepting *any* PEP by anyone is out of the question until the governance situation gets resolved. That's the only reason why PEP 544 is not yet accepted for example.

- ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/a29cde26/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/a29cde26/attachment.sig>

From lukasz at langa.pl  Wed Oct  3 11:12:04 2018
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Wed, 3 Oct 2018 17:12:04 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <08061C62-97C5-4245-81FA-DC624F4D8730@langa.pl>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <08061C62-97C5-4245-81FA-DC624F4D8730@langa.pl>
Message-ID: <820E6A21-6112-4956-8003-02F220277EEB@langa.pl>


> On 3 Oct 2018, at 17:06, ?ukasz Langa <lukasz at langa.pl> wrote:
> 
> My understand is

?????

...and no ability to edit to correct it. It's like this forever now, my grand children will ridicule me for this.

- ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/36719557/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/36719557/attachment.sig>

From wes.turner at gmail.com  Wed Oct  3 11:12:26 2018
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 3 Oct 2018 11:12:26 -0400
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
Message-ID: <CACfEFw-GgT9z-112z6izHho=LHphgYGUEzAiuxnVgOmE7SoZmw@mail.gmail.com>

On Wednesday, October 3, 2018, INADA Naoki <songofacandy at gmail.com> wrote:

>
> 2018?10?3?(?) 21:24 Jeroen Demeyer <J.Demeyer at ugent.be>:
>
>> Hello,
>>
>>
>> I am well aware of the current governance issues, but several people
>> have mentioned that the BDFL-Delegate process can still continue for
>> now.
>
>
> Really?
> I don't know process to assign BDFL-delegate without BDFL.
>

AFAIU, there is not yet a documented process for BDFL-delegate assignment.

There's this in the devguide; which links to PEP1:

"20.2. PEP Process?"
https://devguide.python.org/langchanges/#pep-process
https://github.com/python/devguide/blob/master/langchanges.rst


And PEP 1:

"PEP 1 -- PEP Purpose and Guidelines"
  "PEP Workflow"
https://www.python.org/dev/peps/pep-0001/#pep-workflow
  "PEP Editors"
https://www.python.org/dev/peps/pep-0001/#pep-editors
  "PEP Editor Responsibilities & Workflow"
https://www.python.org/dev/peps/pep-0001/#pep-editor-responsibilities-workflow

https://github.com/python/peps/blob/master/pep-0001.txt

And the devguide has a list of experts:
https://devguide.python.org/experts/


Maybe PEP1 is the place to list current BDFL-Delegates
(in addition to in the PEP metadata as in the OT PR:
python/peps#797
"PEP 580: Petr Viktorin as BDFL-Delegate"
)?


Not to bikeshed, but is BDFL-Delegate still the current term because that's
what's in all the other PEPs' metadata?


>
> This PEP is mainly for third party tools.
> I want to get much feedback from them before new APIs become stable (e.g.
> 3.8b1)
>
> So I want this PEP is approved (or
> Provisionally Accepted) and reference implementation is merged as fast as
> possible.
>
> Regards,
>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/08746f6d/attachment-0001.html>

From J.Demeyer at UGent.be  Wed Oct  3 11:19:05 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Wed, 3 Oct 2018 17:19:05 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <d3999ce47a9242aba67500d1e4802c02@xmail103.UGent.be>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <d3999ce47a9242aba67500d1e4802c02@xmail103.UGent.be>
Message-ID: <5BB4DDE9.5010309@UGent.be>

On 2018-10-03 17:12, Wes Turner wrote:
> AFAIU, there is not yet a documented process for BDFL-delegate assignment.

PEP 1 says:

"""
However, whenever a new PEP is put forward, any core developer that
believes they are suitably experienced to make the final decision on
that PEP may offer to serve as the BDFL's delegate (or "PEP czar") for
that PEP. If their self-nomination is accepted by the other core
developers and the BDFL, then they will have the authority to approve
(or reject) that PEP.
"""

I know that it says "core developers and the BDFL". However, if the core
developers agree that Petr can become BDFL-Delegate, I don't see why
that wouldn't be possible.


Jeroen.

From steve at pearwood.info  Wed Oct  3 11:59:37 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 4 Oct 2018 01:59:37 +1000
Subject: [Python-Dev] Should assert continue to do a LOAD_GLOBAL on
 AssertionError?
Message-ID: <20181003155937.GM21220@ando.pearwood.info>

On the bug tracker, there's a discussion about the current behaviour of 
the assert statement, where shadowing AssertionError will change the 
behaviour of the assertion.

https://bugs.python.org/issue34880

Currently, assert does a LOAD_GLOBAL on AssertionError, which means if 
you shadow the name, you get a different exception. This behaviour goes 
back to Python 1.5.

I'm looking for guidance here, is this the intended behaviour, or an 
accident? Should it be changed to better match other builtins?

(For example, shadowing iter doesn't effect for loops.)


Thanks,


Steve

From J.Demeyer at UGent.be  Wed Oct  3 12:08:59 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Wed, 3 Oct 2018 18:08:59 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <0d922ba1b09b46ad8efc08056c6e5771@xmail103.UGent.be>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <0d922ba1b09b46ad8efc08056c6e5771@xmail103.UGent.be>
Message-ID: <5BB4E99B.3000905@UGent.be>

On 2018-10-03 17:06, ?ukasz Langa wrote:
> That's the only
> reason why PEP 544 is not yet accepted for example.

Did you actually try to get PEP 544 accepted or to appoint a 
BDFL-Delegate? I don't find any discussions about PEP 544 after the 
stepping down of the BDFL.

From guido at python.org  Wed Oct  3 12:41:41 2018
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Oct 2018 09:41:41 -0700
Subject: [Python-Dev] PEP 544 status (forked off "Petr Viktorin as
 BDFL-Delegate for PEP 580")
In-Reply-To: <5BB4E99B.3000905@UGent.be>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <0d922ba1b09b46ad8efc08056c6e5771@xmail103.UGent.be>
 <5BB4E99B.3000905@UGent.be>
Message-ID: <CAP7+vJJfJugqYFBhP8Dtf0=X3ju7xGFh43BXrXxiS_5dvG1iug@mail.gmail.com>

The process for PEP 544 is off-topic for that thread so I'm starting a new
one.

I have promised its author to approve it after certain minor changes (that
we both agree on) have been made. It's not an example of how PEP acceptance
in general will works until governance is sorted out though -- PEP 544 is a
very unique special case. (For one, it's uncontroversial -- the reason it's
not been accepted yet is that its author is busy with other things.)

On Wed, Oct 3, 2018 at 9:12 AM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> On 2018-10-03 17:06, ?ukasz Langa wrote:
> > That's the only
> > reason why PEP 544 is not yet accepted for example.
>
> Did you actually try to get PEP 544 accepted or to appoint a
> BDFL-Delegate? I don't find any discussions about PEP 544 after the
> stepping down of the BDFL.
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/50feaf28/attachment.html>

From barry at python.org  Wed Oct  3 12:55:41 2018
From: barry at python.org (Barry Warsaw)
Date: Wed, 3 Oct 2018 09:55:41 -0700
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <08061C62-97C5-4245-81FA-DC624F4D8730@langa.pl>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <08061C62-97C5-4245-81FA-DC624F4D8730@langa.pl>
Message-ID: <09935642-FB02-4183-A6D6-2F22126A5119@python.org>

On Oct 3, 2018, at 08:06, ?ukasz Langa <lukasz at langa.pl> wrote:
> 
>> On 3 Oct 2018, at 15:51, INADA Naoki <songofacandy at gmail.com> wrote:
>> 
>> Really?
>> I don't know process to assign BDFL-delegate without BDFL.
> 
> My understand is that accepting *any* PEP by anyone is out of the question until the governance situation gets resolved. That's the only reason why PEP 544 is not yet accepted for example.

Correct.  It?s entirely possible that the different governance models will have different ways to pick delegates.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/14f025bb/attachment.sig>

From aixtools at felt.demon.nl  Wed Oct  3 12:57:00 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Wed, 3 Oct 2018 18:57:00 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <pp13ga$i2i$1@blaine.gmane.org>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
 <pp0o5i$e2a$1@blaine.gmane.org>
 <d1a07423-d09a-626d-3f49-4e382d83e802@felt.demon.nl>
 <pp13ga$i2i$1@blaine.gmane.org>
Message-ID: <a1f8f4eb-67b1-f7c7-c931-f218884363dc@felt.demon.nl>



On 10/3/2018 2:48 AM, Terry Reedy wrote:
> On 10/2/2018 7:16 PM, Michael Felt wrote:
>>
>>
>> On 10/2/2018 11:34 PM, Terry Reedy wrote:
>>> On 10/2/2018 12:41 PM, Simon Cross wrote:
>>>> Are there any core devs that Michael or Erik could collaborate with?
>>>> Rather than rely on adhoc patch review from random core developers.
>>>
>>> You two might collaborate with each other to the extent of reviewing
>>> some of each other's PRs.
>
>> Might be difficult. We both, or at least I, claim ignorance of the
>> others platform.
>
> Partial reviews, short of accept/change are better than no review and
> can make a merge decision easier for a core dev.? You should each be
> or become familiar with PEP 7 and somewhat familiar with local C
> idioms. Do names follow local standards.? Do C-API calls make sense.
Sounds simple enough. The tricky part is "the details".
>
> >>? I still have a lot of PEP to learn, and my idea of a
> >> bug-fix (for Python2) was seen by core-dev as a feature change.
>
> Failures of current tests would seem to me to be bugs.? However, some
> bug fixes require a feature change.? It is an awkward situation.? We
> are increasingly reluctant to patch 2.7.
Some are quite simple to fix, even if hard to find: such as:
"elif cmd is None:" -> "elif notcmd orcmd is None:"

Some are not bugs at all - very hard to find! Instead, "textual"
differences because a library is overly optimized - the expected
exception occurs - but no error message. Linking with a less optimized
(libssl.a and libcrypto.a) resolved many reported test "failures".

Nearly three years ago I was keen to see things in Python2(.7), but not
so much now. I also feel the time is to push hard towards current
Python3 versions.
>
>>> That still leaves the issue of merging.
>> How much confidence is there in all the "CI" tests? Does that not offer
>> sufficient confidence for a core-dev to press merge.
>
> Code for new features or bugs that escaped the tests should have new
> tests.? AIX-specific code should (as in must ;-) be tested before
> being submitted, since it will not be properly tested by CI.? With CI
> now covering Windows twice, Linux twice, and Mac, I believe it has
> become rarer for buildbots to fail after CI passes.? Victor would know.
>
> I? believe that you are initially dealing with bugs that do not pass
> current tests.
I am dealing with tests that do not pass. The dilemma: what is wrong -
the test, or what it is testing? Generally speaking, I cannot call
Python3 (master) broken. So I look for a "root cause" in a test
assumption that is wrong, and find a way to correct that.

Sometimes, it is a bit of both - and those are very hard to resolve
without feedback.

See the discussion, elsewhere, regarding MACADDR. It has never been that
platform Y does not have a MACADDR - rather, platform Y formats it
differently than (all) other platforms.

>
>> How about "master" continuing to be what it is, but insert a new
>> "pre-master" branch that the buildbots actually test on (e.g., what is
>> now the 3.X) and have a 3.8 buildbot - for what is now the "master".
>>
>> PR would still be done based on master, but an "initial" merge would be
>> via the pre-master aka 3.X buildbot tests.
>>
>> How "friendly" git is - that it not become such a workload to keep it
>> clean - I cannot say. Still learning to use git. Better, but still do
>> not want to assume it would be easy.
>
> Too complicated.
>
>> My hope is that it would make it easier to consider a "merge" step that
>> gets all the buildbots involved for even broader CI tests.
>
> I considered the wider buildbot fleet to be post-merge CI ;-).
>
>>> I think for tests, a separate test_aix.py might be a good idea for
>>> aix-only tests
>
> I may be wrong on this.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/aixtools%40felt.demon.nl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/d41874dd/attachment.html>

From guido at python.org  Wed Oct  3 13:15:03 2018
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Oct 2018 10:15:03 -0700
Subject: [Python-Dev] Should assert continue to do a LOAD_GLOBAL on
 AssertionError?
In-Reply-To: <20181003155937.GM21220@ando.pearwood.info>
References: <20181003155937.GM21220@ando.pearwood.info>
Message-ID: <CAP7+vJJ2j1X08TStEfZrYCwVHdD85DXjrsFabNSA=5gE+=mLNg@mail.gmail.com>

Feels like an accident to me. Generally syntactic constructs should be
unaffected by what's in any namespace except when the override is
intentional (e.g. __import__).

On Wed, Oct 3, 2018 at 9:02 AM Steven D'Aprano <steve at pearwood.info> wrote:

> On the bug tracker, there's a discussion about the current behaviour of
> the assert statement, where shadowing AssertionError will change the
> behaviour of the assertion.
>
> https://bugs.python.org/issue34880
>
> Currently, assert does a LOAD_GLOBAL on AssertionError, which means if
> you shadow the name, you get a different exception. This behaviour goes
> back to Python 1.5.
>
> I'm looking for guidance here, is this the intended behaviour, or an
> accident? Should it be changed to better match other builtins?
>
> (For example, shadowing iter doesn't effect for loops.)
>
>
> Thanks,
>
>
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/e4dd9674/attachment.html>

From aixtools at felt.demon.nl  Wed Oct  3 13:22:26 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Wed, 3 Oct 2018 19:22:26 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <20181002234626.uaz5qisonzseywiz@python.ca>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <20181002234626.uaz5qisonzseywiz@python.ca>
Message-ID: <55451d2a-6f2d-c453-9d5a-c863c795f443@felt.demon.nl>



On 10/3/2018 1:46 AM, Neil Schemenauer wrote:
> On 2018-10-02, Michael Felt wrote:
>> I am sorry, for myself obviously - but also for Python. Obviously, I am
>> doing it all wrong - as I see lots of other issues being picked up
>> immediately.
> I'm not sure that's the case.  There are a lot of PRs or bugs that
> sit there without getting reviews.  The problem is that few (or no)
> core developers get paid to work on Python.  So, the time they spend
> is motivated by their specific "itch".  Getting reviews on any PR is
> difficult, even for core developers.  In their case, they have to
> option of forcing the issue, I guess.
>
> This is a problem we should try to deal with somehow.  Turning off
> valuable contributors like you is bad.  I'm not sure how to do it
> though.  At the core Python sprint in September there was some talk
> about how CPython developers might get funding.  Maybe that could
> help deal with the backlog of reviews required.
>
>> And, while you may not give a damn about anything other than Windows,
>> macos and/or Linux - there are other platforms that would like a stable
>> Python.
> There is probably some truth in not caring about other platforms.
> The problem from the reviewer perspective is the question of "what
> is the potential downsides of this PR vs what are the benefits?".
> The safest thing is to not approve the PR.  No core developer wants
> to be the person who broke CPython.  You must admit, AIX is an
> extremely niche platform at this point.  I bet if you picked 1000
> software developers at random, it would be likely that zero of them
> have ever used AIX.  So, it's not that we don't care at all about
> AIX but that the cost/benefit equation makes accepting AIX specific
> changes more difficult.
Nods. However - this is a chicken/egg issue (imho). AIX is seen a weak
platform because noone has ever tackled these. When I started on this I
had never expected to have found a resolution to them all.

Platforms have differences and when the tests miss that difference that
the tests give a false result. e.g., one accepted PR was because AIX
libc printf() output for printf(NULL) is "" while other platforms output
"(null)".


>
> One specific suggestion I have about your PR is to try to make your
> changes not AIX specific.  Or at least, make the AIX checking as
> localized as possible.  So, as an example, in test_uuid you have:
>
>     _notAIX = not sys.platform.startswith("aix")
a) I thought/hoped this was better practice and performance - calling?
sys.platform.startswith("aix")only once, rather than X times.
b) more maintainable (e.g., change to not platform.system()
c) iirc - this got changed to AIX = ...., and throughout the test is "if
not AIX"...
>
> then later in the module you check that flag.  While that is the
> most direct approach to fixing the issue and making the test pass,
> it is not good for the long term maintainability of the code.  You
> end up with boolean flags like _notAIX spread about the logic.  Over
> time, code like that becomes a nightmare to maintain.
>
> Instead, I would suggest test_uuid is making platform specific
> assumptions that are not true on AIX and possibly other platforms.
> So, do something like:
>
>     
>     _IS_AIX = sys.platform.startswith("aix")
better name.
>
>     _HAVE_MACADDR = (os.name == 'posix' and not _IS_AIX)
AIX has MACADDR, but formatted with '.' rather than ':' and uses a
single hex-digit when value between dots is < 16 (decimal)
>
>     @unittest.skipUnless(_HAVE_MACADDR, 'requires Posix with macaddr')
>     def test_arp_getnode(self):
>         ...
>
> The _HAVE_MACADDR test is relatively simple and clear, does this
> platform have this capability.  Later in the code, a check for
> _HAVE_MACADDR is also quite clear.  If someone comes along with
> another platform that doesn't support macaddr, they only have to
> change one line of code.
>
> This kind of capability checking is similar to what happened with
> web browsers.  In that case, people discovered that checking the
> User Agent header was a bad idea.  Instead, you should probe for
> specific functionality and not assume based on browser IDs.  For the
> macaddr case, is there some way to you probe the arp command to see
> if supports macaddr? 
I suppose if someone had written the original test with "check program
to see if ..." it would have worked already.
I am trying to get current tests to work with minimal changes.

I am certainly not "blaming" anyone for not knowing this unique behavior
of this platform. Before debugging this I did not know of the difference
either. I agree that wherever a generic resolution is possible - it
should be done. However, as an "outsider" I do not feel empowered enough
to propose such a change to existing (and well proven for other
platforms) tests.
>  That way your test doesn't have to include any
> AIX specific check at all.  Further, it would have some hope of
> working on platforms other than AIX that also don't support macaddr
> but are POSIX and have 'arp'.  The code could be something like:
>
> _HAVE_MACADDR = False
> if os.name == 'posix':
>     if <check arp to see if it supports macaddr>:
>         _HAVE_MACADDR = True
>
> Hope that is helpful.
All feedback and constructive criticism is helpful. Thank you for taking
the time to share!
>
>   Neil


From solipsis at pitrou.net  Wed Oct  3 14:08:23 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Oct 2018 20:08:23 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
Message-ID: <20181003200823.034802d9@fsol>

On Wed, 3 Oct 2018 22:51:24 +0900
INADA Naoki <songofacandy at gmail.com> wrote:
> 2018?10?3?(?) 21:24 Jeroen Demeyer <J.Demeyer at ugent.be>:
> 
> > Hello,
> >
> >
> > I am well aware of the current governance issues, but several people
> > have mentioned that the BDFL-Delegate process can still continue for
> > now.  
> 
> 
> Really?
> I don't know process to assign BDFL-delegate without BDFL.

That's probably why we should call it "PEP-delegate" :-)

Consensus would obviously work (if no-one opposes the proposed person,
then surely we don't need an elaborate governance model to decree that
said person can become the PEP delegate, no?).

Regards

Antoine.



From solipsis at pitrou.net  Wed Oct  3 14:09:42 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Oct 2018 20:09:42 +0200
Subject: [Python-Dev] Should assert continue to do a LOAD_GLOBAL on
 AssertionError?
References: <20181003155937.GM21220@ando.pearwood.info>
Message-ID: <20181003200942.5360344e@fsol>

On Thu, 4 Oct 2018 01:59:37 +1000
Steven D'Aprano <steve at pearwood.info> wrote:
> On the bug tracker, there's a discussion about the current behaviour of 
> the assert statement, where shadowing AssertionError will change the 
> behaviour of the assertion.
> 
> https://bugs.python.org/issue34880
> 
> Currently, assert does a LOAD_GLOBAL on AssertionError, which means if 
> you shadow the name, you get a different exception. This behaviour goes 
> back to Python 1.5.
> 
> I'm looking for guidance here, is this the intended behaviour, or an 
> accident? Should it be changed to better match other builtins?

I would make it an implementation detail myself, i.e. any
implementation is free to make it work as it prefers.

Regards

Antoine.



From tjreedy at udel.edu  Wed Oct  3 17:10:17 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 3 Oct 2018 17:10:17 -0400
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <5BB4B21D.2070302@UGent.be>
References: <5BB4B21D.2070302@UGent.be>
Message-ID: <pp3b3o$tl2$1@blaine.gmane.org>

On 10/3/2018 8:12 AM, Jeroen Demeyer wrote:
> Hello,
> 
> I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, 
> titled "The C call protocol". He has co-authored several PEPs (PEP 394, 
> PEP 489, PEP 534, PEP 547, PEP 573), several of which involve extension 
> modules.
> 
> Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also 
> Antoine Pitrou, INADA Naoki and Nick Coghlan have approved Petr being 
> BDFL-Delegate.

To me, three experienced core devs approving of a 4th person as 
PEP-examiner is sufficient to proceed on a CPython implementation 
proposal.  I don't think we need to be paralyzed on this.  And indeed, 
when it comes to sub-PEP C-API changes, we seem not to be.  This change, 
if made, should be early in the cycle for the next version, rather than 
landing just before the first beta.

A language syntax-change proposal would be something else.


-- 
Terry Jan Reedy


From guido at python.org  Wed Oct  3 17:27:16 2018
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Oct 2018 14:27:16 -0700
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <pp3b3o$tl2$1@blaine.gmane.org>
References: <5BB4B21D.2070302@UGent.be> <pp3b3o$tl2$1@blaine.gmane.org>
Message-ID: <CAP7+vJJdC=05OBaHS4DKHbHQPn6K5SGCoTOg=M9t26v9m=QfpQ@mail.gmail.com>

On Wed, Oct 3, 2018 at 2:13 PM Terry Reedy <tjreedy at udel.edu> wrote:

> A language syntax-change proposal would be something else.
>

IMO changes to the C API should be taken just as seriously -- the potential
for breaking the world is just about the same (since most serious Python
applications use C extensions that risk breaking).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/5372b3fa/attachment.html>

From tjreedy at udel.edu  Wed Oct  3 18:14:47 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 3 Oct 2018 18:14:47 -0400
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <CAP7+vJJdC=05OBaHS4DKHbHQPn6K5SGCoTOg=M9t26v9m=QfpQ@mail.gmail.com>
References: <5BB4B21D.2070302@UGent.be> <pp3b3o$tl2$1@blaine.gmane.org>
 <CAP7+vJJdC=05OBaHS4DKHbHQPn6K5SGCoTOg=M9t26v9m=QfpQ@mail.gmail.com>
Message-ID: <pp3esr$82t$1@blaine.gmane.org>

On 10/3/2018 5:27 PM, Guido van Rossum wrote:
> On Wed, Oct 3, 2018 at 2:13 PM Terry Reedy <tjreedy at udel.edu 
> <mailto:tjreedy at udel.edu>> wrote:
> 
>     A language syntax-change proposal would be something else.
> 
> 
> IMO changes to the C API should be taken just as seriously -- the 
> potential for breaking the world is just about the same (since most 
> serious Python applications use C extensions that risk breaking).

I agree.  My observation is that PEP-delegates have taken their 
responsibility *very* seriously, and I think that the evidence is that 
Petr would.  If you think otherwise, please explain.  On reason for a 
serious examination to start now is to allow adequate time.

The difference I was referring to is the philosophical basis of and 
technical evaluation skill needed for a decision.  I feel competent to 
opine on syntax proposals, but not on technical details of CPyython 
calls.  Moreover, as long as an internal change does not break anything, 
and at least does not hinder writing C extensions, I have little reason 
to care, whereas syntax changes will affect me, even if they are 'good' 
changes.

-- 
Terry Jan Reedy


From guido at python.org  Wed Oct  3 18:23:58 2018
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Oct 2018 15:23:58 -0700
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <pp3esr$82t$1@blaine.gmane.org>
References: <5BB4B21D.2070302@UGent.be> <pp3b3o$tl2$1@blaine.gmane.org>
 <CAP7+vJJdC=05OBaHS4DKHbHQPn6K5SGCoTOg=M9t26v9m=QfpQ@mail.gmail.com>
 <pp3esr$82t$1@blaine.gmane.org>
Message-ID: <CAP7+vJLSyAPrFFQPkW2rFE9viB21daEQ4Mj17vEYHJBzhrCXfA@mail.gmail.com>

Well, it's not my call any more, so I'll happily stop arguing.

On Wed, Oct 3, 2018 at 3:19 PM Terry Reedy <tjreedy at udel.edu> wrote:

> On 10/3/2018 5:27 PM, Guido van Rossum wrote:
> > On Wed, Oct 3, 2018 at 2:13 PM Terry Reedy <tjreedy at udel.edu
> > <mailto:tjreedy at udel.edu>> wrote:
> >
> >     A language syntax-change proposal would be something else.
> >
> >
> > IMO changes to the C API should be taken just as seriously -- the
> > potential for breaking the world is just about the same (since most
> > serious Python applications use C extensions that risk breaking).
>
> I agree.  My observation is that PEP-delegates have taken their
> responsibility *very* seriously, and I think that the evidence is that
> Petr would.  If you think otherwise, please explain.  On reason for a
> serious examination to start now is to allow adequate time.
>
> The difference I was referring to is the philosophical basis of and
> technical evaluation skill needed for a decision.  I feel competent to
> opine on syntax proposals, but not on technical details of CPyython
> calls.  Moreover, as long as an internal change does not break anything,
> and at least does not hinder writing C extensions, I have little reason
> to care, whereas syntax changes will affect me, even if they are 'good'
> changes.
>
> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/b2aa760f/attachment.html>

From seanharr11 at gmail.com  Wed Oct  3 21:30:21 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Wed, 3 Oct 2018 21:30:21 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
Message-ID: <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>

Hi guys -

The solution to "lazily initialize" an expensive object in the worker
process (i.e. via @lru_cache) is a great solution (that I must admit I did
not think of). Additionally, in the second use case of "*passing a large
object to each worker process*", I also agree with your suggestion to
"shelter functions in a different module to avoid exposure to globals" as a
good solution if one is wary of globals.

That said, I still think "*passing a large object from parent process to
worker processes*" should be easier when using Pool. Would either of you be
open to something like the following?

           def func(x, big_cache=None):
               return big_cache[x]

           big_cache =  { str(k): k for k in range(10000) }

           ls = [ i for i in range(1000) ]

with Pool(func_kwargs={"big_cache": big_cache}) as pool:

    pool.map(func, ls)


It's a much cleaner interface (which presumably requires a more difficult
implementation) than my initial proposal. This also reads a lot better than
the "initializer + global" recipe (clear flow of data), and is less
constraining than the "define globals in parent" recipe. Most importantly,
when taking sequential code and parallelizing via Pool.map, this does not
force the user to re-implement "func" such that it consumes a global
(rather than a kwarg). It allows "func" to be used elsewhere (i.e. in the
parent process, from a different module, testing w/o globals, etc...)..

This would essentially be an efficient implementation of Pool.starmap(),
where kwargs are static, and passed to each application of "func" over our
iterable.

Thoughts?


On Sat, Sep 29, 2018 at 3:00 PM Michael Selik <mike at selik.org> wrote:

> On Sat, Sep 29, 2018 at 5:24 AM Sean Harrington <seanharr11 at gmail.com>
> wrote:
> >> On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington <seanharr11 at gmail.com>
> wrote:
> >> > My simple argument is that the developer should not be constrained to
> make the objects passed globally available in the process, as this MAY
> break encapsulation for large projects.
> >>
> >> I could imagine someone switching from Pool to ThreadPool and getting
> >> into trouble, but in my mind using threads is caveat emptor. Are you
> >> worried about breaking encapsulation in a different scenario?
> >
> > >> Without a specific example on-hand, you could imagine a tree of
> function calls that occur in the worker process (even newly created
> objects), that should not necessarily have access to objects passed from
> parent -> worker. In every case given the current implementation, they will.
>
> Echoing Antoine: If you want some functions to not have access to a
> module's globals, you can put those functions in a different module.
> Note that multiprocessing already encapsulates each subprocesses'
> globals in essentially a separate namespace.
>
> Without a specific example, this discussion is going to go around in
> circles. You have a clear aversion to globals. Antoine and I do not.
> No one else seems to have found this conversation interesting enough
> to participate, yet.


>>>

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181003/d162a147/attachment.html>

From benjamin at python.org  Wed Oct  3 22:38:34 2018
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 03 Oct 2018 19:38:34 -0700
Subject: [Python-Dev] Should assert continue to do a LOAD_GLOBAL on
 AssertionError?
In-Reply-To: <20181003155937.GM21220@ando.pearwood.info>
References: <20181003155937.GM21220@ando.pearwood.info>
Message-ID: <1538620714.2141716.1530108600.48469F20@webmail.messagingengine.com>



On Wed, Oct 3, 2018, at 08:59, Steven D'Aprano wrote:
> On the bug tracker, there's a discussion about the current behaviour of 
> the assert statement, where shadowing AssertionError will change the 
> behaviour of the assertion.
> 
> https://bugs.python.org/issue34880
> 
> Currently, assert does a LOAD_GLOBAL on AssertionError, which means if 
> you shadow the name, you get a different exception. This behaviour goes 
> back to Python 1.5.
> 
> I'm looking for guidance here, is this the intended behaviour, or an 
> accident? Should it be changed to better match other builtins?

The behavior certainly has been relied on historically by py.test. By replacing builtins.AssertionError, you can improve the error message of the AssertionError by, e.g., inspecting the failing frame. py.test's code to do this was deleted in 2016, but other code bases may still be relying on this hack. It's probably okay to change the behavior in 3.8 with the understanding that a revert may be necessary if some clever hack surfaces.

From ncoghlan at gmail.com  Thu Oct  4 02:14:53 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 4 Oct 2018 16:14:53 +1000
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <20181003200823.034802d9@fsol>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <20181003200823.034802d9@fsol>
Message-ID: <CADiSq7fX=2bLU4noJAiwS4J3bKwcL2AztsQJQvswTa1Pid7ddg@mail.gmail.com>

On Thu., 4 Oct. 2018, 4:12 am Antoine Pitrou, <solipsis at pitrou.net> wrote:

> On Wed, 3 Oct 2018 22:51:24 +0900
> INADA Naoki <songofacandy at gmail.com> wrote:
> > 2018?10?3?(?) 21:24 Jeroen Demeyer <J.Demeyer at ugent.be>:
> >
> > > Hello,
> > >
> > >
> > > I am well aware of the current governance issues, but several people
> > > have mentioned that the BDFL-Delegate process can still continue for
> > > now.
> >
> >
> > Really?
> > I don't know process to assign BDFL-delegate without BDFL.
>
> That's probably why we should call it "PEP-delegate" :-)
>
> Consensus would obviously work (if no-one opposes the proposed person,
> then surely we don't need an elaborate governance model to decree that
> said person can become the PEP delegate, no?).
>

I was figuring we could treat it as a caretaker mode governance: anything
we do before the new governance model is formalised is subject to
ratification by the new council members (or the new BDFL if that option
ends up being chosen).

In this case, I'd consider it unlikely for either the PEP delegate
appointment or any decisions about the PEP itself to be overturned - while
it's a complex topic that definitely needs to go through the PEP process in
order to work out the technical details, it isn't especially controversial
in its own right (the most controversial aspect is whether it needs a new C
level slot or not, and the PEP should clearly lay out the pros and cons of
that)

Cheers,
Nick.



> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181004/2baed804/attachment.html>

From J.Demeyer at UGent.be  Thu Oct  4 02:14:41 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Thu, 4 Oct 2018 08:14:41 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <16ccc51b3d0a46aebd544027fd81a6d1@xmail103.UGent.be>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <08061C62-97C5-4245-81FA-DC624F4D8730@langa.pl>
 <16ccc51b3d0a46aebd544027fd81a6d1@xmail103.UGent.be>
Message-ID: <5BB5AFD1.7010807@UGent.be>

On 2018-10-03 18:55, Barry Warsaw wrote:
> Correct.  It?s entirely possible that the different governance models will have different ways to pick delegates.

And how does that affect *today*'s decision? The new governance model 
will only take effect 1 January (assuming that everything goes as planned).

As long as there is no new governance model yet, can't we just continue 
the PEP 1 process which has worked for many years? I know that we cannot 
literally apply PEP 1 because there is no BDFL, but we can certainly 
continue the spirit of PEP 1 if the other core developers agree with the 
BDFL-Delegate.

From J.Demeyer at UGent.be  Thu Oct  4 02:29:41 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Thu, 4 Oct 2018 08:29:41 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <6b168c8c3a1d46668f89d601c0cedaab@xmail103.UGent.be>
References: <5BB4B21D.2070302@UGent.be> <pp3b3o$tl2$1@blaine.gmane.org>
 <6b168c8c3a1d46668f89d601c0cedaab@xmail103.UGent.be>
Message-ID: <5BB5B355.4040206@UGent.be>

On 2018-10-03 23:27, Guido van Rossum wrote:
> IMO changes to the C API should be taken just as seriously -- the
> potential for breaking the world is just about the same (since most
> serious Python applications use C extensions that risk breaking).

Of course we are taking this seriously. I want this to be taken as 
seriously as any other PEP and any other BDFL-Delegate appointment in 
the past.

To be clear: I'm not trying to rush my PEP though. It has been discussed 
and I have made changes to it based on comments. In fact, this is the 
second PEP with the same subject, I withdrew the first one, PEP 575. At 
some point in the past I asked one person to become BDFL-Delegate but he 
did not answer. And now recently Petr Viktorin made some insightful 
comments on it, so I asked him and he agreed.

From vstinner at redhat.com  Thu Oct  4 03:34:56 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 4 Oct 2018 09:34:56 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <d1a07423-d09a-626d-3f49-4e382d83e802@felt.demon.nl>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
 <pp0o5i$e2a$1@blaine.gmane.org>
 <d1a07423-d09a-626d-3f49-4e382d83e802@felt.demon.nl>
Message-ID: <CA+3bQGGTeuRBY1ibxissRV=7oLGocyeE3NRFCPRuNtuYa19A_A@mail.gmail.com>

Hi,

If IBM wants a better Python support, it would help a lot if IBM pays for
this development. With money, you can easily find core dev contractors.
Antoine Pitrou has been paid in the past to enhance Python support in
Solaris and it worked well.

Victor

Le mercredi 3 octobre 2018, Michael Felt <aixtools at felt.demon.nl> a ?crit :
>
>
> On 10/2/2018 11:34 PM, Terry Reedy wrote:
>> On 10/2/2018 12:41 PM, Simon Cross wrote:
>>> Are there any core devs that Michael or Erik could collaborate with?
>>> Rather than rely on adhoc patch review from random core developers.
>>
>> You two might collaborate with each other to the extent of reviewing
>> some of each other's PRs.
> Might be difficult. We both, or at least I, claim ignorance of the
> others platform. I still have a lot of PEP to learn, and my idea of a
> bug-fix (for Python2) was seen by core-dev as a feature change. I would
> not feel comfortable trying to mentor someone in things PEP, etc..
>> That still leaves the issue of merging.
> How much confidence is there in all the "CI" tests? Does that not offer
> sufficient confidence for a core-dev to press merge.
> How about "master" continuing to be what it is, but insert a new
> "pre-master" branch that the buildbots actually test on (e.g., what is
> now the 3.X) and have a 3.8 buildbot - for what is now the "master".
>
> PR would still be done based on master, but an "initial" merge would be
> via the pre-master aka 3.X buildbot tests.
>
> How "friendly" git is - that it not become such a workload to keep it
> clean - I cannot say. Still learning to use git. Better, but still do
> not want to assume it would be easy.
>
> My hope is that it would make it easier to consider a "merge" step that
> gets all the buildbots involved for even broader CI tests.
>
>>
>>> Michael and Eric: Question -- are you interested in becoming core
>>> developers at least for the purposes of maintaining these platforms in
>>> future?
>>
>> Since adhoc is not working to get merges, I had this same suggestion.
>> Michael and Erik, I presume you have gotten some guidelines on what
>> modifications to C code might be accepted, and what concerns people have.
> imho: guidelines - paraphrased - as little as possible :)
>
> I have many assumptions, and one of those is that my assumptions are
> probably incorrect.
> Goal: have AIX recognized as a Stable platform, even if not in the
> highest supported category.
> And that implies, support as far as I am able, to keep it "Stable".
>>
>> I think for tests, a separate test_aix.py might be a good idea for
>> aix-only tests
> Unclear to me how this would work. Too young in Python I guess (or just
> a very old dog), but what test would be needed for AIX, or any other
> platform, that would not need to be tested in some fashion for the
> 'other' platforms. At a hunch, where there are many platform.system()
> dependencies expected (e.g., test_posix, maybe doing something in the
> class definition (is there a "Root" Object/Class that all inherit from.
> Maybe a (read-only) "root" attribute (or is property better?) could be
> the value of platform.system(), and iirc, might be used by as @property
> in unittest. (so, if not in "root" class, then in something like
> unittest/__init__.py.
>
> I hope to be "close" in "Python thinking" - enough that someone who
> actually knows how the pieces fit together could come with a better, and
> more appropriate guideline/implementation.
>
>> , while modification of other tests might be limited to adding skips.
>> The idea would be to make it easy to remove aix stuff in the future if
>> it again became unsupported.
> IMHO: IBM and AIX do not mention it, but for openstack cloudmanagement
> (very specifically cloud-init) AIX needs a recognized stable Python
> implementation. I am "surprised" in the level of communication of IBM
> with Python community.
>
> Personally, I do not see AIX as a specialized platform. Feels more like
> the "last-standing" fully supported (commercial OEM) 'POSIX-UNIX'. Of
> course my focus is narrow - so maybe there is a lot of support for
> commercial platforms such as HPUX, Solaris, and other mainstream UNIXes.
> Feel free to correct me!!
>> Ditto for other specialized platforms.
>>
>>
>>
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181004/e08d3de7/attachment-0001.html>

From encukou at gmail.com  Thu Oct  4 03:45:41 2018
From: encukou at gmail.com (Petr Viktorin)
Date: Thu, 4 Oct 2018 09:45:41 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <5BB4B21D.2070302@UGent.be>
References: <5BB4B21D.2070302@UGent.be>
Message-ID: <9375c9a1-ed1b-697a-96b0-c74e5e66ff83@gmail.com>

On 10/3/18 2:12 PM, Jeroen Demeyer wrote:
> Hello,
> 
> I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, 
> titled "The C call protocol". He has co-authored several PEPs (PEP 394, 
> PEP 489, PEP 534, PEP 547, PEP 573), several of which involve extension 
> modules.
> 
> Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also 
> Antoine Pitrou, INADA Naoki and Nick Coghlan have approved Petr being 
> BDFL-Delegate.
> 
> I am well aware of the current governance issues, but several people 
> have mentioned that the BDFL-Delegate process can still continue for 
> now. I created a PR for the peps repository at 
> https://github.com/python/peps/pull/797
> 

Hello,
I don't think it's formally possible to do that now, but the following 
from elsewhere in the thread does make sense:

Antoine Pitrou:
> Consensus would obviously work (if no-one opposes the proposed person,
> then surely we don't need an elaborate governance model to decree that
> said person can become the PEP delegate, no?).

Yes, it would feel very silly to have consensus and not be able to act 
on it. But without a governance (and approval process) it's hard to 
*ensure* we have consensus where all relevant voices have been heard.

On the other hand, I don't agree with Nick Coghlan here:
> In this case, I'd consider it unlikely for either the PEP delegate appointment or any decisions about the PEP itself to be overturned - while it's a complex topic that definitely needs to go through the PEP process in order to work out the technical details, it isn't especially controversial in its own right (the most controversial aspect is whether it needs a new C level slot or not, and the PEP should clearly lay out the pros and cons of that)

PEP 580 *is* controversial -- there's the competing PEP 576 by Mark 
Shannon, who hasn't commented on this recently. Either would be an 
improvement, but choosing between them is a hard trade-off.
I'll leave technical stuff to another thread and concentrate on the 
process here.

When I'm happy with the PEP *and* if Mark says he's OK with it, I'll 
post a summary to Python-dev & Discourse with a special focus on the 
cons (e.g. the size increase of classes, which will affect everyone). If 
there are no "-1"s on the PEP itself or on the way it's discussed, let's 
treat PEP 580 as *provisionally* accepted, to be reverted if the new 
governance doesn't ratify it.

If there is no consensus, we'll need to wait for the new governance to 
decide.

From encukou at gmail.com  Thu Oct  4 03:55:10 2018
From: encukou at gmail.com (Petr Viktorin)
Date: Thu, 4 Oct 2018 09:55:10 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <CA+3bQGGTeuRBY1ibxissRV=7oLGocyeE3NRFCPRuNtuYa19A_A@mail.gmail.com>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
 <pp0o5i$e2a$1@blaine.gmane.org>
 <d1a07423-d09a-626d-3f49-4e382d83e802@felt.demon.nl>
 <CA+3bQGGTeuRBY1ibxissRV=7oLGocyeE3NRFCPRuNtuYa19A_A@mail.gmail.com>
Message-ID: <baaab3a3-02cf-cb6c-2f99-89e33d102899@gmail.com>

On 10/4/18 9:34 AM, Victor Stinner wrote:
> Hi,
> 
> If IBM wants a better Python support, it would help a lot if IBM pays 
> for this development. With money, you can easily find core dev 
> contractors. Antoine Pitrou has been paid in the past to enhance Python 
> support in Solaris and it worked well.

Michael explicitly said this is a personal effort. IBM or other big 
money is not involved.

Is paying the best way to get features into Python? Does becoming a core 
dev mean you can now get paid for approving changes? Some of the 
implications are quite disturbing :(


> Le?mercredi 3 octobre 2018, Michael Felt <aixtools at felt.demon.nl 
> <mailto:aixtools at felt.demon.nl>> a ?crit?:
>  >
>  >
>  > On 10/2/2018 11:34 PM, Terry Reedy wrote:
>  >> On 10/2/2018 12:41 PM, Simon Cross wrote:
>  >>> Are there any core devs that Michael or Erik could collaborate with?
>  >>> Rather than rely on adhoc patch review from random core developers.
>  >>
>  >> You two might collaborate with each other to the extent of reviewing
>  >> some of each other's PRs.
>  > Might be difficult. We both, or at least I, claim ignorance of the
>  > others platform. I still have a lot of PEP to learn, and my idea of a
>  > bug-fix (for Python2) was seen by core-dev as a feature change. I would
>  > not feel comfortable trying to mentor someone in things PEP, etc..
>  >> That still leaves the issue of merging.
>  > How much confidence is there in all the "CI" tests? Does that not offer
>  > sufficient confidence for a core-dev to press merge.
>  > How about "master" continuing to be what it is, but insert a new
>  > "pre-master" branch that the buildbots actually test on (e.g., what is
>  > now the 3.X) and have a 3.8 buildbot - for what is now the "master".
>  >
>  > PR would still be done based on master, but an "initial" merge would be
>  > via the pre-master aka 3.X buildbot tests.
>  >
>  > How "friendly" git is - that it not become such a workload to keep it
>  > clean - I cannot say. Still learning to use git. Better, but still do
>  > not want to assume it would be easy.
>  >
>  > My hope is that it would make it easier to consider a "merge" step that
>  > gets all the buildbots involved for even broader CI tests.
>  >
>  >>
>  >>> Michael and Eric: Question -- are you interested in becoming core
>  >>> developers at least for the purposes of maintaining these platforms in
>  >>> future?
>  >>
>  >> Since adhoc is not working to get merges, I had this same suggestion.
>  >> Michael and Erik, I presume you have gotten some guidelines on what
>  >> modifications to C code might be accepted, and what concerns people 
> have.
>  > imho: guidelines - paraphrased - as little as possible :)
>  >
>  > I have many assumptions, and one of those is that my assumptions are
>  > probably incorrect.
>  > Goal: have AIX recognized as a Stable platform, even if not in the
>  > highest supported category.
>  > And that implies, support as far as I am able, to keep it "Stable".
>  >>
>  >> I think for tests, a separate test_aix.py might be a good idea for
>  >> aix-only tests
>  > Unclear to me how this would work. Too young in Python I guess (or just
>  > a very old dog), but what test would be needed for AIX, or any other
>  > platform, that would not need to be tested in some fashion for the
>  > 'other' platforms. At a hunch, where there are many platform.system()
>  > dependencies expected (e.g., test_posix, maybe doing something in the
>  > class definition (is there a "Root" Object/Class that all inherit from.
>  > Maybe a (read-only) "root" attribute (or is property better?) could be
>  > the value of platform.system(), and iirc, might be used by as @property
>  > in unittest. (so, if not in "root" class, then in something like
>  > unittest/__init__.py.
>  >
>  > I hope to be "close" in "Python thinking" - enough that someone who
>  > actually knows how the pieces fit together could come with a better, and
>  > more appropriate guideline/implementation.
>  >
>  >> , while modification of other tests might be limited to adding skips.
>  >> The idea would be to make it easy to remove aix stuff in the future if
>  >> it again became unsupported.
>  > IMHO: IBM and AIX do not mention it, but for openstack cloudmanagement
>  > (very specifically cloud-init) AIX needs a recognized stable Python
>  > implementation. I am "surprised" in the level of communication of IBM
>  > with Python community.
>  >
>  > Personally, I do not see AIX as a specialized platform. Feels more like
>  > the "last-standing" fully supported (commercial OEM) 'POSIX-UNIX'. Of
>  > course my focus is narrow - so maybe there is a lot of support for
>  > commercial platforms such as HPUX, Solaris, and other mainstream UNIXes.
>  > Feel free to correct me!!
>  >> Ditto for other specialized platforms.
>  >>
>  >>
>  >>
>  >>
>  >
>  > _______________________________________________
>  > Python-Dev mailing list
>  > Python-Dev at python.org <mailto:Python-Dev at python.org>
>  > https://mail.python.org/mailman/listinfo/python-dev
>  > Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
>  >
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com
> 

From aixtools at felt.demon.nl  Thu Oct  4 04:11:29 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Thu, 4 Oct 2018 10:11:29 +0200
Subject: [Python-Dev] AIX to stable, what does that take?
Message-ID: <5cdbb644-4f1d-46a7-de0c-80beebb0822d@felt.demon.nl>

In the buildbots AIX is marked as "unstable"? What is needed to get it
marked as a "stable" platform - that being one of my short-term goals.

My assumption is that it needs to (at least) pass all tests - and that
is why I keep asking for attention. All the PRs to fix individual tests
mean less if they are not merged, for whatever reason.

However, maybe there is another way, or even something additional
needed. Maybe something I cannot provide and then I can adjust my
expectations and goals.

Regards,

Michael


From mike at selik.org  Thu Oct  4 04:14:07 2018
From: mike at selik.org (Michael Selik)
Date: Thu, 4 Oct 2018 01:14:07 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
Message-ID: <CAGgTfkMXK=FejiEWyjGM-nRmd0W2dt3oveFHVF7cZgufxeaMhQ@mail.gmail.com>

You don't like using Pool.starmap and itertools.repeat or a comprehension
that repeats an object?


On Wed, Oct 3, 2018, 6:30 PM Sean Harrington <seanharr11 at gmail.com> wrote:

> Hi guys -
>
> The solution to "lazily initialize" an expensive object in the worker
> process (i.e. via @lru_cache) is a great solution (that I must admit I did
> not think of). Additionally, in the second use case of "*passing a large
> object to each worker process*", I also agree with your suggestion to
> "shelter functions in a different module to avoid exposure to globals" as a
> good solution if one is wary of globals.
>
> That said, I still think "*passing a large object from parent process to
> worker processes*" should be easier when using Pool. Would either of you
> be open to something like the following?
>
>            def func(x, big_cache=None):
>                return big_cache[x]
>
>            big_cache =  { str(k): k for k in range(10000) }
>
>            ls = [ i for i in range(1000) ]
>
> with Pool(func_kwargs={"big_cache": big_cache}) as pool:
>
>     pool.map(func, ls)
>
>
> It's a much cleaner interface (which presumably requires a more difficult
> implementation) than my initial proposal. This also reads a lot better than
> the "initializer + global" recipe (clear flow of data), and is less
> constraining than the "define globals in parent" recipe. Most importantly,
> when taking sequential code and parallelizing via Pool.map, this does not
> force the user to re-implement "func" such that it consumes a global
> (rather than a kwarg). It allows "func" to be used elsewhere (i.e. in the
> parent process, from a different module, testing w/o globals, etc...)..
>
> This would essentially be an efficient implementation of Pool.starmap(),
> where kwargs are static, and passed to each application of "func" over our
> iterable.
>
> Thoughts?
>
>
> On Sat, Sep 29, 2018 at 3:00 PM Michael Selik <mike at selik.org> wrote:
>
>> On Sat, Sep 29, 2018 at 5:24 AM Sean Harrington <seanharr11 at gmail.com>
>> wrote:
>> >> On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington <seanharr11 at gmail.com>
>> wrote:
>> >> > My simple argument is that the developer should not be constrained
>> to make the objects passed globally available in the process, as this MAY
>> break encapsulation for large projects.
>> >>
>> >> I could imagine someone switching from Pool to ThreadPool and getting
>> >> into trouble, but in my mind using threads is caveat emptor. Are you
>> >> worried about breaking encapsulation in a different scenario?
>> >
>> > >> Without a specific example on-hand, you could imagine a tree of
>> function calls that occur in the worker process (even newly created
>> objects), that should not necessarily have access to objects passed from
>> parent -> worker. In every case given the current implementation, they will.
>>
>> Echoing Antoine: If you want some functions to not have access to a
>> module's globals, you can put those functions in a different module.
>> Note that multiprocessing already encapsulates each subprocesses'
>> globals in essentially a separate namespace.
>>
>> Without a specific example, this discussion is going to go around in
>> circles. You have a clear aversion to globals. Antoine and I do not.
>> No one else seems to have found this conversation interesting enough
>> to participate, yet.
>
>
> >>>
>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181004/4446352a/attachment.html>

From aixtools at felt.demon.nl  Thu Oct  4 04:29:19 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Thu, 4 Oct 2018 10:29:19 +0200
Subject: [Python-Dev] dear core-devs
In-Reply-To: <baaab3a3-02cf-cb6c-2f99-89e33d102899@gmail.com>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
 <pp0o5i$e2a$1@blaine.gmane.org>
 <d1a07423-d09a-626d-3f49-4e382d83e802@felt.demon.nl>
 <CA+3bQGGTeuRBY1ibxissRV=7oLGocyeE3NRFCPRuNtuYa19A_A@mail.gmail.com>
 <baaab3a3-02cf-cb6c-2f99-89e33d102899@gmail.com>
Message-ID: <01cbe584-6c04-982e-4d35-d20f8d48dfb9@felt.demon.nl>



On 10/4/2018 9:55 AM, Petr Viktorin wrote:
> On 10/4/18 9:34 AM, Victor Stinner wrote:
>> Hi,
>>
>> If IBM wants a better Python support, it would help a lot if IBM pays
>> for this development.
I agree. If IBM ...
>> ... Antoine Pitrou has been paid in the past to enhance Python
>> support in Solaris and it worked well.
>
FYI - as I now have access to the gccfarm, and in the spirit of more
generalized "posix" additions I looked for an HPUX and a Solais system
to build master on.

make test never finished (one test was still hanging after over 20
minutes, and I had to go. Of the 419, 17 or 18 had failed. Roughly where
AIX plus xlc was at last July without my PRs for tests.

So, while it worked - money stopped and Solaris is in no better
numerical shape (test wise) than AIX.
> Michael explicitly said this is a personal effort. IBM or other big
> money is not involved.
IBM is my employer. As I am not a developer (merely a systems and
management consultant) I do not face losing my job by working on OSS. I
have been called off certain OSS projects because IBM was providing
money and/or developers. This is one of the reasons (being called off
elsewhere) that I have been hesitant to be more involved than I was in
2015-2017.

So, let me be explicit - I can only speak for myself. And as long as no
manager says "No, cannot work on that" I have given a commitment to work
on this. "Some things cannot be bought" - such as un-biased (I call it
"maverick" rather than merely independent.) On the one hand IBM policy
is to encourage independent thought. The core goal is to help customers
succeed. But individual managers up and down the line occasionally have
additional business needs, and then workers as myself apologize and take
a step back - in a word - adjust.

Short answer: my involvement is mine to give at no price. I am
considered one of the worlds AIX experts on matters of integration,
performance and security.

So, I have just simple questions for you? Do you value my expertise? May
I assist?

>
> Is paying the best way to get features into Python? Does becoming a
> core dev mean you can now get paid for approving changes? Some of the
> implications are quite disturbing :(
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181004/4714c02d/attachment.html>

From songofacandy at gmail.com  Thu Oct  4 04:30:40 2018
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 4 Oct 2018 17:30:40 +0900
Subject: [Python-Dev] AIX to stable, what does that take?
In-Reply-To: <5cdbb644-4f1d-46a7-de0c-80beebb0822d@felt.demon.nl>
References: <5cdbb644-4f1d-46a7-de0c-80beebb0822d@felt.demon.nl>
Message-ID: <CAEfz+TxGQ8rz55FSOYpN+JVHqUarvP9gv8h6F9723XPwwaUGzQ@mail.gmail.com>

Hello,

First of all, congratulations on passing all test on AIX.

> My assumption is that it needs to (at least) pass all tests - and that
> is why I keep asking for attention. All the PRs to fix individual tests
> mean less if they are not merged, for whatever reason.
>
> However, maybe there is another way, or even something additional
> needed. Maybe something I cannot provide and then I can adjust my
> expectations and goals.

As a one of core developer, I don't know anything about AIX.
If my change breaks AIX build, I can't investigate what's happened.

So I think we need following in devguide:

* Brief description about AIX, from developer's point of view.
* How to run AIX on (VirtualBox, AWS EC2, Azure, GCP) easily.
* How to set up a development environment for Python.
* How to build Python.
* How to debug C code.

And even though there is a developer guide, it will take more long time
than fixing issues on AIX, compared Linux, macOS, and Windows.

But without this guide, it feels almost impossible to maintain AIX build to me.

Regards,

-- 
INADA Naoki  <songofacandy at gmail.com>

From steve at pearwood.info  Thu Oct  4 04:56:34 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 4 Oct 2018 18:56:34 +1000
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
Message-ID: <20181004085633.GP21220@ando.pearwood.info>

While keyword arguments have to be identifiers, using **kwargs allows 
arbitrary strings which aren't identifiers:

py> def spam(**kwargs):
...     print(kwargs)
...
py> spam(**{"something arbitrary": 1, '\n': 2})
{'something arbitrary': 1, '\n': 2}


There is some discussion on Python-Ideas on whether or not that 
behaviour ought to be considered a language feature, an accident of 
implementation, or a bug.

Can we get some guidence on this please?


Thanks,


-- 
Steve

From aixtools at felt.demon.nl  Thu Oct  4 05:13:51 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Thu, 4 Oct 2018 11:13:51 +0200
Subject: [Python-Dev] AIX to stable, what does that take?
In-Reply-To: <CAEfz+TxGQ8rz55FSOYpN+JVHqUarvP9gv8h6F9723XPwwaUGzQ@mail.gmail.com>
References: <5cdbb644-4f1d-46a7-de0c-80beebb0822d@felt.demon.nl>
 <CAEfz+TxGQ8rz55FSOYpN+JVHqUarvP9gv8h6F9723XPwwaUGzQ@mail.gmail.com>
Message-ID: <cd367b92-14a2-11b3-baa6-db9181407cbc@felt.demon.nl>



On 10/4/2018 10:30 AM, INADA Naoki wrote:
> Hello,
>
> First of all, congratulations on passing all test on AIX.
>
>> My assumption is that it needs to (at least) pass all tests - and that
>> is why I keep asking for attention. All the PRs to fix individual tests
>> mean less if they are not merged, for whatever reason.
>>
>> However, maybe there is another way, or even something additional
>> needed. Maybe something I cannot provide and then I can adjust my
>> expectations and goals.
> As a one of core developer, I don't know anything about AIX.
> If my change breaks AIX build, I can't investigate what's happened.
>
> So I think we need following in devguide:
>
> * Brief description about AIX, from developer's point of view.
This I might be able to do. Bullet form:
* Committed to POSIX standard (valid when release came out, so AIX 5.3
confirms to a different standard than AIX 7.2)
* While Linux affinity is recognized - GNU (or GNP - GNU not POSIX)
integration is not guaranteed. - GNU rte is not provided under support.
There is a so-called Toolbox, GNU an other OSS utilities supplied by
many packaged as RPMs. Unfortunately, different RPM providers (Michael
Perlz, BULL Freeware, IBM, and others) have different numbering (the
part after the package version, e.g., python-2.7.10-XXX so they do not
mix well). Headache for both admins and developers trying to develop in
a GNU-like environment.
* As a consultant, fedup with what is called the "RPM hell" by many AIX
admins - I do not use any RPMs. I build everything myself, using xlc
(gcc introduces the need for a GNU RTE, e.g., glibc). I package using
installp (the "native") AIX package manager, and strive to make the
packages independent (these days). When there are dependencies I try to
build them as static libraries so that they do not become an additional
install dependency.
* finally, a bit deeper: while the AIX linker loader supports svr4
shared libraries (it is the data, not the file name) it also supports
having multiple shared libraries in a classic archive. So, rather that
.../lib/libxxx.so and .../lib64/libxxx.so AIX prefers .../lib/libxxx.a
with two so-called members, with same or different names. The one found
is not it's name, but the symbol name and size of the ABI (32-bit or 64-bit)
* Hope that is enough of the key issues for now.
** In general, GNU autotools (autoreconf -f -v) works well, as does
configure.ac et al. for createing OSS Makefiles.
> * How to run AIX on (VirtualBox, AWS EC2, Azure, GCP) easily.
Easily! ? :) - well, on a POWER server it was easy enough for me to
follow the step by step instructions for creating a buildbot. If I had a
POWER server with more resources I would examine using the AIX internal
WPAR - however, a POWER server configured to use PowerVC uses "EC2" aka
cloud-init for creating a virtual machine. With that environment it
should be "easy" to provide additional instructions to cloud-init-ec2.

Or, I provide you a login on my personal server that I run the buildbot
on. etc. etc. - Where there is a will, there is a way.
> * How to set up a development environment for Python.
Again, follow the instructions for setting up a buildbot.
> * How to build Python.
git clone ...
autoreconf -v -f (as needed)
./configure --with-pydebug? #gcc compiler
./configure --with-pydebug --without-computed-gotos # xlc compiler
make
make test
> * How to debug C code.
I learned, 40 years ago, using adb (a debugger) - I do a lot of
single-stepping. gdb is not the default debugger. If I were a developer
I would probably dig into the AIX debuggers (there are at least two, kdb
(kernel debugger, which I do use occaisionally for performance issues)
and another I try to avoid. I add fprintf statements and am looking at
learning how to use probevue.

In short, you probably have many much better ideas on how to debug C
than I do :)
>
> And even though there is a developer guide, it will take more long time
> than fixing issues on AIX, compared Linux, macOS, and Windows.
>
> But without this guide, it feels almost impossible to maintain AIX build to me.
IMHO: The AIX build is stable, but this is unrecognized because it does
have differences that cause tests to fail. I can think of one test that
PASSes, but should fail. And another test that passes, but should have
failed (in test_uuid) I have submitted a PR.

I tried to fix "all" in one PR, which confused people - so redid it as
two (got _uuid working in Python 3.7 ! yes!!) but the "original" to fix
uuid.py and test_uuid.py is still "awaiting change review".

My gut feeling to maintaining AIX is: a) all test pass so a potential
regression is flagged; b) someone such as myself who knows the platform
and can establish a "root cause" on why it is failing with AIX so that
c) a developer becomes aware and can decide to ignore or adjust; d)
alternatives - such as work around an implementation limitation (as I
try to do, e.g., for test_time and the related _datetime.c) is yet
another path.

In other words - it needs to be a shared responsibility - some
volunteers with a passion for platform stability (in this specific case
AIX and me as "passionate-person" and perhaps someone as youself who
wants to focus on the language itself - ideally without deep (or any!)
concern for platform differences.
>
> Regards,
My 6 bits!


From storchaka at gmail.com  Thu Oct  4 05:18:43 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 4 Oct 2018 12:18:43 +0300
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <20181004085633.GP21220@ando.pearwood.info>
References: <20181004085633.GP21220@ando.pearwood.info>
Message-ID: <pp4lpg$8us$1@blaine.gmane.org>

04.10.18 11:56, Steven D'Aprano ????:
> While keyword arguments have to be identifiers, using **kwargs allows
> arbitrary strings which aren't identifiers:
> 
> py> def spam(**kwargs):
> ....     print(kwargs)
> ....
> py> spam(**{"something arbitrary": 1, '\n': 2})
> {'something arbitrary': 1, '\n': 2}
> 
> 
> There is some discussion on Python-Ideas on whether or not that
> behaviour ought to be considered a language feature, an accident of
> implementation, or a bug.
> 
> Can we get some guidence on this please?

This is an implementation detail. Currently CPython doesn't ensure that 
keyword argument names are identifiers for performance reasons. But this 
can be changed in future versions or in other implementations.


From seanharr11 at gmail.com  Thu Oct  4 05:55:29 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Thu, 4 Oct 2018 05:55:29 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAGgTfkMXK=FejiEWyjGM-nRmd0W2dt3oveFHVF7cZgufxeaMhQ@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAGgTfkMXK=FejiEWyjGM-nRmd0W2dt3oveFHVF7cZgufxeaMhQ@mail.gmail.com>
Message-ID: <CAFbyWc2-=+P7yPzF07Re3Omdpbzgbh8TggBRzzum_o3ELmwvCg@mail.gmail.com>

Starmap will serialize/deserialize the ?big object? once for each task
created, so this is not performant. The goal is to pay the ?one time cost?
of serialization of the ?big object?, and still pass this object to func at
each iteration.
On Thu, Oct 4, 2018 at 4:14 AM Michael Selik <mike at selik.org> wrote:

> You don't like using Pool.starmap and itertools.repeat or a comprehension
> that repeats an object?
>
>
>
> On Wed, Oct 3, 2018, 6:30 PM Sean Harrington <seanharr11 at gmail.com> wrote:
>
>> Hi guys -
>>
>> The solution to "lazily initialize" an expensive object in the worker
>> process (i.e. via @lru_cache) is a great solution (that I must admit I did
>> not think of). Additionally, in the second use case of "*passing a large
>> object to each worker process*", I also agree with your suggestion to
>> "shelter functions in a different module to avoid exposure to globals" as a
>> good solution if one is wary of globals.
>>
>> That said, I still think "*passing a large object from parent process to
>> worker processes*" should be easier when using Pool. Would either of you
>> be open to something like the following?
>>
>>            def func(x, big_cache=None):
>>                return big_cache[x]
>>
>>            big_cache =  { str(k): k for k in range(10000) }
>>
>>            ls = [ i for i in range(1000) ]
>>
>> with Pool(func_kwargs={"big_cache": big_cache}) as pool:
>>
>>     pool.map(func, ls)
>>
>>
>> It's a much cleaner interface (which presumably requires a more difficult
>> implementation) than my initial proposal. This also reads a lot better than
>> the "initializer + global" recipe (clear flow of data), and is less
>> constraining than the "define globals in parent" recipe. Most importantly,
>> when taking sequential code and parallelizing via Pool.map, this does not
>> force the user to re-implement "func" such that it consumes a global
>> (rather than a kwarg). It allows "func" to be used elsewhere (i.e. in the
>> parent process, from a different module, testing w/o globals, etc...)..
>>
>> This would essentially be an efficient implementation of Pool.starmap(),
>> where kwargs are static, and passed to each application of "func" over our
>> iterable.
>>
>> Thoughts?
>>
>>
>> On Sat, Sep 29, 2018 at 3:00 PM Michael Selik <mike at selik.org> wrote:
>>
>>> On Sat, Sep 29, 2018 at 5:24 AM Sean Harrington <seanharr11 at gmail.com>
>>> wrote:
>>> >> On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington <seanharr11 at gmail.com>
>>> wrote:
>>> >> > My simple argument is that the developer should not be constrained
>>> to make the objects passed globally available in the process, as this MAY
>>> break encapsulation for large projects.
>>> >>
>>> >> I could imagine someone switching from Pool to ThreadPool and getting
>>> >> into trouble, but in my mind using threads is caveat emptor. Are you
>>> >> worried about breaking encapsulation in a different scenario?
>>> >
>>> > >> Without a specific example on-hand, you could imagine a tree of
>>> function calls that occur in the worker process (even newly created
>>> objects), that should not necessarily have access to objects passed from
>>> parent -> worker. In every case given the current implementation, they will.
>>>
>>> Echoing Antoine: If you want some functions to not have access to a
>>> module's globals, you can put those functions in a different module.
>>> Note that multiprocessing already encapsulates each subprocesses'
>>> globals in essentially a separate namespace.
>>>
>>> Without a specific example, this discussion is going to go around in
>>> circles. You have a clear aversion to globals. Antoine and I do not.
>>> No one else seems to have found this conversation interesting enough
>>> to participate, yet.
>>
>>
>> >>>
>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181004/59f10922/attachment.html>

From njs at pobox.com  Thu Oct  4 06:15:31 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 4 Oct 2018 03:15:31 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
Message-ID: <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>

On Wed, Oct 3, 2018 at 6:30 PM, Sean Harrington <seanharr11 at gmail.com> wrote:
> with Pool(func_kwargs={"big_cache": big_cache}) as pool:
>     pool.map(func, ls)

I feel like it would be nicer to spell this:

with Pool() as pool:
    pool.map(functools.partial(func, big_cache=big_cache), ls)

And this might also solve your problem, if pool.map is clever enough
to only send the function object once to each worker?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From steve at pearwood.info  Thu Oct  4 06:25:40 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 4 Oct 2018 20:25:40 +1000
Subject: [Python-Dev] dear core-devs
In-Reply-To: <baaab3a3-02cf-cb6c-2f99-89e33d102899@gmail.com>
References: <4af231bd-6a84-5d2b-a832-4f1959e6bb8d@felt.demon.nl>
 <e78c2c90-5c08-2b3d-f10c-7c240a806884@palladion.com>
 <CAOTD34agkZ7DnOTYGjyyRUB8fuUt=ErOFhovDNhVwzGJTr1a6Q@mail.gmail.com>
 <CAD5NRCH-n8+c+t+mD=gkUF1sJYgdGcZaN1ccuA9fjH8eMzifoQ@mail.gmail.com>
 <pp0o5i$e2a$1@blaine.gmane.org>
 <d1a07423-d09a-626d-3f49-4e382d83e802@felt.demon.nl>
 <CA+3bQGGTeuRBY1ibxissRV=7oLGocyeE3NRFCPRuNtuYa19A_A@mail.gmail.com>
 <baaab3a3-02cf-cb6c-2f99-89e33d102899@gmail.com>
Message-ID: <20181004102540.GS21220@ando.pearwood.info>

On Thu, Oct 04, 2018 at 09:55:10AM +0200, Petr Viktorin wrote:

> Is paying the best way to get features into Python?

No, but it is *a* way to get features into Python.

It may be a good way to get a feature that is important to (generic) 
you, but the core devs don't have enough time, interest or energy to add 
in their own free time.

It is not a way to get features that are opposed by the core devs.


> Does becoming a core 
> dev mean you can now get paid for approving changes? Some of the 
> implications are quite disturbing :(

Naturally whenever money is involved, there is the risks of a conflict 
of interest, and of course we should be careful about such risks. But 
they are small and manageable risks.

We're unlikely to be talking about huge amounts of money, enough to 
corrupt people, and so long as there is transparency about who does what 
and why, open source and free software is compatible with payment for 
work done. Even Richard Stallman accepts money for code :-)


-- 
Steve

From lukasz at langa.pl  Thu Oct  4 07:38:12 2018
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Thu, 4 Oct 2018 13:38:12 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <pp3b3o$tl2$1@blaine.gmane.org>
References: <5BB4B21D.2070302@UGent.be> <pp3b3o$tl2$1@blaine.gmane.org>
Message-ID: <DE5EEA5F-9FDD-4AEF-BD39-01AFDFC9C4E6@langa.pl>


> On 3 Oct 2018, at 23:10, Terry Reedy <tjreedy at udel.edu> wrote:
> 
> On 10/3/2018 8:12 AM, Jeroen Demeyer wrote:
>> Hello,
>> I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, titled "The C call protocol". He has co-authored several PEPs (PEP 394, PEP 489, PEP 534, PEP 547, PEP 573), several of which involve extension modules.
>> Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also Antoine Pitrou, INADA Naoki and Nick Coghlan have approved Petr being BDFL-Delegate.
> 
> To me, three experienced core devs approving of a 4th person as PEP-examiner is sufficient to proceed on a CPython implementation proposal.  I don't think we need to be paralyzed on this.

What you're saying is sensible, the team is small enough and tightly knit that we trust each other. However, trust is not the point. It's about clear expectations and avoiding anarchy. As Nick points out elsewhere, circumventing the lack of governance by "asking a few friends" on the core team creates a need for the new leadership to ratify those changes. Speaking frankly, it would be a major shit show if any of those changes were to be reverted. As the release manager of this version of Python, can I ask you please not to risk this?

Ironically, the governance model I am championing is one that would closely resemble what you're describing. A community of experts, no kings: https://discuss.python.org/t/pep-8012-the-community-model/156/

So it's really not that I disagree with you, I do. It's not that I don't trust Petr, I do. It's that I believe the core team needs to formalize how they want the project to proceed before they go run approving PEPs.


> And indeed, when it comes to sub-PEP C-API changes, we seem not to be.

Any sub-PEP changes are fair game at the moment.


> This change, if made, should be early in the cycle for the next version, rather than landing just before the first beta.

There is little to be gained in landing "early in the cycle" when alpha releases are not widely available anywhere and there is a 6-month long beta period. More importantly, using "we need to land fast" as an argument to rush things in is self-defeating.


> A language syntax-change proposal would be something else.

Anything that is big enough to require a PEP is by definition a substantial change and controversial that way. As somebody else here points out, there is even a competing PEP. That sounds controversial to me.

- ?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181004/143997ab/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181004/143997ab/attachment.sig>

From lukasz at langa.pl  Thu Oct  4 07:39:57 2018
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Thu, 4 Oct 2018 13:39:57 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <CADiSq7fX=2bLU4noJAiwS4J3bKwcL2AztsQJQvswTa1Pid7ddg@mail.gmail.com>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <20181003200823.034802d9@fsol>
 <CADiSq7fX=2bLU4noJAiwS4J3bKwcL2AztsQJQvswTa1Pid7ddg@mail.gmail.com>
Message-ID: <35E5F735-5AE6-47BE-9977-AD3F83531575@langa.pl>


> On 4 Oct 2018, at 08:14, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> I was figuring we could treat it as a caretaker mode governance: anything we do before the new governance model is formalised is subject to ratification by the new council members (or the new BDFL if that option ends up being chosen).

Unless we end up with neither a BDFL nor a council... *cough* PEP 8012 *cough* ;-)

- ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181004/7cc9dd22/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181004/7cc9dd22/attachment.sig>

From encukou at gmail.com  Thu Oct  4 08:00:19 2018
From: encukou at gmail.com (Petr Viktorin)
Date: Thu, 4 Oct 2018 14:00:19 +0200
Subject: [Python-Dev] Petr Viktorin as BDFL-Delegate for PEP 580
In-Reply-To: <DE5EEA5F-9FDD-4AEF-BD39-01AFDFC9C4E6@langa.pl>
References: <5BB4B21D.2070302@UGent.be> <pp3b3o$tl2$1@blaine.gmane.org>
 <DE5EEA5F-9FDD-4AEF-BD39-01AFDFC9C4E6@langa.pl>
Message-ID: <a90ba50b-4bf8-8111-b40e-b9a1bda620d2@gmail.com>

On 10/4/18 1:38 PM, ?ukasz Langa wrote:
> 
>> On 3 Oct 2018, at 23:10, Terry Reedy <tjreedy at udel.edu 
>> <mailto:tjreedy at udel.edu>> wrote:
>>
>> On 10/3/2018 8:12 AM, Jeroen Demeyer wrote:
>>> Hello,
>>> I would like to propose Petr Viktorin as BDFL-Delegate for PEP 580, 
>>> titled "The C call protocol". He has co-authored several PEPs (PEP 
>>> 394, PEP 489, PEP 534, PEP 547, PEP 573), several of which involve 
>>> extension modules.
>>> Petr has agreed to become BDFL-Delegate for PEP 580 if asked. Also 
>>> Antoine Pitrou, INADA Naoki and Nick Coghlan have approved Petr being 
>>> BDFL-Delegate.
>>
>> To me, three experienced core devs approving of a 4th person as 
>> PEP-examiner is sufficient to proceed on a CPython implementation 
>> proposal. ?I don't think we need to be paralyzed on this.
> 
> What you're saying is sensible, the team is small enough and tightly 
> knit that we trust each other. However, trust is not the point. It's 
> about clear expectations and avoiding anarchy. As Nick points out 
> elsewhere, circumventing the lack of governance by "asking a few 
> friends" on the core team creates a need for the new leadership to 
> ratify those changes. Speaking frankly, it would be a major shit show if 
> any of those changes were to be reverted. As the release manager of this 
> version of Python, can I ask you please not to risk this?
> 
> Ironically, the governance model I am championing is one that would 
> closely resemble what you're describing. A community of experts, no 
> kings: https://discuss.python.org/t/pep-8012-the-community-model/156/
> 
> So it's really not that I disagree with you, I do. It's not that I don't 
> trust Petr, I do. It's that I believe the core team needs to formalize 
> how they want the project to proceed *before* they go run approving PEPs.

?ukasz, as the release manager for 3.8 you're the closest we have to an 
authority, so I defer to your judgment. No PEPs can currently be accepted.


Anyway, even if I was a *-delegate here, I would need to hear Mark 
Shannon's opinion on the PEP. Convincing him will probably be harder 
than convincing me.

From michael.haubenwallner at ssi-schaefer.com  Fri Oct  5 10:15:29 2018
From: michael.haubenwallner at ssi-schaefer.com (Michael Haubenwallner)
Date: Fri, 5 Oct 2018 16:15:29 +0200
Subject: [Python-Dev] AIX to stable, what does that take?
In-Reply-To: <cd367b92-14a2-11b3-baa6-db9181407cbc@felt.demon.nl>
References: <5cdbb644-4f1d-46a7-de0c-80beebb0822d@felt.demon.nl>
 <CAEfz+TxGQ8rz55FSOYpN+JVHqUarvP9gv8h6F9723XPwwaUGzQ@mail.gmail.com>
 <cd367b92-14a2-11b3-baa6-db9181407cbc@felt.demon.nl>
Message-ID: <bd343118-a8d3-0a56-518b-48220157c760@ssi-schaefer.com>

Hi Michael,

being on a similar road with Gentoo Prefix, I really do appreciate
your AIX related work!

However, for two (not so minor) topics I've got a little different
experience, which I think should be mentioned here for completion:

On 10/04/2018 11:13 AM, Michael Felt wrote:
> On 10/4/2018 10:30 AM, INADA Naoki wrote:
>> Hello,
>>
>> First of all, congratulations on passing all test on AIX.

>> As a one of core developer, I don't know anything about AIX.
>> If my change breaks AIX build, I can't investigate what's happened.
>>
>> So I think we need following in devguide:
>>
>> * Brief description about AIX, from developer's point of view.
> This I might be able to do. Bullet form:

> ... I build everything myself, using xlc
> (gcc introduces the need for a GNU RTE, e.g., glibc).

Using gcc does *not* require to use glibc or even GNU binutils at all.
Except for gcc's own runtime libraries, there's no need for a GNU RTE.
In fact, in Gentoo Prefix I do use gcc as the compiler, configured to
use AIX provided binutils (as, ld, nm, ...), with AIX libc as RTE.

> * finally, a bit deeper: while the AIX linker loader supports svr4
> shared libraries (it is the data, not the file name) it also supports
> having multiple shared libraries in a classic archive. So, rather that
> .../lib/libxxx.so and .../lib64/libxxx.so AIX prefers .../lib/libxxx.a
> with two so-called members, with same or different names. The one found
> is not it's name, but the symbol name and size of the ABI (32-bit or 64-bit)

While this all is true, having multiple *versions* of one shared library in
one single file is a PITA for package managers - both human or software.

But fortunately, the AIX linker does support so called "Import Files",
allowing for *filename based* shared library versioning like on Linux,
while still allowing for both ABIs in a single library archive file.

For example, libtool provides the --with-aix-soname={aix|svr4|both}
configure flag since libtool-2.4.4.  Although the default will stay
at 'aix' here, in Gentoo Prefix I do use 'svr4' only.  This actually
is a package manager's decision, ideally for all depending packages.
As gcc does use libtool, for more information please refer to
https://gcc.gnu.org/install/configure.html#WithAixSoname
But note that "Import Files" should work with xlc as well.

Thanks!
/haubi/

From status at bugs.python.org  Fri Oct  5 12:10:07 2018
From: status at bugs.python.org (Python tracker)
Date: Fri,  5 Oct 2018 18:10:07 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20181005161007.EC73B11683A@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2018-09-28 - 2018-10-05)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6813 (+32)
  closed 39845 (+42)
  total  46658 (+74)

Open issues with patches: 2722 


Issues opened (53)
==================

#18291: codecs.open interprets FS, RS, GS as line ends
https://bugs.python.org/issue18291  reopened by serhiy.storchaka

#31310: semaphore tracker isn't protected against crashes
https://bugs.python.org/issue31310  reopened by serhiy.storchaka

#32117: Tuple unpacking in return and yield statements
https://bugs.python.org/issue32117  reopened by serhiy.storchaka

#34811: test_gdb fails with latest gdb
https://bugs.python.org/issue34811  reopened by vstinner

#34834: test_ssl.test_options does not correctly account for built-in 
https://bugs.python.org/issue34834  opened by xnox

#34836: test_ssl.test_default_ecdh_curve needs no tls1.3 flag in 2.7, 
https://bugs.python.org/issue34836  opened by xnox

#34837: Multiprocessing.pool API Extension - Pass Data to Workers w/o 
https://bugs.python.org/issue34837  opened by seanharr11

#34838: Improve arg clinic code generation for cases with type checkin
https://bugs.python.org/issue34838  opened by rhettinger

#34839: doctest: Change example under warnings section
https://bugs.python.org/issue34839  opened by cheryl.sabella

#34840: dlopen() error with no error message from dlerror()
https://bugs.python.org/issue34840  opened by shuoz

#34841: Script???s directory not in sys.path with embeddable Windows d
https://bugs.python.org/issue34841  opened by ssapin

#34844: logging.Formatter enhancement - Checking on style and fmt fiel
https://bugs.python.org/issue34844  opened by BNMetrics

#34846: Runtime failure with Failed to import site module
https://bugs.python.org/issue34846  opened by sat.tho at gmail.com

#34847: asyncio: Add PHA for TLS 1.3
https://bugs.python.org/issue34847  opened by fantix

#34848: range.index only takes one argument when it's documented as ta
https://bugs.python.org/issue34848  opened by bup

#34849: Drop logging when asyncio waits in selector.select()
https://bugs.python.org/issue34849  opened by asvetlov

#34850: Emit a syntax warning for "is" with a literal
https://bugs.python.org/issue34850  opened by serhiy.storchaka

#34852: Counter-intuitive behavior of Server.close() / wait_closed()
https://bugs.python.org/issue34852  opened by aymeric.augustin

#34855: batch file variables
https://bugs.python.org/issue34855  opened by lindblad

#34856: Make the repr of lambda containing the signature and body expr
https://bugs.python.org/issue34856  opened by serhiy.storchaka

#34857: IDLE: SyntaxWarning not handled properly
https://bugs.python.org/issue34857  opened by terry.reedy

#34858: MappingProxy objects should JSON serialize just like a diction
https://bugs.python.org/issue34858  opened by Michael Smith2

#34861: Improve cProfile standard output
https://bugs.python.org/issue34861  opened by Anders.Hovm??ller

#34864: In Idle, Mac tabs make editor status line disappear.
https://bugs.python.org/issue34864  opened by andyharrington

#34866: CGI DOS vulnerability via long post list
https://bugs.python.org/issue34866  opened by Matthew Belisle

#34867: Add mode to disable small integer and interned string caches
https://bugs.python.org/issue34867  opened by steven.daprano

#34870: Core dump when Python VSCode debugger is attached
https://bugs.python.org/issue34870  opened by Per Lundberg

#34872: investigate task/future cancellation in asynciomodule.c
https://bugs.python.org/issue34872  opened by yselivanov

#34876: Python3.8 changes how decorators are traced
https://bugs.python.org/issue34876  opened by nedbat

#34877: Inconsistent Behavior Of futures.ProcessPoolExecutor
https://bugs.python.org/issue34877  opened by TensorTom

#34880: About the "assert" bytecode
https://bugs.python.org/issue34880  opened by vtheno athena

#34881: unnecessary encoded-words usage breaks DKIM signatures
https://bugs.python.org/issue34881  opened by bryced

#34882: f(a=1, *args) should be a SyntaxError
https://bugs.python.org/issue34882  opened by metaxm

#34883: test_lzma: Multiple test failures when liblzma is built withou
https://bugs.python.org/issue34883  opened by mgorny

#34884: Python loads incorrect libraries
https://bugs.python.org/issue34884  opened by Tim Hutt

#34885: asyncio documention has lost its paragraph about cancellation
https://bugs.python.org/issue34885  opened by abki

#34886: subprocess.run throws exception when input and stdin are passe
https://bugs.python.org/issue34886  opened by aecant

#34888: Python3.8 optimizes away a "while" line
https://bugs.python.org/issue34888  opened by nedbat

#34890: Support functools.partial in inspect.is*function() checks
https://bugs.python.org/issue34890  opened by asvetlov

#34891: Multi-processing example inaccurate warning
https://bugs.python.org/issue34891  opened by anthony-flury

#34893: Add 2to3 fixer to change send and recv methods of socket objec
https://bugs.python.org/issue34893  opened by devarakondapranav

#34895: Mark optional stdlib modules in documentation
https://bugs.python.org/issue34895  opened by of4tvziy

#34897: distutils test errors when CXX is not set
https://bugs.python.org/issue34897  opened by Michael.Felt

#34898: add mtime argument to gzip.compress
https://bugs.python.org/issue34898  opened by guoci

#34899: Possible assertion failure due to int_from_bytes_impl()
https://bugs.python.org/issue34899  opened by ZackerySpytz

#34900: unittest subTests() fails when called from debug()
https://bugs.python.org/issue34900  opened by Bruno Oliveira

#34901: Missing isolated (-I) flag in sys.flags table
https://bugs.python.org/issue34901  opened by danishprakash

#34902: Azure pipelines PR build fails with "Unexpected vmImage 'vs201
https://bugs.python.org/issue34902  opened by xtreak

#34903: strptime %d handling of single digit day of month
https://bugs.python.org/issue34903  opened by Mike Gleen

#34904: Crash in ZipFile.close() when writing zip file to /dev/null
https://bugs.python.org/issue34904  opened by erik.bray

#34905: Cannot assign memoryview values from array.array
https://bugs.python.org/issue34905  opened by aparamon

#34906: Fix typo in the documentation
https://bugs.python.org/issue34906  opened by matrixise

#34907: calculation not working properly
https://bugs.python.org/issue34907  opened by hwk_un1te



Most recent 15 issues with no replies (15)
==========================================

#34907: calculation not working properly
https://bugs.python.org/issue34907

#34905: Cannot assign memoryview values from array.array
https://bugs.python.org/issue34905

#34902: Azure pipelines PR build fails with "Unexpected vmImage 'vs201
https://bugs.python.org/issue34902

#34899: Possible assertion failure due to int_from_bytes_impl()
https://bugs.python.org/issue34899

#34898: add mtime argument to gzip.compress
https://bugs.python.org/issue34898

#34897: distutils test errors when CXX is not set
https://bugs.python.org/issue34897

#34895: Mark optional stdlib modules in documentation
https://bugs.python.org/issue34895

#34884: Python loads incorrect libraries
https://bugs.python.org/issue34884

#34883: test_lzma: Multiple test failures when liblzma is built withou
https://bugs.python.org/issue34883

#34877: Inconsistent Behavior Of futures.ProcessPoolExecutor
https://bugs.python.org/issue34877

#34870: Core dump when Python VSCode debugger is attached
https://bugs.python.org/issue34870

#34866: CGI DOS vulnerability via long post list
https://bugs.python.org/issue34866

#34861: Improve cProfile standard output
https://bugs.python.org/issue34861

#34857: IDLE: SyntaxWarning not handled properly
https://bugs.python.org/issue34857

#34855: batch file variables
https://bugs.python.org/issue34855



Most recent 15 issues waiting for review (15)
=============================================

#34906: Fix typo in the documentation
https://bugs.python.org/issue34906

#34901: Missing isolated (-I) flag in sys.flags table
https://bugs.python.org/issue34901

#34900: unittest subTests() fails when called from debug()
https://bugs.python.org/issue34900

#34899: Possible assertion failure due to int_from_bytes_impl()
https://bugs.python.org/issue34899

#34898: add mtime argument to gzip.compress
https://bugs.python.org/issue34898

#34897: distutils test errors when CXX is not set
https://bugs.python.org/issue34897

#34872: investigate task/future cancellation in asynciomodule.c
https://bugs.python.org/issue34872

#34866: CGI DOS vulnerability via long post list
https://bugs.python.org/issue34866

#34861: Improve cProfile standard output
https://bugs.python.org/issue34861

#34856: Make the repr of lambda containing the signature and body expr
https://bugs.python.org/issue34856

#34850: Emit a syntax warning for "is" with a literal
https://bugs.python.org/issue34850

#34849: Drop logging when asyncio waits in selector.select()
https://bugs.python.org/issue34849

#34847: asyncio: Add PHA for TLS 1.3
https://bugs.python.org/issue34847

#34844: logging.Formatter enhancement - Checking on style and fmt fiel
https://bugs.python.org/issue34844

#34838: Improve arg clinic code generation for cases with type checkin
https://bugs.python.org/issue34838



Top 10 most discussed issues (10)
=================================

#34751: Hash collisions for tuples
https://bugs.python.org/issue34751  39 msgs

#34850: Emit a syntax warning for "is" with a literal
https://bugs.python.org/issue34850  21 msgs

#18291: codecs.open interprets FS, RS, GS as line ends
https://bugs.python.org/issue18291  11 msgs

#34711: Lib/http/server.py: Return HTTPStatus.NOT_FOUND if path.endswi
https://bugs.python.org/issue34711  10 msgs

#34880: About the "assert" bytecode
https://bugs.python.org/issue34880  10 msgs

#34313: IDLE crashes with Tk-related error on macOS with ActiveTcl 8.6
https://bugs.python.org/issue34313   9 msgs

#34872: investigate task/future cancellation in asynciomodule.c
https://bugs.python.org/issue34872   9 msgs

#34812: [EASY] support.args_from_interpreter_flags() doesn't inherit -
https://bugs.python.org/issue34812   8 msgs

#34867: Add mode to disable small integer and interned string caches
https://bugs.python.org/issue34867   8 msgs

#34014: loop.run_in_executor should propagate current contextvars
https://bugs.python.org/issue34014   7 msgs



Issues closed (42)
==================

#25300: Enable Intel MPX (Memory protection Extensions) feature
https://bugs.python.org/issue25300  closed by benjamin.peterson

#26005: Denial of Service in SimpleHTTPServer and BaseHTTPServer
https://bugs.python.org/issue26005  closed by martin.panter

#28441: Change sys.executable to include executable suffix
https://bugs.python.org/issue28441  closed by inada.naoki

#29551: unittest: TestSuite.debug() does not like subTest()
https://bugs.python.org/issue29551  closed by berker.peksag

#30156: PYTHONDUMPREFS segfaults on exit
https://bugs.python.org/issue30156  closed by vstinner

#31660: sys.executable different in os.execv'd python3.6 virtualenv se
https://bugs.python.org/issue31660  closed by cheryl.sabella

#31865: html.unescape does not work as per documentation
https://bugs.python.org/issue31865  closed by ezio.melotti

#32756: argparse: parse_known_args: raising exception on unknown arg f
https://bugs.python.org/issue32756  closed by paul.j3

#32833: argparse doesn't recognise two option aliases as equal
https://bugs.python.org/issue32833  closed by paul.j3

#33117: asyncio example uses non-existing/documented method
https://bugs.python.org/issue33117  closed by asvetlov

#34172: multiprocessing.Pool and ThreadPool leak resources after being
https://bugs.python.org/issue34172  closed by pitrou

#34476: asyncio.sleep(0) not documented
https://bugs.python.org/issue34476  closed by asvetlov

#34739: Get rid of tp_getattro in xml.etree.ElementTree.XMLParser
https://bugs.python.org/issue34739  closed by serhiy.storchaka

#34740: Get rid of tp_getattro in ossaudiodev.oss_audio_device
https://bugs.python.org/issue34740  closed by serhiy.storchaka

#34779: IDLE internals show up in tracebacks when returning objects th
https://bugs.python.org/issue34779  closed by terry.reedy

#34797: Convert heapq to the argument clinic
https://bugs.python.org/issue34797  closed by rhettinger

#34801: codecs.getreader() splits lines containing control characters
https://bugs.python.org/issue34801  closed by serhiy.storchaka

#34835: Multiprocessing module update fails with pip3
https://bugs.python.org/issue34835  closed by steven.daprano

#34842: Incorrect error messages in bisect
https://bugs.python.org/issue34842  closed by rhettinger

#34843: logging cookbook docs: remove 'recent' when referring to multi
https://bugs.python.org/issue34843  closed by rhettinger

#34845: allow exprlist as the iterators of comprehensions to be consis
https://bugs.python.org/issue34845  closed by gvanrossum

#34851: re.error - for the search function in the re module
https://bugs.python.org/issue34851  closed by ronaldoussoren

#34853: Python django cors
https://bugs.python.org/issue34853  closed by ammar2

#34854: Crash in string annotations in lambda with keyword-only argume
https://bugs.python.org/issue34854  closed by serhiy.storchaka

#34859: python core in string substring search
https://bugs.python.org/issue34859  closed by ammar2

#34860: fix test_sqlite for AIX
https://bugs.python.org/issue34860  closed by Michael.Felt

#34862: No longer builds on OpenBSD due to missing definition of conve
https://bugs.python.org/issue34862  closed by benjamin.peterson

#34863: Idle Mac scrolling only down
https://bugs.python.org/issue34863  closed by terry.reedy

#34865: Incorrect assignment of optional argument when partial match w
https://bugs.python.org/issue34865  closed by eric.smith

#34868: bad error message when combining _ grouping specifier with inv
https://bugs.python.org/issue34868  closed by benjamin.peterson

#34869: remove LDLAST
https://bugs.python.org/issue34869  closed by benjamin.peterson

#34871: test_site fails if run after test_inspect
https://bugs.python.org/issue34871  closed by yselivanov

#34873: re.finditer behaviour in re.MULTILINE mode fails to match firs
https://bugs.python.org/issue34873  closed by tdawes

#34874: Python 3.6.3 command script wrapped in single quotes produces 
https://bugs.python.org/issue34874  closed by Tim McDonough

#34875: Change .js mime to "text/javascript"
https://bugs.python.org/issue34875  closed by asvetlov

#34878: Lock Objects documentation bug
https://bugs.python.org/issue34878  closed by benjamin.peterson

#34879: bytesobject.c: Possible null pointer dereference due to format
https://bugs.python.org/issue34879  closed by serhiy.storchaka

#34887: bytes subclass __repr__ raise SystemError when set to bytes.de
https://bugs.python.org/issue34887  closed by benjamin.peterson

#34889: int.to_bytes and int.from_bytes should default to the system b
https://bugs.python.org/issue34889  closed by benjamin.peterson

#34892: persistence of attributes with new instance
https://bugs.python.org/issue34892  closed by serhiy.storchaka

#34894: Unexpected error while unpickling lxml.etree.Element object
https://bugs.python.org/issue34894  closed by serhiy.storchaka

#34896: Unable to install Python 3.5
https://bugs.python.org/issue34896  closed by zach.ware

From robb at datalogics.com  Fri Oct  5 16:01:02 2018
From: robb at datalogics.com (Rob Boehne)
Date: Fri, 5 Oct 2018 20:01:02 +0000
Subject: [Python-Dev] AIX to stable, what does that take?
Message-ID: <C8704FD8-F421-4050-9767-E879A5152C36@datalogics.com>



?On 10/5/18, 10:33 AM, "Python-Dev on behalf of Michael Haubenwallner" <python-dev-bounces+robb=datalogics.com at python.org on behalf of michael.haubenwallner at ssi-schaefer.com> wrote:

    >
    >... I build everything myself, using xlc
    >(gcc introduces the need for a GNU RTE, e.g., glibc).
    
    Using gcc does *not* require to use glibc or even GNU binutils at all.
    Except for gcc's own runtime libraries, there's no need for a GNU RTE.
    In fact, in Gentoo Prefix I do use gcc as the compiler, configured to
    use AIX provided binutils (as, ld, nm, ...), with AIX libc as RTE.
    
I think the author was referring to the dependency on libgcc_s when using gcc.
It's typical for native UNIX package builders to use gcc only when necessary because the correct runtime is always installed (if the os running it is newer) and therefore won't clash when something else in the process space is using a different version of libgcc_s (I'm not sure what the ABI guarantees are with libgcc_s specifically, and neither are UNIX packagers - not necessarily anyway)
It also eliminates the need to ship a version of libgcc_s as a shared library.



From aixtools at felt.demon.nl  Fri Oct  5 16:17:31 2018
From: aixtools at felt.demon.nl (Michael)
Date: Fri, 5 Oct 2018 22:17:31 +0200
Subject: [Python-Dev] AIX to stable, what does that take?
In-Reply-To: <bd343118-a8d3-0a56-518b-48220157c760@ssi-schaefer.com>
References: <5cdbb644-4f1d-46a7-de0c-80beebb0822d@felt.demon.nl>
 <CAEfz+TxGQ8rz55FSOYpN+JVHqUarvP9gv8h6F9723XPwwaUGzQ@mail.gmail.com>
 <cd367b92-14a2-11b3-baa6-db9181407cbc@felt.demon.nl>
 <bd343118-a8d3-0a56-518b-48220157c760@ssi-schaefer.com>
Message-ID: <3f623b24-68d5-67a3-f83f-ef97c7a630b7@felt.demon.nl>

On 05/10/2018 16:15, Michael Haubenwallner wrote:
> Hi Michael,
>
> being on a similar road with Gentoo Prefix, I really do appreciate
> your AIX related work!
>
> However, for two (not so minor) topics I've got a little different
> experience, which I think should be mentioned here for completion:
Always.
>
> On 10/04/2018 11:13 AM, Michael Felt wrote:
>> On 10/4/2018 10:30 AM, INADA Naoki wrote:
>>> Hello,
>>>
>>> First of all, congratulations on passing all test on AIX.
>>> As a one of core developer, I don't know anything about AIX.
>>> If my change breaks AIX build, I can't investigate what's happened.
>>>
>>> So I think we need following in devguide:
>>>
>>> * Brief description about AIX, from developer's point of view.
>> This I might be able to do. Bullet form:
>> ... I build everything myself, using xlc
>> (gcc introduces the need for a GNU RTE, e.g., glibc).
> Using gcc does *not* require to use glibc or even GNU binutils at all.
> Except for gcc's own runtime libraries, there's no need for a GNU RTE.
> In fact, in Gentoo Prefix I do use gcc as the compiler, configured to
> use AIX provided binutils (as, ld, nm, ...), with AIX libc as RTE.
Well, this is something I learned - second hand - from someone who
worked hard to make much more OSS available than I. Probably wrong then
- in how I came to my conclusion - but the few things I tried to compile
"asis" to shared libraries would not work without also a lot of the gcc
compiler libraries. While I could have bitten the bullet and just found
a way to add those I was warned that different versions of gcc need
different level of supporting files.
>> * finally, a bit deeper: while the AIX linker loader supports svr4
>> shared libraries (it is the data, not the file name) it also supports
>> having multiple shared libraries in a classic archive. So, rather that
>> .../lib/libxxx.so and .../lib64/libxxx.so AIX prefers .../lib/libxxx.a
>> with two so-called members, with same or different names. The one found
>> is not it's name, but the symbol name and size of the ABI (32-bit or 64-bit)
> While this all is true, having multiple *versions* of one shared library in
> one single file is a PITA for package managers - both human or software.
Yes, it is a necessary pain. My secret is a) do not touch /usr/lib -
leave what is as it is, and in the few situations where it must be in
/usr/lib I add/replace named archive members with my new ones - and all
the other ones get extracted, modify a flag in their respective header -
so that the linker knows they are only to be used for applications that
expect them - not for new applications.
> But fortunately, the AIX linker does support so called "Import Files",
> allowing for *filename based* shared library versioning like on Linux,
> while still allowing for both ABIs in a single library archive file.
>
> For example, libtool provides the --with-aix-soname={aix|svr4|both}
> configure flag since libtool-2.4.4.  Although the default will stay
> at 'aix' here, in Gentoo Prefix I do use 'svr4' only.  This actually
> is a package manager's decision, ideally for all depending packages.
> As gcc does use libtool, for more information please refer to
> https://gcc.gnu.org/install/configure.html#WithAixSoname
> But note that "Import Files" should work with xlc as well.
Actually, more detail than I really want to know. I recall the day when
a library was a collection of "static" .o files, and ranlib was nearly
always needed - Or you told the linker to link against that library
multiple times. And I recall going to my first conference where "RPC"
was the next greatest thing, and shared libraries were going to be such
a space savior - both on disk and in memory. And I was always more of a
bsd fan having schooled myself on UNIX v7, then bsd (2.9 iirc) and bsd
4.1, 4.2 (in comes tcpip) and 4.3. But I diverge :p

To return briefly to the question of what is AIX for the developer -
very flexible. You can choose your architecture and it generally, just
works. I wrote some scripts as a front-end for packaging and most
packages are a one-liner - that runs configure, make, makeinstall (using
DESTDIR) and then packaging the DESTDIR.

As far as python development and AIX goes I am open to helping others.
Been doing that for more years than I care to count. :)
>
> Thanks!
> /haubi/




From aixtools at felt.demon.nl  Fri Oct  5 16:27:20 2018
From: aixtools at felt.demon.nl (Michael)
Date: Fri, 5 Oct 2018 22:27:20 +0200
Subject: [Python-Dev] AIX to stable, what does that take?
In-Reply-To: <C8704FD8-F421-4050-9767-E879A5152C36@datalogics.com>
References: <C8704FD8-F421-4050-9767-E879A5152C36@datalogics.com>
Message-ID: <da3ffa9b-1e3d-6d47-0f89-a308727fa305@felt.demon.nl>

On 05/10/2018 22:01, Rob Boehne wrote:
> ?On 10/5/18, 10:33 AM, "Python-Dev on behalf of Michael Haubenwallner" <python-dev-bounces+robb=datalogics.com at python.org on behalf of michael.haubenwallner at ssi-schaefer.com> wrote:
>
>     >
>     >... I build everything myself, using xlc
>     >(gcc introduces the need for a GNU RTE, e.g., glibc).
>     
>     Using gcc does *not* require to use glibc or even GNU binutils at all.
>     Except for gcc's own runtime libraries, there's no need for a GNU RTE.
>     In fact, in Gentoo Prefix I do use gcc as the compiler, configured to
>     use AIX provided binutils (as, ld, nm, ...), with AIX libc as RTE.
>     
> I think the author was referring to the dependency on libgcc_s when using gcc.
> It's typical for native UNIX package builders to use gcc only when necessary because the correct runtime is always installed (if the os running it is newer) and therefore won't clash when something else in the process space is using a different version of libgcc_s (I'm not sure what the ABI guarantees are with libgcc_s specifically, and neither are UNIX packagers - not necessarily anyway)
Thank you Rob. My core mistake is calling it glibc (that is the gnome
libc not that I think back), and libgcc* are something else entirely.

In any case, I need to get my facts more accurate.
> It also eliminates the need to ship a version of libgcc_s as a shared library.
That would make life easier. Would probably have to package gcc on my
own to get it work that way though.
>
>
>


From brett at python.org  Fri Oct  5 17:49:27 2018
From: brett at python.org (Brett Cannon)
Date: Fri, 5 Oct 2018 14:49:27 -0700
Subject: [Python-Dev] PEP 544 status (forked off "Petr Viktorin as
 BDFL-Delegate for PEP 580")
In-Reply-To: <CAP7+vJJfJugqYFBhP8Dtf0=X3ju7xGFh43BXrXxiS_5dvG1iug@mail.gmail.com>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <0d922ba1b09b46ad8efc08056c6e5771@xmail103.UGent.be>
 <5BB4E99B.3000905@UGent.be>
 <CAP7+vJJfJugqYFBhP8Dtf0=X3ju7xGFh43BXrXxiS_5dvG1iug@mail.gmail.com>
Message-ID: <CAP1=2W6Xbw6CwsaKW9_gaFSdrBSqm9cooHMu_nxxpxpBETjxww@mail.gmail.com>

I think whatever governance we end up with would have named you
BDFL-delegate anyway, Guido, so I think you're just taking the time machine
for a spin again. ;)

On Wed, 3 Oct 2018 at 09:40, Guido van Rossum <guido at python.org> wrote:

> The process for PEP 544 is off-topic for that thread so I'm starting a new
> one.
>
> I have promised its author to approve it after certain minor changes (that
> we both agree on) have been made. It's not an example of how PEP acceptance
> in general will works until governance is sorted out though -- PEP 544 is a
> very unique special case. (For one, it's uncontroversial -- the reason it's
> not been accepted yet is that its author is busy with other things.)
>
> On Wed, Oct 3, 2018 at 9:12 AM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
>
>> On 2018-10-03 17:06, ?ukasz Langa wrote:
>> > That's the only
>> > reason why PEP 544 is not yet accepted for example.
>>
>> Did you actually try to get PEP 544 accepted or to appoint a
>> BDFL-Delegate? I don't find any discussions about PEP 544 after the
>> stepping down of the BDFL.
>>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181005/f3decadc/attachment.html>

From brett at python.org  Fri Oct  5 17:50:30 2018
From: brett at python.org (Brett Cannon)
Date: Fri, 5 Oct 2018 14:50:30 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <pp4lpg$8us$1@blaine.gmane.org>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
Message-ID: <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>

I'm also fine with saying that keys in **kwargs that are not proper
identifiers is an implementation detail.

On Thu, 4 Oct 2018 at 02:20, Serhiy Storchaka <storchaka at gmail.com> wrote:

> 04.10.18 11:56, Steven D'Aprano ????:
> > While keyword arguments have to be identifiers, using **kwargs allows
> > arbitrary strings which aren't identifiers:
> >
> > py> def spam(**kwargs):
> > ....     print(kwargs)
> > ....
> > py> spam(**{"something arbitrary": 1, '\n': 2})
> > {'something arbitrary': 1, '\n': 2}
> >
> >
> > There is some discussion on Python-Ideas on whether or not that
> > behaviour ought to be considered a language feature, an accident of
> > implementation, or a bug.
> >
> > Can we get some guidence on this please?
>
> This is an implementation detail. Currently CPython doesn't ensure that
> keyword argument names are identifiers for performance reasons. But this
> can be changed in future versions or in other implementations.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181005/aa9b7ff3/attachment-0001.html>

From brett at python.org  Fri Oct  5 17:52:39 2018
From: brett at python.org (Brett Cannon)
Date: Fri, 5 Oct 2018 14:52:39 -0700
Subject: [Python-Dev] Should assert continue to do a LOAD_GLOBAL on
 AssertionError?
In-Reply-To: <1538620714.2141716.1530108600.48469F20@webmail.messagingengine.com>
References: <20181003155937.GM21220@ando.pearwood.info>
 <1538620714.2141716.1530108600.48469F20@webmail.messagingengine.com>
Message-ID: <CAP1=2W5+48UkZ8rs-fNk1AQ4XQ5YXLGo4v1kzeDjvYks79YdrA@mail.gmail.com>

On Wed, 3 Oct 2018 at 19:39, Benjamin Peterson <benjamin at python.org> wrote:

>
>
> On Wed, Oct 3, 2018, at 08:59, Steven D'Aprano wrote:
> > On the bug tracker, there's a discussion about the current behaviour of
> > the assert statement, where shadowing AssertionError will change the
> > behaviour of the assertion.
> >
> > https://bugs.python.org/issue34880
> >
> > Currently, assert does a LOAD_GLOBAL on AssertionError, which means if
> > you shadow the name, you get a different exception. This behaviour goes
> > back to Python 1.5.
> >
> > I'm looking for guidance here, is this the intended behaviour, or an
> > accident? Should it be changed to better match other builtins?
>
> The behavior certainly has been relied on historically by py.test. By
> replacing builtins.AssertionError, you can improve the error message of the
> AssertionError by, e.g., inspecting the failing frame. py.test's code to do
> this was deleted in 2016, but other code bases may still be relying on this
> hack. It's probably okay to change the behavior in 3.8 with the
> understanding that a revert may be necessary if some clever hack surfaces.
>

I like Guido's reasoning that syntax probably shouldn't be affected by
overloads unless otherwise documented as so, and Benjamin's approach to
solving it for 3.8 and then potentially reverting if it breaks too much.

-Brett


> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181005/d65f4af2/attachment.html>

From ncoghlan at gmail.com  Sat Oct  6 10:54:43 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 7 Oct 2018 00:54:43 +1000
Subject: [Python-Dev] PEP 544 status (forked off "Petr Viktorin as
 BDFL-Delegate for PEP 580")
In-Reply-To: <CAP1=2W6Xbw6CwsaKW9_gaFSdrBSqm9cooHMu_nxxpxpBETjxww@mail.gmail.com>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <0d922ba1b09b46ad8efc08056c6e5771@xmail103.UGent.be>
 <5BB4E99B.3000905@UGent.be>
 <CAP7+vJJfJugqYFBhP8Dtf0=X3ju7xGFh43BXrXxiS_5dvG1iug@mail.gmail.com>
 <CAP1=2W6Xbw6CwsaKW9_gaFSdrBSqm9cooHMu_nxxpxpBETjxww@mail.gmail.com>
Message-ID: <CADiSq7eyXECxANCWe1+HZ8zsRpZdZpK8NoPrAvfMV8HePQzg0A@mail.gmail.com>

On Sat, 6 Oct 2018 at 07:57, Brett Cannon <brett at python.org> wrote:
>
> I think whatever governance we end up with would have named you BDFL-delegate anyway, Guido, so I think you're just taking the time machine for a spin again. ;)

I was going to suggest that Guido list himself as BDFL-Delegate as
then it could follow the same caretaker model we're suggesting for the
fast C calling API.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Sat Oct  6 16:53:06 2018
From: guido at python.org (Guido van Rossum)
Date: Sat, 6 Oct 2018 13:53:06 -0700
Subject: [Python-Dev] PEP 544 status (forked off "Petr Viktorin as
 BDFL-Delegate for PEP 580")
In-Reply-To: <CADiSq7eyXECxANCWe1+HZ8zsRpZdZpK8NoPrAvfMV8HePQzg0A@mail.gmail.com>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <0d922ba1b09b46ad8efc08056c6e5771@xmail103.UGent.be>
 <5BB4E99B.3000905@UGent.be>
 <CAP7+vJJfJugqYFBhP8Dtf0=X3ju7xGFh43BXrXxiS_5dvG1iug@mail.gmail.com>
 <CAP1=2W6Xbw6CwsaKW9_gaFSdrBSqm9cooHMu_nxxpxpBETjxww@mail.gmail.com>
 <CADiSq7eyXECxANCWe1+HZ8zsRpZdZpK8NoPrAvfMV8HePQzg0A@mail.gmail.com>
Message-ID: <CAP7+vJ+RVMsujnsdnSO79ay5EkBKdOC6guVj-TtZimHHnXJ5HA@mail.gmail.com>

OK, then maybe someone (not me) can merge
https://github.com/python/peps/pull/799 for me.


On Sat, Oct 6, 2018 at 7:54 AM Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Sat, 6 Oct 2018 at 07:57, Brett Cannon <brett at python.org> wrote:
> >
> > I think whatever governance we end up with would have named you
> BDFL-delegate anyway, Guido, so I think you're just taking the time machine
> for a spin again. ;)
>
> I was going to suggest that Guido list himself as BDFL-Delegate as
> then it could follow the same caretaker model we're suggesting for the
> fast C calling API.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181006/9184361c/attachment.html>

From rosuav at gmail.com  Sat Oct  6 16:56:17 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 7 Oct 2018 07:56:17 +1100
Subject: [Python-Dev] PEP 544 status (forked off "Petr Viktorin as
 BDFL-Delegate for PEP 580")
In-Reply-To: <CAP7+vJ+RVMsujnsdnSO79ay5EkBKdOC6guVj-TtZimHHnXJ5HA@mail.gmail.com>
References: <5BB4B21D.2070302@UGent.be>
 <CAEfz+TyMjHbE7LvM=RYnGcKhm1VbQ5BnY6kL-BLopQmOFCe9dw@mail.gmail.com>
 <0d922ba1b09b46ad8efc08056c6e5771@xmail103.UGent.be>
 <5BB4E99B.3000905@UGent.be>
 <CAP7+vJJfJugqYFBhP8Dtf0=X3ju7xGFh43BXrXxiS_5dvG1iug@mail.gmail.com>
 <CAP1=2W6Xbw6CwsaKW9_gaFSdrBSqm9cooHMu_nxxpxpBETjxww@mail.gmail.com>
 <CADiSq7eyXECxANCWe1+HZ8zsRpZdZpK8NoPrAvfMV8HePQzg0A@mail.gmail.com>
 <CAP7+vJ+RVMsujnsdnSO79ay5EkBKdOC6guVj-TtZimHHnXJ5HA@mail.gmail.com>
Message-ID: <CAPTjJmpnY=nmCnSu98dqVTc=kDXKq0BWw4ORzoS6DdXOnXM0Vw@mail.gmail.com>

On Sun, Oct 7, 2018 at 7:55 AM Guido van Rossum <guido at python.org> wrote:
>
> OK, then maybe someone (not me) can merge https://github.com/python/peps/pull/799 for me.
>

Per discussion, merged.

ChrisA

From nad at python.org  Sun Oct  7 01:34:19 2018
From: nad at python.org (Ned Deily)
Date: Sun, 7 Oct 2018 01:34:19 -0400
Subject: [Python-Dev] 3.7.1 and 3.6.7 Release Update
Message-ID: <E4F22DD5-4C14-4EB8-971E-302184351533@python.org>

[duplicated from https://discuss.python.org/c/committers]

We issued the 3.7.1rc1 and 3.6.7rc1 release candidates a little over 11 days ago with the plan to have final releases for each available by today, assuming no new issues arose since the rc1 cutoffs.  While the good news is that, to the best of my knowledge, no regressions have been reported for the release candidates (or remain unfixed), several critical and security-related issues have identified during the rc phase.  Most of these have already been addressed (thanks for the quick responses!) but there remain a couple that still need to be finalized.  Since there is a non-zero amount of fixes going in after rc1, I think we need to do another set of release candidates once everything is resolved.  Because a number of core developers are right now attending various Python-related events, I am going to aim for tagging 3.7.1rc2 and 3.6.7rc2 after 2018-10-09 23:59 AoE, if at all possible, with final releases targeted for about 10 days later.  I will send out an update as we approach that time.

Just a reminder to core developers: as always please try to get fixes merged in to all relevant branches as soon as you feel they are ready.  There is no need to hold off during a release candidate phase.  Release managers will cherry pick as necessary appropriate fixes from the release branches into subsequent release candidates or final releases.  The sooner changes are merged, the sooner they will get buildbot exposure and potential exposure in the wider community.

Thanks!

--
  Ned Deily
  nad at python.org -- []


From chris.barker at noaa.gov  Sun Oct  7 13:34:57 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 7 Oct 2018 10:34:57 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
Message-ID: <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>

On Fri, Oct 5, 2018 at 3:01 PM Brett Cannon <brett at python.org> wrote:

> I'm also fine with saying that keys in **kwargs that are not proper
> identifiers is an implementation detail.
>

It's not just **kwargs -- you can also use arbitrary names with setattr() /
getattr() :

In [6]: setattr(foo, "4 not an identifier", "this works")

In [7]: getattr(foo, "4 not an identifier")
Out[7]: 'this works'

Which brings up a question I've had for years -- is the fact that cPython
uses a regular old dict for namespaces (and **kwargs) part of the language
spec, or an implementation detail?

I would say that for the get/setattr() example, it is kinda handy when you
want to use a class instance to model some external data structure that may
have different identifier rules. Though I tend to think that's mingling
data and code too much.

-CHB


>
> On Thu, 4 Oct 2018 at 02:20, Serhiy Storchaka <storchaka at gmail.com> wrote:
>
>> 04.10.18 11:56, Steven D'Aprano ????:
>> > While keyword arguments have to be identifiers, using **kwargs allows
>> > arbitrary strings which aren't identifiers:
>> >
>> > py> def spam(**kwargs):
>> > ....     print(kwargs)
>> > ....
>> > py> spam(**{"something arbitrary": 1, '\n': 2})
>> > {'something arbitrary': 1, '\n': 2}
>> >
>> >
>> > There is some discussion on Python-Ideas on whether or not that
>> > behaviour ought to be considered a language feature, an accident of
>> > implementation, or a bug.
>> >
>> > Can we get some guidence on this please?
>>
>> This is an implementation detail. Currently CPython doesn't ensure that
>> keyword argument names are identifiers for performance reasons. But this
>> can be changed in future versions or in other implementations.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181007/0c685d69/attachment.html>

From jmcs at jsantos.eu  Sun Oct  7 14:42:02 2018
From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=)
Date: Sun, 7 Oct 2018 20:42:02 +0200
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
Message-ID: <CAH_XWH3uLdu++iU9dteztTiqq2W0s+iG_WLusod33gyk+2+JEw@mail.gmail.com>

*locals *and *globals* are documented as dictionaries (for example exec's
documentation states that " If only *globals* is provided, it must be a
dictionary") but __dict__ is described as " [a] dictionary or other mapping
object".


On Sun, 7 Oct 2018 at 19:38, Chris Barker via Python-Dev <
python-dev at python.org> wrote:

> On Fri, Oct 5, 2018 at 3:01 PM Brett Cannon <brett at python.org> wrote:
>
>> I'm also fine with saying that keys in **kwargs that are not proper
>> identifiers is an implementation detail.
>>
>
> It's not just **kwargs -- you can also use arbitrary names with setattr()
> / getattr() :
>
> In [6]: setattr(foo, "4 not an identifier", "this works")
>
> In [7]: getattr(foo, "4 not an identifier")
> Out[7]: 'this works'
>
> Which brings up a question I've had for years -- is the fact that cPython
> uses a regular old dict for namespaces (and **kwargs) part of the language
> spec, or an implementation detail?
>
> I would say that for the get/setattr() example, it is kinda handy when you
> want to use a class instance to model some external data structure that may
> have different identifier rules. Though I tend to think that's mingling
> data and code too much.
>
> -CHB
>
>
>>
>> On Thu, 4 Oct 2018 at 02:20, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>>
>>> 04.10.18 11:56, Steven D'Aprano ????:
>>> > While keyword arguments have to be identifiers, using **kwargs allows
>>> > arbitrary strings which aren't identifiers:
>>> >
>>> > py> def spam(**kwargs):
>>> > ....     print(kwargs)
>>> > ....
>>> > py> spam(**{"something arbitrary": 1, '\n': 2})
>>> > {'something arbitrary': 1, '\n': 2}
>>> >
>>> >
>>> > There is some discussion on Python-Ideas on whether or not that
>>> > behaviour ought to be considered a language feature, an accident of
>>> > implementation, or a bug.
>>> >
>>> > Can we get some guidence on this please?
>>>
>>> This is an implementation detail. Currently CPython doesn't ensure that
>>> keyword argument names are identifiers for performance reasons. But this
>>> can be changed in future versions or in other implementations.
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jmcs%40jsantos.eu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181007/abe7dfd6/attachment.html>

From hodgestar+pythondev at gmail.com  Sun Oct  7 14:45:18 2018
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Sun, 7 Oct 2018 20:45:18 +0200
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <20181004085633.GP21220@ando.pearwood.info>
References: <20181004085633.GP21220@ando.pearwood.info>
Message-ID: <CAD5NRCERhWLK+mcg6-rq9FYmkV21wHYGkqt6aqh3mLGW8KWrVA@mail.gmail.com>

I would consider it a feature. My reasoning is that the restriction on
what can be used as an identifier is a syntax restriction, not a
general restriction on what attributes or names can be.

From chris.barker at noaa.gov  Sun Oct  7 15:48:54 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 7 Oct 2018 12:48:54 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CAH_XWH3uLdu++iU9dteztTiqq2W0s+iG_WLusod33gyk+2+JEw@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <CAH_XWH3uLdu++iU9dteztTiqq2W0s+iG_WLusod33gyk+2+JEw@mail.gmail.com>
Message-ID: <CALGmxEJZ425JyMn9AmmpHFzekiK06sqGbj2jEmT2QD0aH3eD_Q@mail.gmail.com>

On Sun, Oct 7, 2018 at 11:42 AM Jo?o Santos <jmcs at jsantos.eu> wrote:

> *locals *and *globals* are documented as dictionaries (for example exec's
> documentation states that " If only *globals* is provided, it must be a
> dictionary")
>

well, that is specifically about exec() -- it may or may not apply to
everywhere nameapaces are used in the interpreter...


> but __dict__ is described as " [a] dictionary or other mapping object".
>

exactly.

-CHB

On Sun, 7 Oct 2018 at 19:38, Chris Barker via Python-Dev <
> python-dev at python.org> wrote:
>
>> On Fri, Oct 5, 2018 at 3:01 PM Brett Cannon <brett at python.org> wrote:
>>
>>> I'm also fine with saying that keys in **kwargs that are not proper
>>> identifiers is an implementation detail.
>>>
>>
>> It's not just **kwargs -- you can also use arbitrary names with setattr()
>> / getattr() :
>>
>> In [6]: setattr(foo, "4 not an identifier", "this works")
>>
>> In [7]: getattr(foo, "4 not an identifier")
>> Out[7]: 'this works'
>>
>> Which brings up a question I've had for years -- is the fact that cPython
>> uses a regular old dict for namespaces (and **kwargs) part of the language
>> spec, or an implementation detail?
>>
>> I would say that for the get/setattr() example, it is kinda handy when
>> you want to use a class instance to model some external data structure that
>> may have different identifier rules. Though I tend to think that's mingling
>> data and code too much.
>>
>> -CHB
>>
>>
>>>
>>> On Thu, 4 Oct 2018 at 02:20, Serhiy Storchaka <storchaka at gmail.com>
>>> wrote:
>>>
>>>> 04.10.18 11:56, Steven D'Aprano ????:
>>>> > While keyword arguments have to be identifiers, using **kwargs allows
>>>> > arbitrary strings which aren't identifiers:
>>>> >
>>>> > py> def spam(**kwargs):
>>>> > ....     print(kwargs)
>>>> > ....
>>>> > py> spam(**{"something arbitrary": 1, '\n': 2})
>>>> > {'something arbitrary': 1, '\n': 2}
>>>> >
>>>> >
>>>> > There is some discussion on Python-Ideas on whether or not that
>>>> > behaviour ought to be considered a language feature, an accident of
>>>> > implementation, or a bug.
>>>> >
>>>> > Can we get some guidence on this please?
>>>>
>>>> This is an implementation detail. Currently CPython doesn't ensure that
>>>> keyword argument names are identifiers for performance reasons. But
>>>> this
>>>> can be changed in future versions or in other implementations.
>>>>
>>>> _______________________________________________
>>>> Python-Dev mailing list
>>>> Python-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/python-dev
>>>> Unsubscribe:
>>>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/jmcs%40jsantos.eu
>>
>

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181007/7fce87df/attachment.html>

From tjreedy at udel.edu  Sun Oct  7 18:44:54 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 7 Oct 2018 18:44:54 -0400
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
Message-ID: <ppe253$oie$1@blaine.gmane.org>

On 10/7/2018 1:34 PM, Chris Barker via Python-Dev wrote:
> On Fri, Oct 5, 2018 at 3:01 PM Brett Cannon <brett at python.org 
> <mailto:brett at python.org>> wrote:
> 
>     I'm also fine with saying that keys in **kwargs that are not proper
>     identifiers is an implementation detail.
> 
> 
> It's not just **kwargs -- you can also use arbitrary names with 
> setattr() / getattr() :
> 
> In [6]: setattr(foo, "4 not an identifier", "this works")
> 
> In [7]: getattr(foo, "4 not an identifier")
> Out[7]: 'this works'

When this behavior of set/getattr was discussed a decade or so ago, 
Guido said not to disable it, but I believe he said it should not be 
considered a language feature.  There are other situations where CPython 
is 'looser' than the spec.

-- 
Terry Jan Reedy


From turnbull.stephen.fw at u.tsukuba.ac.jp  Sun Oct  7 22:22:32 2018
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Mon, 8 Oct 2018 11:22:32 +0900
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <ppe253$oie$1@blaine.gmane.org>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
Message-ID: <23482.49000.212708.744665@turnbull.sk.tsukuba.ac.jp>

Terry Reedy writes:

 > When this behavior of set/getattr was discussed a decade or so ago,
 > Guido said not to disable it, but I believe he said it should not
 > be considered a language feature.  There are other situations where
 > CPython is 'looser' than the spec.

I'm pretty sure that all of these mappings that create namespaces are
*not* specified to be dicts.  globals() and locals() need to return
*something*, and dict is the only builtin mapping.  On the other hand,
it is explicit that changes to these dicts need not be reflected in the
namespace.

Note that the namespace defined by a class is *not* a dict, it's a
union (it may be a dict, or it may be slots):

    >>> class Foo:
    ...  __slots__ = ('a')
    ...  def __init__(self, x):
    ...   self.a = x
    ...   self.b = (x,)
    ... 
    >>> fu = Foo(1)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<stdin>", line 6, in __init__
    AttributeError: 'Foo' object has no attribute 'c'
    >>> class Foo:
    ...  __slots__ = ('a')
    ...  def __init__(self, x):
    ...   self.a = x
    ... 
    >>> fu = Foo(1)
    >>> print(fu.a)
    1
    >>> print(fu.__dict__)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'Foo' object has no attribute '__dict__'

This is a useful optimization if there are a lot of Foo objects, and
is somewhat faster.  As I understand it, while nobody has yet found a
reason to optimize other namespaces in such a way (or extend them in
some way, for that matter), the option is intentionally open.

Steve

From mingw.android at gmail.com  Tue Oct  9 02:58:53 2018
From: mingw.android at gmail.com (Ray Donnelly)
Date: Tue, 9 Oct 2018 07:58:53 +0100
Subject: [Python-Dev] Use of objdump within ctypes _get_soname()
Message-ID: <CAOYw7dswp8KHAQby-DD8wARi6paj9owYv_6PtDP-2O6MjkZiZw@mail.gmail.com>

Hi,

We ran into an issue on the Anaconda Distribution recently where we
added libarchive-c to conda-build (so we can un-compress more source
archive formats than tarfile supports) and everything was good a few
hours, until it hit various CI systems where objdump is not installed.

I was a bit surprised by this dependency and wondered if there'd be
interest in a fallback path that inspects the elf with some fairly
simple python code to determine the soname instead? I have code that
works already - though it could do with and a tidy up - and has been
tested on a variety of architectures. Would CPython be interested in
an attempt to upstream this?

Is it documented anywhere that objdump is needed to load some
extension modules on Linux?


Best regards,

Ray Donnelly,
Anaconda Inc,

From greg at krypto.org  Tue Oct  9 03:12:55 2018
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 9 Oct 2018 00:12:55 -0700
Subject: [Python-Dev] Use of objdump within ctypes _get_soname()
In-Reply-To: <CAOYw7dswp8KHAQby-DD8wARi6paj9owYv_6PtDP-2O6MjkZiZw@mail.gmail.com>
References: <CAOYw7dswp8KHAQby-DD8wARi6paj9owYv_6PtDP-2O6MjkZiZw@mail.gmail.com>
Message-ID: <CAGE7PNJgNyan7DC9NmHj8tR+mWLf6rk8OFZfmgPFpqKewxKq4g@mail.gmail.com>

On Mon, Oct 8, 2018 at 11:59 PM Ray Donnelly <mingw.android at gmail.com>
wrote:

> Hi,
>
> We ran into an issue on the Anaconda Distribution recently where we
> added libarchive-c to conda-build (so we can un-compress more source
> archive formats than tarfile supports) and everything was good a few
> hours, until it hit various CI systems where objdump is not installed.
>
> I was a bit surprised by this dependency and wondered if there'd be
> interest in a fallback path that inspects the elf with some fairly
> simple python code to determine the soname instead? I have code that
> works already - though it could do with and a tidy up - and has been
> tested on a variety of architectures. Would CPython be interested in
> an attempt to upstream this?
>
> Is it documented anywhere that objdump is needed to load some
> extension modules on Linux?
>

Wow, that looks like gross code buried within ctypes.util (which
libarchive-c uses) that calls various platforms versions of objdump or
equivalent.  File a bugs.python.org issue and submit a PR, native ELF
header reading code for this makes sense.

-G


>
>
> Best regards,
>
> Ray Donnelly,
> Anaconda Inc,
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181009/393f9a66/attachment.html>

From jdhardy at gmail.com  Tue Oct  9 12:37:48 2018
From: jdhardy at gmail.com (Jeff Hardy)
Date: Tue, 9 Oct 2018 09:37:48 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <ppe253$oie$1@blaine.gmane.org>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
Message-ID: <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>

On Sun, Oct 7, 2018 at 3:45 PM Terry Reedy <tjreedy at udel.edu> wrote:
>
> On 10/7/2018 1:34 PM, Chris Barker via Python-Dev wrote:
> > On Fri, Oct 5, 2018 at 3:01 PM Brett Cannon <brett at python.org
> > <mailto:brett at python.org>> wrote:
> >
> >     I'm also fine with saying that keys in **kwargs that are not proper
> >     identifiers is an implementation detail.
> >
> >
> > It's not just **kwargs -- you can also use arbitrary names with
> > setattr() / getattr() :
> >
> > In [6]: setattr(foo, "4 not an identifier", "this works")
> >
> > In [7]: getattr(foo, "4 not an identifier")
> > Out[7]: 'this works'
>
> When this behavior of set/getattr was discussed a decade or so ago,
> Guido said not to disable it, but I believe he said it should not be
> considered a language feature.  There are other situations where CPython
> is 'looser' than the spec.

>From an alternative implementation point of view, CPython's behaviour
*is* the spec. Practicality beats purity and all that.

- Jeff

From guido at python.org  Tue Oct  9 13:26:50 2018
From: guido at python.org (Guido van Rossum)
Date: Tue, 9 Oct 2018 10:26:50 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
Message-ID: <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>

My feeling is that limiting it to strings is fine, but checking those
strings for resembling identifiers is pointless and wasteful.

On Tue, Oct 9, 2018 at 9:40 AM Jeff Hardy <jdhardy at gmail.com> wrote:

> On Sun, Oct 7, 2018 at 3:45 PM Terry Reedy <tjreedy at udel.edu> wrote:
> >
> > On 10/7/2018 1:34 PM, Chris Barker via Python-Dev wrote:
> > > On Fri, Oct 5, 2018 at 3:01 PM Brett Cannon <brett at python.org
> > > <mailto:brett at python.org>> wrote:
> > >
> > >     I'm also fine with saying that keys in **kwargs that are not proper
> > >     identifiers is an implementation detail.
> > >
> > >
> > > It's not just **kwargs -- you can also use arbitrary names with
> > > setattr() / getattr() :
> > >
> > > In [6]: setattr(foo, "4 not an identifier", "this works")
> > >
> > > In [7]: getattr(foo, "4 not an identifier")
> > > Out[7]: 'this works'
> >
> > When this behavior of set/getattr was discussed a decade or so ago,
> > Guido said not to disable it, but I believe he said it should not be
> > considered a language feature.  There are other situations where CPython
> > is 'looser' than the spec.
>
> From an alternative implementation point of view, CPython's behaviour
> *is* the spec. Practicality beats purity and all that.
>
> - Jeff
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
-- 
--Guido (mobile)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181009/d39ba8a1/attachment.html>

From gregory.szorc at gmail.com  Tue Oct  9 17:02:02 2018
From: gregory.szorc at gmail.com (Gregory Szorc)
Date: Tue, 9 Oct 2018 14:02:02 -0700
Subject: [Python-Dev] Python startup time
In-Reply-To: <de448a2e-89c1-9fe2-f9e7-1c88c484bc2e@gmail.com>
References: <CAMpsgwY5-DgeXByK4FkCvWsvoOL_Ke1Aid20P+GC7MkJ+H6K7g@mail.gmail.com>
 <12e547e7-de6f-2dca-d3fe-47b63e108a8b@hastings.org>
 <de448a2e-89c1-9fe2-f9e7-1c88c484bc2e@gmail.com>
Message-ID: <6c2836f3-cc30-9d3b-569a-825b6dd0e25c@gmail.com>

On 5/1/2018 8:26 PM, Gregory Szorc wrote:
> On 7/19/2017 12:15 PM, Larry Hastings wrote:
>>
>>
>> On 07/19/2017 05:59 AM, Victor Stinner wrote:
>>> Mercurial startup time is already 45.8x slower than Git whereas tested
>>> Mercurial runs on Python 2.7.12. Now try to sell Python 3 to Mercurial
>>> developers, with a startup time 2x - 3x slower...
>>
>> When Matt Mackall spoke at the Python Language Summit some years back, I
>> recall that he specifically complained about Python startup time.? He
>> said Python 3 "didn't solve any problems for [them]"--they'd already
>> solved their Unicode hygiene problems--and that Python's slow startup
>> time was already a big problem for them.? Python 3 being /even slower/
>> to start was absolutely one of the reasons why they didn't want to upgrade.
>>
>> You might think "what's a few milliseconds matter".? But if you run
>> hundreds of commands in a shell script it adds up.? git's speed is one
>> of the few bright spots in its UX, and hg's comparative slowness here is
>> a palpable disadvantage.
>>
>>
>>> So please continue efforts for make Python startup even faster to beat
>>> all other programming languages, and finally convince Mercurial to
>>> upgrade ;-)
>>
>> I believe Mercurial is, finally, slowly porting to Python 3.
>>
>>     https://www.mercurial-scm.org/wiki/Python3
>>
>> Nevertheless, I can't really be annoyed or upset at them moving slowly
>> to adopt Python 3, as Matt's objections were entirely legitimate.
> 
> I just now found found this thread when searching the archive for
> threads about startup time. And I was searching for threads about
> startup time because Mercurial's startup time has been getting slower
> over the past few months and this is causing substantial pain.
> 
> As I posted back in 2014 [1], CPython's startup overhead was >10% of the
> total CPU time in Mercurial's test suite. And when you factor in the
> time to import modules that get Mercurial to a point where it can run
> commands, it was more like 30%!
> 
> Mercurial's full test suite currently runs `hg` ~25,000 times. Using
> Victor's startup time numbers of 6.4ms for 2.7 and 14.5ms for
> 3.7/master, Python startup overhead contributes ~160s on 2.7 and ~360s
> on 3.7/master. Even if you divide this by the number of available CPU
> cores, we're talking dozens of seconds of wall time just waiting for
> CPython to get to a place where Mercurial's first bytecode can execute.
> 
> And the problem is worse when you factor in the time it takes to import
> Mercurial's own modules.
> 
> As a concrete example, I recently landed a Mercurial patch [2] that
> stubs out zope.interface to prevent the import of 9 modules on every
> `hg` invocation. This "only" saved ~6.94ms for a typical `hg`
> invocation. But this decreased the CPU time required to run the test
> suite on my i7-6700K from ~4450s to ~3980s (~89.5% of original) - a
> reduction of almost 8 minutes of CPU time (and over 1 minute of wall time)!
> 
> By the time CPython gets Mercurial to a point where we can run useful
> code, we've already blown most of or past the time budget where humans
> perceive an action/command as instantaneous. If you ignore startup
> overhead, Mercurial's performance compares quite well to Git's for many
> operations. But the reality is that CPython startup overhead makes it
> look like Mercurial is non-instantaneous before Mercurial even has the
> opportunity to execute meaningful code!
> 
> Mercurial provides a `chg` program that essentially spins up a daemon
> `hg` process running a "command server" so the `chg` program [written in
> C - no startup overhead] can dispatch commands to an already-running
> Python/`hg` process and avoid paying the startup overhead cost. When you
> run Mercurial's test suite using `chg`, it completes *minutes* faster.
> `chg` exists mainly as a workaround for slow startup overhead.
> 
> Changing gears, my day job is maintaining Firefox's build system. We use
> Python heavily in the build system. And again, Python startup overhead
> is problematic. I don't have numbers offhand, but we invoke likely a few
> hundred Python processes as part of building Firefox. It should be
> several thousand. But, we've had to "hack" parts of the build system to
> "batch" certain build actions in single process invocations in order to
> avoid Python startup overhead. This undermines the ability of some build
> tools to formulate a reasonable understanding of the DAG and it causes a
> bit of pain for build system developers and makes it difficult to
> achieve "no-op" and fast incremental builds because we're always
> invoking certain Python processes because we've had to move DAG
> awareness out of the build backend and into Python. At some point, we'll
> likely replace Python code with Rust so the build system is more "pure"
> and easier to maintain and reason about.
> 
> I've seen posts in this thread and elsewhere in the CPython development
> universe that challenge whether milliseconds in startup time matter.
> Speaking as a Mercurial and Firefox build system developer,
> *milliseconds absolutely matter*. Going further, *fractions of
> milliseconds matter*. For Mercurial's test suite with its ~25,000 Python
> process invocations, 1ms translates to ~25s of CPU time. With 2.7,
> Mercurial can dispatch commands in ~50ms. When you load common
> extensions, it isn't uncommon to see process startup overhead of
> 100-150ms! A millisecond here. A millisecond there. Before you know it,
> we're talking *minutes* of CPU (and potentially wall) time in order to
> run Mercurial's test suite (or build Firefox, or ...).
> 
> From my perspective, Python process startup and module import overhead
> is a severe problem for Python. I don't say this lightly, but in my mind
> the problem causes me to question the viability of Python for popular
> use cases, such as CLI applications. When choosing a programming
> language, I want one that will scale as a project grows. Vanilla process
> overhead has Python starting off significantly slower than compiled code
> (or even Perl) and adding module import overhead into the mix makes
> Python slower and slower as projects grow. As someone who has to deal
> with this slowness on a daily basis, I can tell you that it is extremely
> frustrating and it does matter. I hope that the importance of the
> problem will be acknowledged (milliseconds *do* matter) and that
> creative minds will band together to address it. Since I am
> disproportionately impacted by this issue, if there's anything I can do
> to help, let me know.

We were debugging abysmally slow execution of Mercurial's test harness
on macOS and we discovered a new wrinkle to the startup time problem.

It appears that APFS acquires some shared locks/mutexes in the kernel
when executing readdir() and other filesystem system calls. When you
have several Python processes all starting at the same time, I/O
attached to module importing (import.c:case_ok() by the looks of it for
Python 2.7) becomes a stress test of sorts for this lock acquisition. On
my 6+6 core MacBook Pro, ~75% of overall system CPU is spent in the
kernel when executing the test harness with 12 parallel tests.

If we run the test harness with the persistent `chg` command server
(which eliminates Python process startup overhead), wall execution time
drops from ~37:43s to ~9:06s.

This problem of shared locks on read-only operations appears to be
similar to that of AUFS, which I've blogged about [1].

It is pretty common for non-compiled languages (like Python, Ruby, PHP,
Perl, etc) to stat() the world as part of looking for modules to load.
Typically, the filesystem's stat cache will save you and the overhead
from hundreds or thousands of lookups is trivial (after first load). But
it appears APFS is quite sensitive to it. Any work to reduce the number
of filesystem API calls during Python startup will likely have a
profound impact on APFS when multiple Python processes are starting. A
"frozen" application where modules are in a shared container file is
probably ideal.

Python 3.7 doesn't exhibit as much of a problem. But it is still there.
A brief audit of the importer code and call stacks confirms it is the
same problem - just less prevalent. Wall time execution of the test
harness from Python 2.7 to Python 3.7 drops from ~37:43s to ~20:39.
Overall kernel CPU time drops from ~75% to ~19%. And that wall time
improvement is despite Python 3's slower process startup. So locking in
the kernel is really a killer on Python 2.7.

While we're here, CPython might want to look into getdirentriesattr() as
a replacement for readdir(). We switched to it in Mercurial several
years ago to make `hg status` operations significantly faster [2]. I'm
not sure if it will yield a speedup on APFS though. But it's worth a
try. (If it does, you could probably make
os.listdir()/os.scandir()/os.walk() significantly faster on macOS.)

I hope someone finds this information useful to further improving
[startup] performance. (And given that Python 3.7 is substantially
faster by avoiding excessive readdir(), I wouldn't be surprised if this
problem is already known!)

[1] https://gregoryszorc.com/blog/2017/12/08/good-riddance-to-aufs/
[2] https://www.mercurial-scm.org/repo/hg/rev/05ccfe6763f1

From solipsis at pitrou.net  Tue Oct  9 17:23:42 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Oct 2018 23:23:42 +0200
Subject: [Python-Dev] Python startup time
References: <CAMpsgwY5-DgeXByK4FkCvWsvoOL_Ke1Aid20P+GC7MkJ+H6K7g@mail.gmail.com>
 <12e547e7-de6f-2dca-d3fe-47b63e108a8b@hastings.org>
 <de448a2e-89c1-9fe2-f9e7-1c88c484bc2e@gmail.com>
 <6c2836f3-cc30-9d3b-569a-825b6dd0e25c@gmail.com>
Message-ID: <20181009232342.666c2b48@fsol>


Hi,

On Tue, 9 Oct 2018 14:02:02 -0700
Gregory Szorc <gregory.szorc at gmail.com> wrote:
> 
> Python 3.7 doesn't exhibit as much of a problem. But it is still there.
> A brief audit of the importer code and call stacks confirms it is the
> same problem - just less prevalent. Wall time execution of the test
> harness from Python 2.7 to Python 3.7 drops from ~37:43s to ~20:39.
> Overall kernel CPU time drops from ~75% to ~19%. And that wall time
> improvement is despite Python 3's slower process startup. So locking in
> the kernel is really a killer on Python 2.7.

Thanks for the detailed feedback.

> I hope someone finds this information useful to further improving
> [startup] performance. (And given that Python 3.7 is substantially
> faster by avoiding excessive readdir(), I wouldn't be surprised if this
> problem is already known!)

The macOS problem wasn't known, but the general problem of filesystem
calls was (in relation with e.g. networked filesystems).

Significant work went into improving Python 3 in that regard after the
import mechanism was rewritten in pure Python.  Nowadays Python caches
the contents of all sys.path directories, so (once the cache is primed)
it's mostly a single stat() call per directory to check whether the
cache is up-to-date.  This is not entirely free, but massively better
than what Python 2 did, which was to stat() many filename patterns in
each sys.path directory.

(of course, the fact that Python 3 imports many more modules at startup
mitigates the end result a bit)


As a sidenote, I was always shocked with how the Mercurial test suite
was architected.  You're wasting so much time launching processes that
I wonder why you kept it that way for so long :-)

Regards

Antoine.



From steve at pearwood.info  Tue Oct  9 19:06:05 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 10 Oct 2018 10:06:05 +1100
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
Message-ID: <20181009230605.GV3817@ando.pearwood.info>

On Tue, Oct 09, 2018 at 09:37:48AM -0700, Jeff Hardy wrote:

> > When this behavior of set/getattr was discussed a decade or so ago,
> > Guido said not to disable it, but I believe he said it should not be
> > considered a language feature.  There are other situations where CPython
> > is 'looser' than the spec.
> 
> From an alternative implementation point of view, CPython's behaviour
> *is* the spec. Practicality beats purity and all that.

Are you speaking on behalf of all authors of alternate implementations, 
or even of some of them?

It certainly is not true that CPython's behaviour "is" the spec. PyPy 
keeps a list of CPython behaviour they don't match, either because they 
choose not to for other reasons, or because they believe that the 
CPython behaviour is buggy. I daresay IronPython and Jython have 
similar.

And this especially applies when CPython explicitly states that certain 
behaviour is implementation-dependent and could change in the future.


-- 
Steve

From steve at pearwood.info  Tue Oct  9 19:21:35 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 10 Oct 2018 10:21:35 +1100
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
Message-ID: <20181009232134.GW3817@ando.pearwood.info>

On Tue, Oct 09, 2018 at 10:26:50AM -0700, Guido van Rossum wrote:
> My feeling is that limiting it to strings is fine, but checking those
> strings for resembling identifiers is pointless and wasteful.

Sure. The question is, do we have to support uses where people 
intentionally smuggle non-identifier strings as keys via **kwargs?

I'm not saying we need to guard against it, only asking if we need to 
officially support it. The discussion on Python-Ideas is (partly) about 
making this a language feature.


-- 
Steve

From barry at python.org  Tue Oct  9 20:14:14 2018
From: barry at python.org (Barry Warsaw)
Date: Tue, 9 Oct 2018 17:14:14 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <20181009232134.GW3817@ando.pearwood.info>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
Message-ID: <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>

On Oct 9, 2018, at 16:21, Steven D'Aprano <steve at pearwood.info> wrote:
> 
> On Tue, Oct 09, 2018 at 10:26:50AM -0700, Guido van Rossum wrote:
>> My feeling is that limiting it to strings is fine, but checking those
>> strings for resembling identifiers is pointless and wasteful.
> 
> Sure. The question is, do we have to support uses where people
> intentionally smuggle non-identifier strings as keys via **kwargs?

I would not be in favor of that.  I think it doesn?t make sense to be able to smuggle those in via **kwargs when it?s not supported by Python?s grammar/syntax.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181009/1a5fc5fe/attachment.sig>

From guido at python.org  Tue Oct  9 20:26:11 2018
From: guido at python.org (Guido van Rossum)
Date: Tue, 9 Oct 2018 17:26:11 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
 <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
Message-ID: <CAP7+vJ+8xvrje_VT6ORtoms+HL4dMhD7S4T0YKD9xAJS9vNs+Q@mail.gmail.com>

On Tue, Oct 9, 2018 at 5:17 PM Barry Warsaw <barry at python.org> wrote:

> On Oct 9, 2018, at 16:21, Steven D'Aprano <steve at pearwood.info> wrote:
> >
> > On Tue, Oct 09, 2018 at 10:26:50AM -0700, Guido van Rossum wrote:
> >> My feeling is that limiting it to strings is fine, but checking those
> >> strings for resembling identifiers is pointless and wasteful.
> >
> > Sure. The question is, do we have to support uses where people
> > intentionally smuggle non-identifier strings as keys via **kwargs?
>
> I would not be in favor of that.  I think it doesn?t make sense to be able
> to smuggle those in via **kwargs when it?s not supported by Python?s
> grammar/syntax.
>

Well, it currently works in all Python implementations (definitely in
CPython, and presumably in PyPy and Jython because they tend to follow
CPython carefully). The less the spec leaves undefined the better, IMO, and
I fully expect we'll be breaking code that is doing this. So we might as
well make it the law.

For example, in some code bases it's a pretty common pattern to pass dicts
around using **kwds several levels deep, with no intention to unpack it
into individual keyword arguments -- the caller sends a dict, and the
receiver accepts a dict and does dict-y things to it. Sure, they probably
shouldn't be abusing **kwds, but they are, and I can't really blame them --
possibly this code evolved from a situation that did use keyword args.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181009/b406d4ac/attachment-0001.html>

From benjamin at python.org  Tue Oct  9 22:12:16 2018
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 09 Oct 2018 19:12:16 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
 <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
Message-ID: <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>



On Tue, Oct 9, 2018, at 17:14, Barry Warsaw wrote:
> On Oct 9, 2018, at 16:21, Steven D'Aprano <steve at pearwood.info> wrote:
> > 
> > On Tue, Oct 09, 2018 at 10:26:50AM -0700, Guido van Rossum wrote:
> >> My feeling is that limiting it to strings is fine, but checking those
> >> strings for resembling identifiers is pointless and wasteful.
> > 
> > Sure. The question is, do we have to support uses where people
> > intentionally smuggle non-identifier strings as keys via **kwargs?
> 
> I would not be in favor of that.  I think it doesn?t make sense to be 
> able to smuggle those in via **kwargs when it?s not supported by 
> Python?s grammar/syntax.

Can anyone think of a situation where it would be advantageous for an implementation to reject non-identifier string kwargs? I can't.

I agree with Guido?banning it would be too much trouble for no benefit.

From chris.jerdonek at gmail.com  Tue Oct  9 22:46:56 2018
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Tue, 9 Oct 2018 19:46:56 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
 <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
 <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>
Message-ID: <CAOTb1weprvDBMTiHs4_HdBw2vK+hnrSaxj+mj8RFk5Fxmta24Q@mail.gmail.com>

On Tue, Oct 9, 2018 at 7:13 PM Benjamin Peterson <benjamin at python.org> wrote:
> On Tue, Oct 9, 2018, at 17:14, Barry Warsaw wrote:
> > On Oct 9, 2018, at 16:21, Steven D'Aprano <steve at pearwood.info> wrote:
> > >
> > > On Tue, Oct 09, 2018 at 10:26:50AM -0700, Guido van Rossum wrote:
> > >> My feeling is that limiting it to strings is fine, but checking those
> > >> strings for resembling identifiers is pointless and wasteful.
> > >
> > > Sure. The question is, do we have to support uses where people
> > > intentionally smuggle non-identifier strings as keys via **kwargs?
> >
> > I would not be in favor of that.  I think it doesn?t make sense to be
> > able to smuggle those in via **kwargs when it?s not supported by
> > Python?s grammar/syntax.
>
> Can anyone think of a situation where it would be advantageous for an implementation to reject non-identifier string kwargs? I can't.

One possibility is that it could foreclose certain security bugs from
happening. For example, if someone has an API that accepts **kwargs,
they might have the mistaken assumption that the keys are identifiers
without special characters like ";" etc, and so they could make the
mistake of thinking they don't need to escape / sanitize them.

--Chris

>
> I agree with Guido?banning it would be too much trouble for no benefit.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com

From guido at python.org  Tue Oct  9 23:56:16 2018
From: guido at python.org (Guido van Rossum)
Date: Tue, 9 Oct 2018 20:56:16 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CAOTb1weprvDBMTiHs4_HdBw2vK+hnrSaxj+mj8RFk5Fxmta24Q@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
 <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
 <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>
 <CAOTb1weprvDBMTiHs4_HdBw2vK+hnrSaxj+mj8RFk5Fxmta24Q@mail.gmail.com>
Message-ID: <CAP7+vJLJ4FjuU1CG42oCgjg8fikmm7JC_2QOWMSm1RBkOq2zHA@mail.gmail.com>

On Tue, Oct 9, 2018 at 7:49 PM Chris Jerdonek <chris.jerdonek at gmail.com>
wrote:

> On Tue, Oct 9, 2018 at 7:13 PM Benjamin Peterson <benjamin at python.org>
> wrote:
> > On Tue, Oct 9, 2018, at 17:14, Barry Warsaw wrote:
> > > On Oct 9, 2018, at 16:21, Steven D'Aprano <steve at pearwood.info> wrote:
> > > >
> > > > On Tue, Oct 09, 2018 at 10:26:50AM -0700, Guido van Rossum wrote:
> > > >> My feeling is that limiting it to strings is fine, but checking
> those
> > > >> strings for resembling identifiers is pointless and wasteful.
> > > >
> > > > Sure. The question is, do we have to support uses where people
> > > > intentionally smuggle non-identifier strings as keys via **kwargs?
> > >
> > > I would not be in favor of that.  I think it doesn?t make sense to be
> > > able to smuggle those in via **kwargs when it?s not supported by
> > > Python?s grammar/syntax.
> >
> > Can anyone think of a situation where it would be advantageous for an
> implementation to reject non-identifier string kwargs? I can't.
>
> One possibility is that it could foreclose certain security bugs from
> happening. For example, if someone has an API that accepts **kwargs,
> they might have the mistaken assumption that the keys are identifiers
> without special characters like ";" etc, and so they could make the
> mistake of thinking they don't need to escape / sanitize them.
>

Hm, that's not an entirely unreasonable concern. How would an attacker get
such keys *into* the dict? One possible scenario would be something that
parses a traditional web query string into a dict, passes it down through
**kwds, and then turns it back into another query string without proper
quoting. But the most common (and easiest) way to turn a dict into a query
string is calling urlencode(), which quotes unsafe characters.

I think we needn't rush this (and when in doubt, status quo wins, esp. when
there's no BDFL :-).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181009/80527f7f/attachment.html>

From v+python at g.nevcal.com  Tue Oct  9 23:53:56 2018
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 9 Oct 2018 20:53:56 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CAOTb1weprvDBMTiHs4_HdBw2vK+hnrSaxj+mj8RFk5Fxmta24Q@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
 <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
 <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>
 <CAOTb1weprvDBMTiHs4_HdBw2vK+hnrSaxj+mj8RFk5Fxmta24Q@mail.gmail.com>
Message-ID: <92cb458e-00e0-54aa-2510-17fba330ab67@g.nevcal.com>

On 10/9/2018 7:46 PM, Chris Jerdonek wrote:
> On Tue, Oct 9, 2018 at 7:13 PM Benjamin Peterson <benjamin at python.org> wrote:
>> On Tue, Oct 9, 2018, at 17:14, Barry Warsaw wrote:
>>> On Oct 9, 2018, at 16:21, Steven D'Aprano <steve at pearwood.info> wrote:
>>>> On Tue, Oct 09, 2018 at 10:26:50AM -0700, Guido van Rossum wrote:
>>>>> My feeling is that limiting it to strings is fine, but checking those
>>>>> strings for resembling identifiers is pointless and wasteful.
>>>> Sure. The question is, do we have to support uses where people
>>>> intentionally smuggle non-identifier strings as keys via **kwargs?
>>> I would not be in favor of that.  I think it doesn?t make sense to be
>>> able to smuggle those in via **kwargs when it?s not supported by
>>> Python?s grammar/syntax.
>> Can anyone think of a situation where it would be advantageous for an implementation to reject non-identifier string kwargs? I can't.
> One possibility is that it could foreclose certain security bugs from
> happening. For example, if someone has an API that accepts **kwargs,
> they might have the mistaken assumption that the keys are identifiers
> without special characters like ";" etc, and so they could make the
> mistake of thinking they don't need to escape / sanitize them.
>
> --Chris
With that line of reasoning, one should make sure the data are 
identifiers too :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181009/7f067eea/attachment.html>

From chris.jerdonek at gmail.com  Wed Oct 10 02:41:32 2018
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Tue, 9 Oct 2018 23:41:32 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CAP7+vJLJ4FjuU1CG42oCgjg8fikmm7JC_2QOWMSm1RBkOq2zHA@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
 <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
 <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>
 <CAOTb1weprvDBMTiHs4_HdBw2vK+hnrSaxj+mj8RFk5Fxmta24Q@mail.gmail.com>
 <CAP7+vJLJ4FjuU1CG42oCgjg8fikmm7JC_2QOWMSm1RBkOq2zHA@mail.gmail.com>
Message-ID: <CAOTb1wdBAaJdw8j7MNnDKVCUtcKNN5wf=aTP94+iUd+WTg-wRg@mail.gmail.com>

On Tue, Oct 9, 2018 at 8:55 PM Guido van Rossum <guido at python.org> wrote:
> On Tue, Oct 9, 2018 at 7:49 PM Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
>> On Tue, Oct 9, 2018 at 7:13 PM Benjamin Peterson <benjamin at python.org> wrote:
>> > Can anyone think of a situation where it would be advantageous for an implementation to reject non-identifier string kwargs? I can't.
>>
>> One possibility is that it could foreclose certain security bugs from
>> happening. For example, if someone has an API that accepts **kwargs,
>> they might have the mistaken assumption that the keys are identifiers
>> without special characters like ";" etc, and so they could make the
>> mistake of thinking they don't need to escape / sanitize them.
>
>
> Hm, that's not an entirely unreasonable concern. How would an attacker get such keys *into* the dict?

I was just thinking json. It could be a config-file type situation, or
a web API that accepts json.

For example, there are JSON-RPC implementations in Python:
https://pypi.org/project/json-rpc/
that translate json dicts directly into **kwargs:
https://github.com/pavlov99/json-rpc/blob/f1b4e5e96661efd4026cb6143dc3acd75c6c4682/jsonrpc/manager.py#L112

On the server side, the application could be doing something like
assuming that the kwargs are e.g. column names paired with values to
construct a string in SQL or in some other language or format.

--Chris



> One possible scenario would be something that parses a traditional web query string into a dict, passes it down through **kwds, and then turns it back into another query string without proper quoting. But the most common (and easiest) way to turn a dict into a query string is calling urlencode(), which quotes unsafe characters.
>
> I think we needn't rush this (and when in doubt, status quo wins, esp. when there's no BDFL :-).
>
> --
> --Guido van Rossum (python.org/~guido)

From storchaka at gmail.com  Wed Oct 10 02:48:42 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 10 Oct 2018 09:48:42 +0300
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
 <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
 <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>
Message-ID: <ppk787$pq0$1@blaine.gmane.org>

10.10.18 05:12, Benjamin Peterson ????:
> On Tue, Oct 9, 2018, at 17:14, Barry Warsaw wrote:
>> On Oct 9, 2018, at 16:21, Steven D'Aprano <steve at pearwood.info> wrote:
>>>
>>> On Tue, Oct 09, 2018 at 10:26:50AM -0700, Guido van Rossum wrote:
>>>> My feeling is that limiting it to strings is fine, but checking those
>>>> strings for resembling identifiers is pointless and wasteful.
>>>
>>> Sure. The question is, do we have to support uses where people
>>> intentionally smuggle non-identifier strings as keys via **kwargs?
>>
>> I would not be in favor of that.  I think it doesn?t make sense to be
>> able to smuggle those in via **kwargs when it?s not supported by
>> Python?s grammar/syntax.
> 
> Can anyone think of a situation where it would be advantageous for an implementation to reject non-identifier string kwargs? I can't.

I can. The space of identifiers is smaller than the space of all 
strings. We need just 6 bits per character for ASCII identifiers and 16 
bits per character for Unicode identifiers. We could use a special kind 
of strings for more compact representation of identifiers. It may be 
even possible to encode all identifiers used in the stdlib and in the 
program as a tagged 64-bit pointer. Currently dict has specialized code 
for string keys, it could have specialization for identifiers (used only 
for keyword arguments, instance dicts, etc). Argument parsing code can 
also utilize the fact that a special hash for short identifiers doesn't 
have collizions and compare just hashes.

All this looks fantastic, but I would not close doors for future 
optimizations.


From ronaldoussoren at mac.com  Wed Oct 10 04:10:01 2018
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 10 Oct 2018 10:10:01 +0200
Subject: [Python-Dev] Python startup time
In-Reply-To: <6c2836f3-cc30-9d3b-569a-825b6dd0e25c@gmail.com>
References: <CAMpsgwY5-DgeXByK4FkCvWsvoOL_Ke1Aid20P+GC7MkJ+H6K7g@mail.gmail.com>
 <12e547e7-de6f-2dca-d3fe-47b63e108a8b@hastings.org>
 <de448a2e-89c1-9fe2-f9e7-1c88c484bc2e@gmail.com>
 <6c2836f3-cc30-9d3b-569a-825b6dd0e25c@gmail.com>
Message-ID: <2EDEFB52-D93B-42A4-9921-B56109984F44@mac.com>



> On 9 Oct 2018, at 23:02, Gregory Szorc <gregory.szorc at gmail.com> wrote:
> 
> 
> 
> While we're here, CPython might want to look into getdirentriesattr() as
> a replacement for readdir(). We switched to it in Mercurial several
> years ago to make `hg status` operations significantly faster [2]. I'm
> not sure if it will yield a speedup on APFS though. But it's worth a
> try. (If it does, you could probably make
> os.listdir()/os.scandir()/os.walk() significantly faster on macOS.)

Note that getdirentriesattr is deprecated as of macOS 10.10, getattrlistbulk
is the non-deprecated replacement (introduced in 10.10). 

Ronald
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181010/29b6ee4c/attachment.html>

From mingw.android at gmail.com  Wed Oct 10 07:24:54 2018
From: mingw.android at gmail.com (Ray Donnelly)
Date: Wed, 10 Oct 2018 12:24:54 +0100
Subject: [Python-Dev] Use of objdump within ctypes _get_soname()
In-Reply-To: <CAGE7PNJgNyan7DC9NmHj8tR+mWLf6rk8OFZfmgPFpqKewxKq4g@mail.gmail.com>
References: <CAOYw7dswp8KHAQby-DD8wARi6paj9owYv_6PtDP-2O6MjkZiZw@mail.gmail.com>
 <CAGE7PNJgNyan7DC9NmHj8tR+mWLf6rk8OFZfmgPFpqKewxKq4g@mail.gmail.com>
Message-ID: <CAOYw7dvdg3FpiBe2kWPoGU1BVByXJS-c3fY2sWzC91YrmPbU8w@mail.gmail.com>

I've looked into this more and it is more subtle than my initial
report suggests (as usual).

We have some custom code in AD around find_library() so it looks in
sys.prefix/lib (on Unix) and I had a bug in that code when _get_soname
returned None. I've fixed that now.

I believe there is a condition under which this could go wrong which
is if the soname returned by objdump differs from the filename and
then the filename is not found. This would happen to people with
objdump installed but not to people without it installed, so I feel
the behaviour here should be unified. If you think we should unify it
by implementing a python replacement for this functionality then I'll
get on with a PR (and file that bug report too), the other option is
to just remove this part and have just the fallbacks happen instead?
On Tue, Oct 9, 2018 at 8:13 AM Gregory P. Smith <greg at krypto.org> wrote:
>
>
>
> On Mon, Oct 8, 2018 at 11:59 PM Ray Donnelly <mingw.android at gmail.com> wrote:
>>
>> Hi,
>>
>> We ran into an issue on the Anaconda Distribution recently where we
>> added libarchive-c to conda-build (so we can un-compress more source
>> archive formats than tarfile supports) and everything was good a few
>> hours, until it hit various CI systems where objdump is not installed.
>>
>> I was a bit surprised by this dependency and wondered if there'd be
>> interest in a fallback path that inspects the elf with some fairly
>> simple python code to determine the soname instead? I have code that
>> works already - though it could do with and a tidy up - and has been
>> tested on a variety of architectures. Would CPython be interested in
>> an attempt to upstream this?
>>
>> Is it documented anywhere that objdump is needed to load some
>> extension modules on Linux?
>
>
> Wow, that looks like gross code buried within ctypes.util (which libarchive-c uses) that calls various platforms versions of objdump or equivalent.  File a bugs.python.org issue and submit a PR, native ELF header reading code for this makes sense.
>
> -G
>
>>
>>
>>
>> Best regards,
>>
>> Ray Donnelly,
>> Anaconda Inc,
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org

From chris.barker at noaa.gov  Thu Oct 11 13:27:08 2018
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Thu, 11 Oct 2018 13:27:08 -0400
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CAOTb1wdBAaJdw8j7MNnDKVCUtcKNN5wf=aTP94+iUd+WTg-wRg@mail.gmail.com>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
 <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
 <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>
 <CAOTb1weprvDBMTiHs4_HdBw2vK+hnrSaxj+mj8RFk5Fxmta24Q@mail.gmail.com>
 <CAP7+vJLJ4FjuU1CG42oCgjg8fikmm7JC_2QOWMSm1RBkOq2zHA@mail.gmail.com>
 <CAOTb1wdBAaJdw8j7MNnDKVCUtcKNN5wf=aTP94+iUd+WTg-wRg@mail.gmail.com>
Message-ID: <CALGmxELdxWE-Zhf5Z5OHQ+tCEg9a7+3g7WPz1zUEzav0z7wZ-Q@mail.gmail.com>

> On the server side, the application could be doing something like
> assuming that the kwargs are e.g. column names

This is exactly a use-case for non-identifier strings in kwargs. The
rules for valid field names could very well be different than Python?s
rules.

The kwargs implementation is not the place for sanitizing to take
place ? each app will need different sanitization rules.

-CHB

From samuele.pedroni at gmail.com  Fri Oct 12 03:08:32 2018
From: samuele.pedroni at gmail.com (Samuele Pedroni)
Date: Fri, 12 Oct 2018 09:08:32 +0200
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <20181004085633.GP21220@ando.pearwood.info>
References: <20181004085633.GP21220@ando.pearwood.info>
Message-ID: <CAGwLUHai7SKxO4=GAa0PGXNgd0G8Lr9Jd180dQiMr02ZQ_VVng@mail.gmail.com>

On Thu, Oct 4, 2018 at 10:58 AM Steven D'Aprano <steve at pearwood.info> wrote:

> While keyword arguments have to be identifiers, using **kwargs allows
> arbitrary strings which aren't identifiers:
>
> py> def spam(**kwargs):
> ...     print(kwargs)
> ...
> py> spam(**{"something arbitrary": 1, '\n': 2})
> {'something arbitrary': 1, '\n': 2}
>
>
> There is some discussion on Python-Ideas on whether or not that
> behaviour ought to be considered a language feature, an accident of
> implementation, or a bug.
>
>
I would expect this to be costly/annoying for implementations to enforce,
doing it at call time is probably too late to be efficient, it would need
help from dicts themselves or even strings.

A hack that currently works because of this is with dict itself:

>>> d = {'a-1': 1, 'a-2': 2, 'a-3': 3}
>>> d1 = dict(d, **{'a-2': -2, 'a-1': -1})
>>> d1 is d
False
>>> d1
{'a-1': -1, 'a-2': -2, 'a-3': 3}
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181012/eecd4783/attachment.html>

From steve at pearwood.info  Fri Oct 12 06:41:11 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 12 Oct 2018 21:41:11 +1100
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <CALGmxELdxWE-Zhf5Z5OHQ+tCEg9a7+3g7WPz1zUEzav0z7wZ-Q@mail.gmail.com>
References: <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <CAP7+vJJCSsibBW5o02F+-jXNKCWshOWgrAWevv9OFwNhjusZvA@mail.gmail.com>
 <20181009232134.GW3817@ando.pearwood.info>
 <364D8E29-B1FF-4082-AFFC-05F2BB6C295C@python.org>
 <1539137536.2690467.1536683584.3C4799A8@webmail.messagingengine.com>
 <CAOTb1weprvDBMTiHs4_HdBw2vK+hnrSaxj+mj8RFk5Fxmta24Q@mail.gmail.com>
 <CAP7+vJLJ4FjuU1CG42oCgjg8fikmm7JC_2QOWMSm1RBkOq2zHA@mail.gmail.com>
 <CAOTb1wdBAaJdw8j7MNnDKVCUtcKNN5wf=aTP94+iUd+WTg-wRg@mail.gmail.com>
 <CALGmxELdxWE-Zhf5Z5OHQ+tCEg9a7+3g7WPz1zUEzav0z7wZ-Q@mail.gmail.com>
Message-ID: <20181012104110.GE3817@ando.pearwood.info>

On Thu, Oct 11, 2018 at 01:27:08PM -0400, Chris Barker - NOAA Federal via Python-Dev wrote:
> > On the server side, the application could be doing something like
> > assuming that the kwargs are e.g. column names
> 
> This is exactly a use-case for non-identifier strings in kwargs.

Why not just pass a dict as an argument, rather than (ab)using kwargs?

Instead of:

- building a dict containing non-identifiers;
- unpacking it in the function call;
- have the interpreter re-pack it to a **kwargs;
- and then process it as a dict

we can cut out the two middle steps. So I must admit, I'm perplexed as 
to why people use an extra (superfluous?) ** to unpack a dict that's 
just going to be packed again. I just don't get it. *shrug*

I also wonder whether the use-cases for this would be reduced if we 
introduced verbatim names? 

https://mail.python.org/pipermail/python-ideas/2018-May/050791.html

Keys containing non-identifier characters like spaces and hyphens would 
still need the kwargs trick, but for reserved words you could just 
escape the argument:

def spam(eggs, \while=None):
    ...

spam(eggs=1234, \while=5678)


which frankly looks much better to me than 

spam(eggs=1234, **{"while": 5678})



-- 
Steve

From eric at trueblade.com  Fri Oct 12 07:32:31 2018
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 12 Oct 2018 07:32:31 -0400
Subject: [Python-Dev] [Python-checkins] bpo-34203: FAQ now recommends
 python 3.x over 2.x (GH-9796)
In-Reply-To: <42Wj0P517lzFrcx@mail.python.org>
References: <42Wj0P517lzFrcx@mail.python.org>
Message-ID: <125297d6-545f-3052-0900-9df2072f0057@trueblade.com>

On 10/12/2018 5:17 AM, Tal Einat wrote:

>   The latest stable releases can always be found on the `Python download page
> -<https://www.python.org/downloads/>`_.  There are two recommended production-ready
> -versions at this point in time, because at the moment there are two branches of
> -stable releases: 2.x and 3.x.  Python 3.x may be less useful than 2.x, since
> -currently there is more third party software available for Python 2 than for
> -Python 3.  Python 2 code will generally not run unchanged in Python 3.
> +<https://www.python.org/downloads/>`_.  There are two production-ready version
> +of Python: 2.x and 3.x, but the recommended one at this times is Python 3.x.

This should be "time", not "times". I'd fix it, but I'm unsure if this 
is being backported or not, and I don't want to mess up any merges 
before they're done.

I do think this should backported to 3.7, 3.6, and 2.7.

> +Although Python 2.x is still widely used, `it will not be
> +maintained after January 1, 2020 <https://www.python.org/dev/peps/pep-0373/>`_.
> +Python 2.x was known for having more third-party libraries available, however,
> +by the time of this writing, most of the widely used libraries support Python 3.x,

Should probably be "at the time of this writing".

> +and some are even dropping the Python 2.x support.

And this would read better as "are even dropping support for Python 2.x".

Eric

From seanharr11 at gmail.com  Fri Oct 12 08:33:32 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Fri, 12 Oct 2018 08:33:32 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
Message-ID: <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>

Hi Nathaniel - this if this solution can be made performant, than I would
be more than satisfied.

I think this would require removing "func" from the "task tuple", and
storing the "func" "once per worker" somewhere globally (maybe a class
attribute set post-fork?).

This also has the beneficial outcome of increasing general performance of
Pool.map and friends. I've seen MANY folks across the interwebs doing
things like passing instance methods to map, resulting in "big" tasks, and
slower-than-sequential parallelized code. Parallelizing "instance methods"
by passing them to map, w/o needing to wrangle with staticmethods and
globals, would be a GREAT feature! It'd just be as easy as:

    Pool.map(self.func, ls)

What do you think about this idea? This is something I'd be able to take
on, assuming I get a few core dev blessings...

On Thu, Oct 4, 2018 at 6:15 AM Nathaniel Smith <njs at pobox.com> wrote:

> On Wed, Oct 3, 2018 at 6:30 PM, Sean Harrington <seanharr11 at gmail.com>
> wrote:
> > with Pool(func_kwargs={"big_cache": big_cache}) as pool:
> >     pool.map(func, ls)
>
> I feel like it would be nicer to spell this:
>
> with Pool() as pool:
>     pool.map(functools.partial(func, big_cache=big_cache), ls)
>
> And this might also solve your problem, if pool.map is clever enough
> to only send the function object once to each worker?
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181012/003ab385/attachment.html>

From solipsis at pitrou.net  Fri Oct 12 08:48:47 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 12 Oct 2018 14:48:47 +0200
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
Message-ID: <20181012144847.7280967b@fsol>

On Fri, 12 Oct 2018 08:33:32 -0400
Sean Harrington <seanharr11 at gmail.com> wrote:
> Hi Nathaniel - this if this solution can be made performant, than I would
> be more than satisfied.
> 
> I think this would require removing "func" from the "task tuple", and
> storing the "func" "once per worker" somewhere globally (maybe a class
> attribute set post-fork?).
> 
> This also has the beneficial outcome of increasing general performance of
> Pool.map and friends. I've seen MANY folks across the interwebs doing
> things like passing instance methods to map, resulting in "big" tasks, and
> slower-than-sequential parallelized code. Parallelizing "instance methods"
> by passing them to map, w/o needing to wrangle with staticmethods and
> globals, would be a GREAT feature! It'd just be as easy as:
> 
>     Pool.map(self.func, ls)
> 
> What do you think about this idea? This is something I'd be able to take
> on, assuming I get a few core dev blessings...

Well, I'm not sure how it would work, so it's difficult to give an
opinion.  How do you plan to avoid passing "self"?  By caching (by
equality? by identity?)?  Something else?  But what happens if "self"
changed value (in the case of a mutable object) in the parent?  Do you
keep using the stale version in the child?  That would break
compatibility...

Regards

Antoine.

From seanharr11 at gmail.com  Fri Oct 12 09:17:59 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Fri, 12 Oct 2018 09:17:59 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <20181012144847.7280967b@fsol>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
Message-ID: <CAFbyWc1P-OFs9qWq6=tH8UpNKxmSEUbUse8nJpfDWMgDgJKcKA@mail.gmail.com>

The implementation details need to be flushed out, but agnostic of these,
do you believe this a valid solution to the initial problem? Do you also
see it as a beneficial optimization to Pool, given that we don't need to
store funcs/bound-methods/partials on the tasks themselves?

The latter concern about "what happens if `self` changed value in the
parent" is the same concern as "what happens if `func` changes in the
parent?" given the current implementation. This is an assumption that is
currently made with Pool.map_async(func, ls). If "func" changes in the
parent, there is no communication with the child. So one just needs to be
aware that calling "map_async(self.func, ls)" while the state of "self" is
changing, will not communicate changes to each worker. The state is frozen
when Pool.map is called, just as is the case now.

On Fri, Oct 12, 2018 at 9:07 AM Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Fri, 12 Oct 2018 08:33:32 -0400
> Sean Harrington <seanharr11 at gmail.com> wrote:
> > Hi Nathaniel - this if this solution can be made performant, than I would
> > be more than satisfied.
> >
> > I think this would require removing "func" from the "task tuple", and
> > storing the "func" "once per worker" somewhere globally (maybe a class
> > attribute set post-fork?).
> >
> > This also has the beneficial outcome of increasing general performance of
> > Pool.map and friends. I've seen MANY folks across the interwebs doing
> > things like passing instance methods to map, resulting in "big" tasks,
> and
> > slower-than-sequential parallelized code. Parallelizing "instance
> methods"
> > by passing them to map, w/o needing to wrangle with staticmethods and
> > globals, would be a GREAT feature! It'd just be as easy as:
> >
> >     Pool.map(self.func, ls)
> >
> > What do you think about this idea? This is something I'd be able to take
> > on, assuming I get a few core dev blessings...
>
> Well, I'm not sure how it would work, so it's difficult to give an
> opinion.  How do you plan to avoid passing "self"?  By caching (by
> equality? by identity?)?  Something else?  But what happens if "self"
> changed value (in the case of a mutable object) in the parent?  Do you
> keep using the stale version in the child?  That would break
> compatibility...
>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181012/fc0f7d8a/attachment.html>

From antoine at python.org  Fri Oct 12 09:24:18 2018
From: antoine at python.org (Antoine Pitrou)
Date: Fri, 12 Oct 2018 15:24:18 +0200
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc1P-OFs9qWq6=tH8UpNKxmSEUbUse8nJpfDWMgDgJKcKA@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAFbyWc1P-OFs9qWq6=tH8UpNKxmSEUbUse8nJpfDWMgDgJKcKA@mail.gmail.com>
Message-ID: <48b08a39-3d6a-3c4b-385e-fafe645f011a@python.org>


Le 12/10/2018 ? 15:17, Sean Harrington a ?crit?:
> The implementation details need to be flushed out, but agnostic of
> these, do you believe this a valid solution to the initial problem? Do
> you also see it as a beneficial optimization to Pool, given that we
> don't need to store funcs/bound-methods/partials on the tasks themselves?

I'm not sure, TBH.  I also think it may be better to leave this to
higher levels (for example Dask will intelligently distribute data on
workers and let you work with a kind of proxy object in the main
process, transfering data only when necessary).

> The latter concern about "what happens if `self` changed value in the
> parent" is the same concern as "what happens if `func` changes in the
> parent?" given the current implementation. This is an assumption that is
> currently made with Pool.map_async(func, ls). If "func" changes in the
> parent, there is no communication with the child. So one just needs to
> be aware that calling "map_async(self.func, ls)" while the state of
> "self" is changing, will not communicate changes to each worker. The
> state is frozen when Pool.map is called, just as is the case now.

If you cache "self" between pool.map calls, then the question is not
"what happens if self changes *during* a map() call" but "what happens
if self changes *between* two map() calls"?  While the former is
intuitively undefined, current users would expect the latter to have a
clear answer, which is: the latest version of self when map() is called
is taken into account.

Regards

Antoine.

From seanharr11 at gmail.com  Fri Oct 12 09:42:50 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Fri, 12 Oct 2018 09:42:50 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <48b08a39-3d6a-3c4b-385e-fafe645f011a@python.org>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAFbyWc1P-OFs9qWq6=tH8UpNKxmSEUbUse8nJpfDWMgDgJKcKA@mail.gmail.com>
 <48b08a39-3d6a-3c4b-385e-fafe645f011a@python.org>
Message-ID: <CAFbyWc2=eeGy2Y06unRJ6ua-ZSwBY45Z-4qfuwHVPsk2F5vZDA@mail.gmail.com>

I would contend that this is much more granular than Dask - this is just an
optimization of Pool.map() to avoid redundantly passing the same `func`
repeatedly, once per task, to each worker, with the primary goal of
eliminating redundant serialization of large-memory-footprinted Callables.
This is a different use case than Dask - I don't intend to approach the
shared memory or distributed computing realms.

And the second call to Pool.map would update the cached "self" as a part of
its initialization workflow, s.t. "the latest version of self when map() is
called is taken into account".

Do you see a difficulty in accomplishing the second behavior?

On Fri, Oct 12, 2018 at 9:25 AM Antoine Pitrou <antoine at python.org> wrote:

>
> Le 12/10/2018 ? 15:17, Sean Harrington a ?crit :
> > The implementation details need to be flushed out, but agnostic of
> > these, do you believe this a valid solution to the initial problem? Do
> > you also see it as a beneficial optimization to Pool, given that we
> > don't need to store funcs/bound-methods/partials on the tasks themselves?
>
> I'm not sure, TBH.  I also think it may be better to leave this to
> higher levels (for example Dask will intelligently distribute data on
> workers and let you work with a kind of proxy object in the main
> process, transfering data only when necessary).
>
> > The latter concern about "what happens if `self` changed value in the
> > parent" is the same concern as "what happens if `func` changes in the
> > parent?" given the current implementation. This is an assumption that is
> > currently made with Pool.map_async(func, ls). If "func" changes in the
> > parent, there is no communication with the child. So one just needs to
> > be aware that calling "map_async(self.func, ls)" while the state of
> > "self" is changing, will not communicate changes to each worker. The
> > state is frozen when Pool.map is called, just as is the case now.
>
> If you cache "self" between pool.map calls, then the question is not
> "what happens if self changes *during* a map() call" but "what happens
> if self changes *between* two map() calls"?  While the former is
> intuitively undefined, current users would expect the latter to have a
> clear answer, which is: the latest version of self when map() is called
> is taken into account.
>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181012/c1c83eb7/attachment-0001.html>

From solipsis at pitrou.net  Fri Oct 12 10:41:04 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 12 Oct 2018 16:41:04 +0200
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAFbyWc1P-OFs9qWq6=tH8UpNKxmSEUbUse8nJpfDWMgDgJKcKA@mail.gmail.com>
 <48b08a39-3d6a-3c4b-385e-fafe645f011a@python.org>
 <CAFbyWc2=eeGy2Y06unRJ6ua-ZSwBY45Z-4qfuwHVPsk2F5vZDA@mail.gmail.com>
Message-ID: <20181012164104.54ee303e@fsol>

On Fri, 12 Oct 2018 09:42:50 -0400
Sean Harrington <seanharr11 at gmail.com> wrote:
> I would contend that this is much more granular than Dask - this is just an
> optimization of Pool.map() to avoid redundantly passing the same `func`
> repeatedly, once per task, to each worker, with the primary goal of
> eliminating redundant serialization of large-memory-footprinted Callables.
> This is a different use case than Dask - I don't intend to approach the
> shared memory or distributed computing realms.
> 
> And the second call to Pool.map would update the cached "self" as a part of
> its initialization workflow, s.t. "the latest version of self when map() is
> called is taken into account".

I still don't understand how that works.  If you "updated the cached
self", then surely you must transmit it to the child, right?

Regards

Antoine.



From seanharr11 at gmail.com  Fri Oct 12 10:49:28 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Fri, 12 Oct 2018 10:49:28 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <20181012164104.54ee303e@fsol>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAFbyWc1P-OFs9qWq6=tH8UpNKxmSEUbUse8nJpfDWMgDgJKcKA@mail.gmail.com>
 <48b08a39-3d6a-3c4b-385e-fafe645f011a@python.org>
 <CAFbyWc2=eeGy2Y06unRJ6ua-ZSwBY45Z-4qfuwHVPsk2F5vZDA@mail.gmail.com>
 <20181012164104.54ee303e@fsol>
Message-ID: <CAFbyWc3X1-iYDdrPc7uiQWivezP6Z9nWr12TSLmV9=1vRxs1Kw@mail.gmail.com>

Yes - ?func? (and ?self? which func is bound to) would be copied to each
child worker process, where they are stored and applied to each element of
the iterable being mapped over.
On Fri, Oct 12, 2018 at 10:41 AM Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Fri, 12 Oct 2018 09:42:50 -0400
> Sean Harrington <seanharr11 at gmail.com> wrote:
> > I would contend that this is much more granular than Dask - this is just
> an
> > optimization of Pool.map() to avoid redundantly passing the same `func`
> > repeatedly, once per task, to each worker, with the primary goal of
> > eliminating redundant serialization of large-memory-footprinted
> Callables.
> > This is a different use case than Dask - I don't intend to approach the
> > shared memory or distributed computing realms.
> >
> > And the second call to Pool.map would update the cached "self" as a part
> of
> > its initialization workflow, s.t. "the latest version of self when map()
> is
> > called is taken into account".
>
> I still don't understand how that works.  If you "updated the cached
> self", then surely you must transmit it to the child, right?
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181012/ac9abaa6/attachment.html>

From antoine at python.org  Fri Oct 12 10:54:48 2018
From: antoine at python.org (Antoine Pitrou)
Date: Fri, 12 Oct 2018 16:54:48 +0200
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc3X1-iYDdrPc7uiQWivezP6Z9nWr12TSLmV9=1vRxs1Kw@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAFbyWc1P-OFs9qWq6=tH8UpNKxmSEUbUse8nJpfDWMgDgJKcKA@mail.gmail.com>
 <48b08a39-3d6a-3c4b-385e-fafe645f011a@python.org>
 <CAFbyWc2=eeGy2Y06unRJ6ua-ZSwBY45Z-4qfuwHVPsk2F5vZDA@mail.gmail.com>
 <20181012164104.54ee303e@fsol>
 <CAFbyWc3X1-iYDdrPc7uiQWivezP6Z9nWr12TSLmV9=1vRxs1Kw@mail.gmail.com>
Message-ID: <d5b35db4-b8d0-425e-0e4c-03c1960bb91c@python.org>


Le 12/10/2018 ? 16:49, Sean Harrington a ?crit?:
> Yes - ?func? (and ?self? which func is bound to) would be copied to each
> child worker process, where they are stored and applied to each element
> of the iterable being mapped over.

Only if it has changed, then, right?

I suspect that would work, but it will break compatibility in some cases
(think of a mutable object that hasn't defined equality - so it defaults
to identity).  It's also introducing a complication in the API which
people didn't have to think of before.

The fact that you're doing all this in order to eschew global variables
for global resources doesn't warm me much to the idea.  Unless other
core developers are enthusiastic I'm not willing to integrate such a change.

Regards

Antoine.


> On Fri, Oct 12, 2018 at 10:41 AM Antoine Pitrou <solipsis at pitrou.net
> <mailto:solipsis at pitrou.net>> wrote:
> 
>     On Fri, 12 Oct 2018 09:42:50 -0400
>     Sean Harrington <seanharr11 at gmail.com <mailto:seanharr11 at gmail.com>>
>     wrote:
>     > I would contend that this is much more granular than Dask - this
>     is just an
>     > optimization of Pool.map() to avoid redundantly passing the same
>     `func`
>     > repeatedly, once per task, to each worker, with the primary goal of
>     > eliminating redundant serialization of large-memory-footprinted
>     Callables.
>     > This is a different use case than Dask - I don't intend to
>     approach the
>     > shared memory or distributed computing realms.
>     >
>     > And the second call to Pool.map would update the cached "self" as
>     a part of
>     > its initialization workflow, s.t. "the latest version of self when
>     map() is
>     > called is taken into account".
> 
>     I still don't understand how that works.? If you "updated the cached
>     self", then surely you must transmit it to the child, right?
> 
>     Regards
> 
>     Antoine.
> 
> 
>     _______________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     https://mail.python.org/mailman/listinfo/python-dev
>     Unsubscribe:
>     https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com
> 

From status at bugs.python.org  Fri Oct 12 12:10:04 2018
From: status at bugs.python.org (Python tracker)
Date: Fri, 12 Oct 2018 18:10:04 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20181012161004.4103457B51@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2018-10-05 - 2018-10-12)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6824 (+10)
  closed 39893 (+49)
  total  46717 (+59)

Open issues with patches: 2721 


Issues opened (38)
==================

#34908: netrc parsing is overly strict
https://bugs.python.org/issue34908  opened by ianwremmel

#34912: Update overflow checks in resize_buffer
https://bugs.python.org/issue34912  opened by Windson Yang

#34914: Clarify text encoding used to enable UTF-8 mode
https://bugs.python.org/issue34914  opened by ncoghlan

#34915: LWPCookieJar.save() creates *.lwp file in 644 mode
https://bugs.python.org/issue34915  opened by aleskva

#34916: include sqlite-3.25+ (with window functions)
https://bugs.python.org/issue34916  opened by Big Stone

#34918: Python 3 tkinter measurement problem
https://bugs.python.org/issue34918  opened by Ackbach

#34919: Crash caused by certain characters in a string
https://bugs.python.org/issue34919  opened by cwickens

#34920: PYTHONWARNINGS entries are escaped
https://bugs.python.org/issue34920  opened by nikratio

#34922: hashlib segmentation fault
https://bugs.python.org/issue34922  opened by shuoz

#34924: inspect.signature isn't aware that types.MethodType can wrap a
https://bugs.python.org/issue34924  opened by bup

#34927: Tkinter-related segfault on macOS (regression between 3.7.0 an
https://bugs.python.org/issue34927  opened by aivarannamaa

#34930: sha1module: Switch sha1 implementation to sha1dc/hardened sha1
https://bugs.python.org/issue34930  opened by antoine.pietri

#34931: os.path.splitext with more dots
https://bugs.python.org/issue34931  opened by xnovakj

#34932: Add macOS TCP_KEEPALIVE to available socket options
https://bugs.python.org/issue34932  opened by llawall

#34934: Consider making Windows select.select interruptible using WSAE
https://bugs.python.org/issue34934  opened by ondrej.kutal

#34936: tkinter.Spinbox.selection_element() raises TclError
https://bugs.python.org/issue34936  opened by j-4321-i

#34938: Fix mimetype.init() to account for from import
https://bugs.python.org/issue34938  opened by YoSTEALTH

#34939: Possibly spurious SyntaxError: annotated name can't be global
https://bugs.python.org/issue34939  opened by rohanpadhye

#34941: xml.etree.ElementTree findall() fails when using custom TreeBu
https://bugs.python.org/issue34941  opened by jackjansen

#34943: sched cancel (wrong item removed from queue)
https://bugs.python.org/issue34943  opened by ajneu

#34944: Update _dirnameW to accept long path names
https://bugs.python.org/issue34944  opened by jopamer

#34945: regression with ./python -m test and pdb
https://bugs.python.org/issue34945  opened by matrixise

#34946: inspect.getcallargs is marked as deprecated in documentation, 
https://bugs.python.org/issue34946  opened by chrisjbremner

#34947: inspect.getclosurevars() does not get all globals
https://bugs.python.org/issue34947  opened by azag0

#34948: Document __warningregister__
https://bugs.python.org/issue34948  opened by steven.daprano

#34949: ntpath.abspath no longer uses normpath
https://bugs.python.org/issue34949  opened by rhwlo

#34950: Parse trusted and signature information from X509 certificate
https://bugs.python.org/issue34950  opened by famendola

#34951: cookielib/cookiejar cookies' Expires date parse fails with lon
https://bugs.python.org/issue34951  opened by alb_moral

#34952: Problems with the warningregistry() decorator in Lib/unittest/
https://bugs.python.org/issue34952  opened by serhiy.storchaka

#34953: Implement `mmap.mmap.__repr__`
https://bugs.python.org/issue34953  opened by cool-RR

#34956: _tkinter built on macOS 10.14 does not link to Tcl and Tk in /
https://bugs.python.org/issue34956  opened by wordtech

#34957: Segementation faults on ARM and ARM64
https://bugs.python.org/issue34957  opened by stefanrink at yahoo.com

#34958: urllib.error.HTTPError.fp is not closed when error is finalize
https://bugs.python.org/issue34958  opened by puxlit

#34960: macOS builds expose stack_size option in LINKEDFOORSHARED and 
https://bugs.python.org/issue34960  opened by xzcvczx

#34961: Global scoping when shadowing local names in class definitions
https://bugs.python.org/issue34961  opened by multun

#34963: String representation for types created with typing.NewType(??
https://bugs.python.org/issue34963  opened by fish2000

#34965: Python on Docker container using flask is going down after som
https://bugs.python.org/issue34965  opened by sri_spl

#34966: Pydoc: better support of method aliases
https://bugs.python.org/issue34966  opened by serhiy.storchaka



Most recent 15 issues with no replies (15)
==========================================

#34966: Pydoc: better support of method aliases
https://bugs.python.org/issue34966

#34963: String representation for types created with typing.NewType(??
https://bugs.python.org/issue34963

#34958: urllib.error.HTTPError.fp is not closed when error is finalize
https://bugs.python.org/issue34958

#34950: Parse trusted and signature information from X509 certificate
https://bugs.python.org/issue34950

#34949: ntpath.abspath no longer uses normpath
https://bugs.python.org/issue34949

#34944: Update _dirnameW to accept long path names
https://bugs.python.org/issue34944

#34938: Fix mimetype.init() to account for from import
https://bugs.python.org/issue34938

#34934: Consider making Windows select.select interruptible using WSAE
https://bugs.python.org/issue34934

#34931: os.path.splitext with more dots
https://bugs.python.org/issue34931

#34924: inspect.signature isn't aware that types.MethodType can wrap a
https://bugs.python.org/issue34924

#34918: Python 3 tkinter measurement problem
https://bugs.python.org/issue34918

#34915: LWPCookieJar.save() creates *.lwp file in 644 mode
https://bugs.python.org/issue34915

#34912: Update overflow checks in resize_buffer
https://bugs.python.org/issue34912

#34898: add mtime argument to gzip.compress
https://bugs.python.org/issue34898

#34897: distutils test errors when CXX is not set
https://bugs.python.org/issue34897



Most recent 15 issues waiting for review (15)
=============================================

#34966: Pydoc: better support of method aliases
https://bugs.python.org/issue34966

#34963: String representation for types created with typing.NewType(??
https://bugs.python.org/issue34963

#34944: Update _dirnameW to accept long path names
https://bugs.python.org/issue34944

#34941: xml.etree.ElementTree findall() fails when using custom TreeBu
https://bugs.python.org/issue34941

#34936: tkinter.Spinbox.selection_element() raises TclError
https://bugs.python.org/issue34936

#34932: Add macOS TCP_KEEPALIVE to available socket options
https://bugs.python.org/issue34932

#34922: hashlib segmentation fault
https://bugs.python.org/issue34922

#34903: strptime %d handling of single digit day of month
https://bugs.python.org/issue34903

#34901: Missing isolated (-I) flag in sys.flags table
https://bugs.python.org/issue34901

#34898: add mtime argument to gzip.compress
https://bugs.python.org/issue34898

#34897: distutils test errors when CXX is not set
https://bugs.python.org/issue34897

#34876: Python3.8 changes how decorators are traced
https://bugs.python.org/issue34876

#34866: CGI DOS vulnerability via long post list
https://bugs.python.org/issue34866

#34861: Improve cProfile standard output
https://bugs.python.org/issue34861

#34856: Make the repr of lambda contain signature and body expression.
https://bugs.python.org/issue34856



Top 10 most discussed issues (10)
=================================

#34839: doctest: Change example under warnings section
https://bugs.python.org/issue34839  12 msgs

#34751: Hash collisions for tuples
https://bugs.python.org/issue34751  10 msgs

#34922: hashlib segmentation fault
https://bugs.python.org/issue34922  10 msgs

#25592: distutils docs: data_files always uses sys.prefix
https://bugs.python.org/issue25592   8 msgs

#34893: Add 2to3 fixer to change send and recv methods of socket objec
https://bugs.python.org/issue34893   8 msgs

#33729: Hashlib/blake2* missing 'data' keyword argument
https://bugs.python.org/issue33729   7 msgs

#34927: Tkinter-related segfault on macOS (regression between 3.7.0 an
https://bugs.python.org/issue34927   7 msgs

#34939: Possibly spurious SyntaxError: annotated name can't be global
https://bugs.python.org/issue34939   6 msgs

#34957: Segementation faults on ARM and ARM64
https://bugs.python.org/issue34957   6 msgs

#34960: macOS builds expose stack_size option in LINKEDFOORSHARED and 
https://bugs.python.org/issue34960   6 msgs



Issues closed (45)
==================

#20304: Argument Clinic: char convertor should use default values of t
https://bugs.python.org/issue20304  closed by taleinat

#23596: gzip argparse interface
https://bugs.python.org/issue23596  closed by matrixise

#31715: Add mimetype for extension .mjs
https://bugs.python.org/issue31715  closed by asvetlov

#32174: nonASCII punctuation characters can not display in python363.c
https://bugs.python.org/issue32174  closed by steve.dower

#32680: smtplib SMTP instances missing a default sock attribute
https://bugs.python.org/issue32680  closed by giampaolo.rodola

#32962: test_gdb fails in debug build with `-mcet -fcf-protection -O0`
https://bugs.python.org/issue32962  closed by vstinner

#33643: Mock functions with autospec STILL don't support assert_called
https://bugs.python.org/issue33643  closed by matrixise

#34334: QueueHandler logs exc_info twice
https://bugs.python.org/issue34334  closed by ned.deily

#34383: asyncio passes SSL certificate error to loop.call_exception_ha
https://bugs.python.org/issue34383  closed by asvetlov

#34445: asyncio support in doctests
https://bugs.python.org/issue34445  closed by asvetlov

#34630: Don't log ssl cert errors in asyncio
https://bugs.python.org/issue34630  closed by asvetlov

#34758: http.server module sets incorrect mimetype for WebAssembly fil
https://bugs.python.org/issue34758  closed by asvetlov

#34769: _asyncgen_finalizer_hook running in wrong thread
https://bugs.python.org/issue34769  closed by yselivanov

#34825: Add more entries to os module to pathlib reference table
https://bugs.python.org/issue34825  closed by asvetlov

#34829: Add missing selection_ methods to tkinter Spinbox
https://bugs.python.org/issue34829  closed by serhiy.storchaka

#34849: Drop logging when asyncio waits in selector.select()
https://bugs.python.org/issue34849  closed by asvetlov

#34870: Core dump when Python VSCode debugger is attached
https://bugs.python.org/issue34870  closed by terry.reedy

#34872: investigate task/future cancellation in asynciomodule.c
https://bugs.python.org/issue34872  closed by yselivanov

#34899: Possible assertion failure due to int_from_bytes_impl()
https://bugs.python.org/issue34899  closed by ZackerySpytz

#34900: unittest subTests() fails when called from debug()
https://bugs.python.org/issue34900  closed by berker.peksag

#34902: Azure pipelines PR build fails with "Unexpected vmImage 'vs201
https://bugs.python.org/issue34902  closed by steve.dower

#34905: Cannot assign memoryview values from array.array
https://bugs.python.org/issue34905  closed by skrah

#34906: Fix typo in the documentation
https://bugs.python.org/issue34906  closed by mdk

#34907: calculation not working properly
https://bugs.python.org/issue34907  closed by matrixise

#34909: StrEnum subclasses cannot be created
https://bugs.python.org/issue34909  closed by ethan.furman

#34910: PyObject_Print() doesn't always return -1 on error
https://bugs.python.org/issue34910  closed by serhiy.storchaka

#34911: Allow Cookies for Secure WebSockets
https://bugs.python.org/issue34911  closed by asvetlov

#34913: Document gzip command line interface
https://bugs.python.org/issue34913  closed by mdk

#34917: add time decorator to timeit.py
https://bugs.python.org/issue34917  closed by ezio.melotti

#34921: NoReturn not allowed by get_type_hints when future import anno
https://bugs.python.org/issue34921  closed by levkivskyi

#34923: Decimal Multiplication problems: Wrong solution
https://bugs.python.org/issue34923  closed by Shadow Raven

#34925: 25% speed-up to common case for bisect()
https://bugs.python.org/issue34925  closed by rhettinger

#34926: Allow querying a Path's mime-type
https://bugs.python.org/issue34926  closed by xtreak

#34928: string method .upper() converts '??' to 'SS' instead of '???'
https://bugs.python.org/issue34928  closed by xtreak

#34929: Extract asyncio sendfile tests into a separate test file
https://bugs.python.org/issue34929  closed by asvetlov

#34933: json.dumps serializes double quotes incorrectly
https://bugs.python.org/issue34933  closed by serhiy.storchaka

#34935: Misleading error message in str.decode()
https://bugs.python.org/issue34935  closed by ezio.melotti

#34937: Extract asyncio tests for sock_*() methods into a separate tes
https://bugs.python.org/issue34937  closed by asvetlov

#34940: Possible assertion failure due to _check_for_legacy_statements
https://bugs.python.org/issue34940  closed by serhiy.storchaka

#34942: Add an FAQ note about floating point representation
https://bugs.python.org/issue34942  closed by xtreak

#34954: Getting an email's subject is error-prone
https://bugs.python.org/issue34954  closed by r.david.murray

#34955: passing a dict to bytes() gives unhelpful error message
https://bugs.python.org/issue34955  closed by rhettinger

#34959: Are the tests optional on travis?
https://bugs.python.org/issue34959  closed by matrixise

#34962: make doctest does not pass :/
https://bugs.python.org/issue34962  closed by mdk

#34964: Make Tkinter sources more readable by adding blank lines
https://bugs.python.org/issue34964  closed by serhiy.storchaka

From njs at pobox.com  Fri Oct 12 15:34:17 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 12 Oct 2018 12:34:17 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <20181012144847.7280967b@fsol>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
Message-ID: <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>

On Fri, Oct 12, 2018, 06:09 Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Fri, 12 Oct 2018 08:33:32 -0400
> Sean Harrington <seanharr11 at gmail.com> wrote:
> > Hi Nathaniel - this if this solution can be made performant, than I would
> > be more than satisfied.
> >
> > I think this would require removing "func" from the "task tuple", and
> > storing the "func" "once per worker" somewhere globally (maybe a class
> > attribute set post-fork?).
> >
> > This also has the beneficial outcome of increasing general performance of
> > Pool.map and friends. I've seen MANY folks across the interwebs doing
> > things like passing instance methods to map, resulting in "big" tasks,
> and
> > slower-than-sequential parallelized code. Parallelizing "instance
> methods"
> > by passing them to map, w/o needing to wrangle with staticmethods and
> > globals, would be a GREAT feature! It'd just be as easy as:
> >
> >     Pool.map(self.func, ls)
> >
> > What do you think about this idea? This is something I'd be able to take
> > on, assuming I get a few core dev blessings...
>
> Well, I'm not sure how it would work, so it's difficult to give an
> opinion.  How do you plan to avoid passing "self"?  By caching (by
> equality? by identity?)?  Something else?  But what happens if "self"
> changed value (in the case of a mutable object) in the parent?  Do you
> keep using the stale version in the child?  That would break
> compatibility...
>

I was just suggesting that within a single call to Pool.map, it would be
reasonable optimization to only send the fn once to each worker. So e.g. if
you have 5 workers and 1000 items, you'd only pickle fn 5 times, rather
than 1000 times like we do now. I wouldn't want to get any fancier than
that with caching data between different map calls or anything.

Of course even this may turn out to be too complicated to implement in a
reasonable way, since it would require managing some extra state on the
workers. But semantically it would be purely an optimization of current
semantics.

-n

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181012/814918c5/attachment.html>

From vadmium+py at gmail.com  Fri Oct 12 19:03:20 2018
From: vadmium+py at gmail.com (Martin Panter)
Date: Fri, 12 Oct 2018 23:03:20 +0000
Subject: [Python-Dev] [Python-checkins] bpo-34203: FAQ now recommends
 python 3.x over 2.x (GH-9796)
In-Reply-To: <125297d6-545f-3052-0900-9df2072f0057@trueblade.com>
References: <42Wj0P517lzFrcx@mail.python.org>
 <125297d6-545f-3052-0900-9df2072f0057@trueblade.com>
Message-ID: <CA+eR4cFzge7eZnMEDZ7v20MURUD0i853ZyMQHDSriuZy8D3eGg@mail.gmail.com>

On 12/10/2018, Eric V. Smith <eric at trueblade.com> wrote:
> On 10/12/2018 5:17 AM, Tal Einat wrote:
>
>>   The latest stable releases can always be found on the `Python download page
>> -<https://www.python.org/downloads/>`_.  There are two recommended production-ready
>> -versions at this point in time, because at the moment there are two branches of
>> -stable releases: 2.x and 3.x.  Python 3.x may be less useful than 2.x, since
>> -currently there is more third party software available for Python 2 than for
>> -Python 3.  Python 2 code will generally not run unchanged in Python 3.
>> +<https://www.python.org/downloads/>`_.  There are two production-ready version
>> +of Python: 2.x and 3.x, but the recommended one at this times is Python 3.x.
>
> This should be "time", not "times". I'd fix it, but I'm unsure if this
> is being backported or not, and I don't want to mess up any merges
> before they're done.

Or just remove ?at this time[s]?; it doesn?t add much.

Also, ?two . . . version? should be changed back to plural: ?two . . .
versions?.

> I do think this should backported to 3.7, 3.6, and 2.7.
>
>> +Although Python 2.x is still widely used, `it will not be
>> +maintained after January 1, 2020
>> <https://www.python.org/dev/peps/pep-0373/>`_.
>> +Python 2.x was known for having more third-party libraries available,
>> however,
>> +by the time of this writing, most of the widely used libraries support
>> Python 3.x,
>
> Should probably be "at the time of this writing".
>
>> +and some are even dropping the Python 2.x support.
>
> And this would read better as "are even dropping support for Python 2.x".

From eric at trueblade.com  Fri Oct 12 20:05:33 2018
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 12 Oct 2018 20:05:33 -0400
Subject: [Python-Dev] [Python-checkins] bpo-34203: FAQ now recommends
 python 3.x over 2.x (GH-9796)
In-Reply-To: <CA+eR4cFzge7eZnMEDZ7v20MURUD0i853ZyMQHDSriuZy8D3eGg@mail.gmail.com>
References: <42Wj0P517lzFrcx@mail.python.org>
 <125297d6-545f-3052-0900-9df2072f0057@trueblade.com>
 <CA+eR4cFzge7eZnMEDZ7v20MURUD0i853ZyMQHDSriuZy8D3eGg@mail.gmail.com>
Message-ID: <9C0147D2-E70D-4C08-93C3-D40DF4DB51C6@trueblade.com>


> On Oct 12, 2018, at 7:03 PM, Martin Panter <vadmium+py at gmail.com> wrote:
> 
>> On 12/10/2018, Eric V. Smith <eric at trueblade.com> wrote:
>>> On 10/12/2018 5:17 AM, Tal Einat wrote:
>>> 
>>>  The latest stable releases can always be found on the `Python download page
>>> -<https://www.python.org/downloads/>`_.  There are two recommended production-ready
>>> -versions at this point in time, because at the moment there are two branches of
>>> -stable releases: 2.x and 3.x.  Python 3.x may be less useful than 2.x, since
>>> -currently there is more third party software available for Python 2 than for
>>> -Python 3.  Python 2 code will generally not run unchanged in Python 3.
>>> +<https://www.python.org/downloads/>`_.  There are two production-ready version
>>> +of Python: 2.x and 3.x, but the recommended one at this times is Python 3.x.
>> 
>> This should be "time", not "times". I'd fix it, but I'm unsure if this
>> is being backported or not, and I don't want to mess up any merges
>> before they're done.
> 
> Or just remove ?at this time[s]?; it doesn?t add much.
> 
> Also, ?two . . . version? should be changed back to plural: ?two . . .
> versions?.

See https://github.com/python/cpython/pull/9821 for some updates. 

Eric 

> 
>> I do think this should backported to 3.7, 3.6, and 2.7.
>> 
>>> +Although Python 2.x is still widely used, `it will not be
>>> +maintained after January 1, 2020
>>> <https://www.python.org/dev/peps/pep-0373/>`_.
>>> +Python 2.x was known for having more third-party libraries available,
>>> however,
>>> +by the time of this writing, most of the widely used libraries support
>>> Python 3.x,
>> 
>> Should probably be "at the time of this writing".
>> 
>>> +and some are even dropping the Python 2.x support.
>> 
>> And this would read better as "are even dropping support for Python 2.x".
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181012/ce803a1f/attachment.html>

From nad at python.org  Sat Oct 13 17:40:00 2018
From: nad at python.org (Ned Deily)
Date: Sat, 13 Oct 2018 17:40:00 -0400
Subject: [Python-Dev] [RELEASE] Python 3.7.1rc2 and 3.6.7rc2 now available
 for testing
Message-ID: <65B2DFC8-F64D-4E48-B33F-9AE88BC440F4@python.org>

Python 3.7.1rc2 and 3.6.7rc2 are now available. 3.7.1rc2 is a release
preview of the first maintenance release of Python 3.7, the latest
feature release of Python. 3.6.7rc2 is a release preview of the next
maintenance release of Python 3.6, the previous feature release of
Python. Assuming no further critical problems are found prior to
2018-10-20, no code changes are planned between these release
candidates and the final releases. These release candidates are
intended to give you the opportunity to test the new security and bug
fixes in 3.7.1 and 3.6.7. We strongly encourage you to test your
projects and report issues found to bugs.python.org as soon as
possible. Please keep in mind that these are preview releases and,
thus, their use is not recommended for production environments.

You can find these releases and more information here:
    https://www.python.org/downloads/release/python-371rc2/
    https://www.python.org/downloads/release/python-367rc2/

--
  Ned Deily
  nad at python.org -- []


From ja.py at farowl.co.uk  Sun Oct 14 14:51:02 2018
From: ja.py at farowl.co.uk (Jeff Allen)
Date: Sun, 14 Oct 2018 19:51:02 +0100
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <20181009230605.GV3817@ando.pearwood.info>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <20181009230605.GV3817@ando.pearwood.info>
Message-ID: <b69bf75c-2e90-e344-de46-eb6e3701a90c@farowl.co.uk>


On 10/10/2018 00:06, Steven D'Aprano wrote:
> On Tue, Oct 09, 2018 at 09:37:48AM -0700, Jeff Hardy wrote:
>
>> ...
>>  From an alternative implementation point of view, CPython's behaviour
>> *is* the spec. Practicality beats purity and all that.
> Are you speaking on behalf of all authors of alternate implementations,
> or even of some of them?
>
> It certainly is not true that CPython's behaviour "is" the spec. PyPy
> keeps a list of CPython behaviour they don't match, either because they
> choose not to for other reasons, or because they believe that the
> CPython behaviour is buggy. I daresay IronPython and Jython have
> similar.
While agreeing with the principle, unless it is one of the fundamental 
differences (GC, GIL), Jython usually lets practicality beat purity. 
When faced with a certain combination of objects, one has to do 
something, and it is least surprising to do what CPython does. It's also 
easier than keeping a record.

Rarely, we manage to exceed CPython (in consistency or coverage) by a 
tiny amount.

Jeff Allen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181014/a4fb69e7/attachment.html>

From seanharr11 at gmail.com  Tue Oct 16 08:06:16 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Tue, 16 Oct 2018 08:06:16 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
Message-ID: <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>

@Nataniel this is what I am suggesting as well. No cacheing - just storing
the `fn` on each worker, rather than pickling it for each item in our
iterable.

As long as we store the `fn` post-fork on the worker process (perhaps as
global), subsequent calls to Pool.map shouldn't be effected (referencing
Antoine's & Michael's points that "multiprocessing encapsulates each
subprocesses globals in a separate namespace").

@Antoine - I'm making an effort to take everything you've said into
consideration here.  My initial PR and talk
<https://www.youtube.com/watch?v=DH0JVSXvxu0> was intended to shed light on
a couple of pitfalls that I often see Python end-users encounter with Pool.
Moving beyond my naive first attempt, and the onslaught of deserved
criticism, it seems that we have an opportunity here: No changes to the
interface, just an optimization to reduce the frequency of pickling.

Raymond Hettinger may also be interested in this optimization, as he speaks
(with great analogies) about different ways you can misuse concurrency in
Python <https://www.youtube.com/watch?v=9zinZmE3Ogk>. This would address
one of the pitfalls that he outlines: the "size of the
serialized/deserialized data".

Is this an optimization that either of you would be willing to review, and
accept, if I find there is a *reasonable way* to implement it?


On Fri, Oct 12, 2018 at 3:40 PM Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, Oct 12, 2018, 06:09 Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> On Fri, 12 Oct 2018 08:33:32 -0400
>> Sean Harrington <seanharr11 at gmail.com> wrote:
>> > Hi Nathaniel - this if this solution can be made performant, than I
>> would
>> > be more than satisfied.
>> >
>> > I think this would require removing "func" from the "task tuple", and
>> > storing the "func" "once per worker" somewhere globally (maybe a class
>> > attribute set post-fork?).
>> >
>> > This also has the beneficial outcome of increasing general performance
>> of
>> > Pool.map and friends. I've seen MANY folks across the interwebs doing
>> > things like passing instance methods to map, resulting in "big" tasks,
>> and
>> > slower-than-sequential parallelized code. Parallelizing "instance
>> methods"
>> > by passing them to map, w/o needing to wrangle with staticmethods and
>> > globals, would be a GREAT feature! It'd just be as easy as:
>> >
>> >     Pool.map(self.func, ls)
>> >
>> > What do you think about this idea? This is something I'd be able to take
>> > on, assuming I get a few core dev blessings...
>>
>> Well, I'm not sure how it would work, so it's difficult to give an
>> opinion.  How do you plan to avoid passing "self"?  By caching (by
>> equality? by identity?)?  Something else?  But what happens if "self"
>> changed value (in the case of a mutable object) in the parent?  Do you
>> keep using the stale version in the child?  That would break
>> compatibility...
>>
>
> I was just suggesting that within a single call to Pool.map, it would be
> reasonable optimization to only send the fn once to each worker. So e.g. if
> you have 5 workers and 1000 items, you'd only pickle fn 5 times, rather
> than 1000 times like we do now. I wouldn't want to get any fancier than
> that with caching data between different map calls or anything.
>
> Of course even this may turn out to be too complicated to implement in a
> reasonable way, since it would require managing some extra state on the
> workers. But semantically it would be purely an optimization of current
> semantics.
>
> -n
>
>> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181016/d70f7635/attachment.html>

From michael.selik at gmail.com  Tue Oct 16 09:27:32 2018
From: michael.selik at gmail.com (Michael Selik)
Date: Tue, 16 Oct 2018 06:27:32 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
Message-ID: <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>

Would this change the other pool method behavior in some way if the user,
for whatever reason, mixed techniques?

imap_unordered will only block when nexting the generator. If the user
mingles nexting that generator with, say, apply_async, could the change
you're proposing have some side-effect?

On Tue, Oct 16, 2018, 5:09 AM Sean Harrington <seanharr11 at gmail.com> wrote:

> @Nataniel this is what I am suggesting as well. No cacheing - just storing
> the `fn` on each worker, rather than pickling it for each item in our
> iterable.
>
> As long as we store the `fn` post-fork on the worker process (perhaps as
> global), subsequent calls to Pool.map shouldn't be effected (referencing
> Antoine's & Michael's points that "multiprocessing encapsulates each
> subprocesses globals in a separate namespace").
>
> @Antoine - I'm making an effort to take everything you've said into
> consideration here.  My initial PR and talk
> <https://www.youtube.com/watch?v=DH0JVSXvxu0> was intended to shed light
> on a couple of pitfalls that I often see Python end-users encounter with
> Pool. Moving beyond my naive first attempt, and the onslaught of deserved
> criticism, it seems that we have an opportunity here: No changes to the
> interface, just an optimization to reduce the frequency of pickling.
>
> Raymond Hettinger may also be interested in this optimization, as he
> speaks (with great analogies) about different ways you can misuse
> concurrency in Python <https://www.youtube.com/watch?v=9zinZmE3Ogk>. This
> would address one of the pitfalls that he outlines: the "size of the
> serialized/deserialized data".
>
> Is this an optimization that either of you would be willing to review, and
> accept, if I find there is a *reasonable way* to implement it?
>
>
> On Fri, Oct 12, 2018 at 3:40 PM Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Fri, Oct 12, 2018, 06:09 Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>>> On Fri, 12 Oct 2018 08:33:32 -0400
>>> Sean Harrington <seanharr11 at gmail.com> wrote:
>>> > Hi Nathaniel - this if this solution can be made performant, than I
>>> would
>>> > be more than satisfied.
>>> >
>>> > I think this would require removing "func" from the "task tuple", and
>>> > storing the "func" "once per worker" somewhere globally (maybe a class
>>> > attribute set post-fork?).
>>> >
>>> > This also has the beneficial outcome of increasing general performance
>>> of
>>> > Pool.map and friends. I've seen MANY folks across the interwebs doing
>>> > things like passing instance methods to map, resulting in "big" tasks,
>>> and
>>> > slower-than-sequential parallelized code. Parallelizing "instance
>>> methods"
>>> > by passing them to map, w/o needing to wrangle with staticmethods and
>>> > globals, would be a GREAT feature! It'd just be as easy as:
>>> >
>>> >     Pool.map(self.func, ls)
>>> >
>>> > What do you think about this idea? This is something I'd be able to
>>> take
>>> > on, assuming I get a few core dev blessings...
>>>
>>> Well, I'm not sure how it would work, so it's difficult to give an
>>> opinion.  How do you plan to avoid passing "self"?  By caching (by
>>> equality? by identity?)?  Something else?  But what happens if "self"
>>> changed value (in the case of a mutable object) in the parent?  Do you
>>> keep using the stale version in the child?  That would break
>>> compatibility...
>>>
>>
>> I was just suggesting that within a single call to Pool.map, it would be
>> reasonable optimization to only send the fn once to each worker. So e.g. if
>> you have 5 workers and 1000 items, you'd only pickle fn 5 times, rather
>> than 1000 times like we do now. I wouldn't want to get any fancier than
>> that with caching data between different map calls or anything.
>>
>> Of course even this may turn out to be too complicated to implement in a
>> reasonable way, since it would require managing some extra state on the
>> workers. But semantically it would be purely an optimization of current
>> semantics.
>>
>> -n
>>
>>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com
>>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181016/549cd6e3/attachment.html>

From jdhardy at gmail.com  Tue Oct 16 12:35:51 2018
From: jdhardy at gmail.com (Jeff Hardy)
Date: Tue, 16 Oct 2018 09:35:51 -0700
Subject: [Python-Dev] Arbitrary non-identifier string keys when using
 **kwargs
In-Reply-To: <b69bf75c-2e90-e344-de46-eb6e3701a90c@farowl.co.uk>
References: <20181004085633.GP21220@ando.pearwood.info>
 <pp4lpg$8us$1@blaine.gmane.org>
 <CAP1=2W6CrF6-mP3ZHkHWwniEK+6bYn1XvMzeGJtyGTGs9-SeyA@mail.gmail.com>
 <CALGmxEKWXUuyEPZbkJb0EnGYmNHEc+scSokh8AVpW030oqbrCA@mail.gmail.com>
 <ppe253$oie$1@blaine.gmane.org>
 <CAF7AXFGos0-GqFvv+J1HTZkqBiq1MfMwX=s5YzpRi-jfn=n92Q@mail.gmail.com>
 <20181009230605.GV3817@ando.pearwood.info>
 <b69bf75c-2e90-e344-de46-eb6e3701a90c@farowl.co.uk>
Message-ID: <CAF7AXFHejpRkgNn6S_-ksSKpfHMbgt0hUe-8EnBwiS5_eL=4Bg@mail.gmail.com>

On Sun, Oct 14, 2018 at 12:15 PM Jeff Allen <ja.py at farowl.co.uk> wrote:
>
>
> On 10/10/2018 00:06, Steven D'Aprano wrote:
>
> On Tue, Oct 09, 2018 at 09:37:48AM -0700, Jeff Hardy wrote:
>
> ...
>
> From an alternative implementation point of view, CPython's behaviour
> *is* the spec. Practicality beats purity and all that.
>
> Are you speaking on behalf of all authors of alternate implementations,
> or even of some of them?
>
> It certainly is not true that CPython's behaviour "is" the spec. PyPy
> keeps a list of CPython behaviour they don't match, either because they
> choose not to for other reasons, or because they believe that the
> CPython behaviour is buggy. I daresay IronPython and Jython have
> similar.
>
> While agreeing with the principle, unless it is one of the fundamental differences (GC, GIL), Jython usually lets practicality beat purity. When faced with a certain combination of objects, one has to do something, and it is least surprising to do what CPython does. It's also easier than keeping a record.

This is how it is for IronPython as well. When the pool of potential
users is already small, one cannot afford to get too pedantic about
whether something is in the spec or not. Matching what CPython does is
the easiest way to make sure as many people as possible can use an
alternative implementation.

- Jeff

From seanharr11 at gmail.com  Tue Oct 16 17:53:19 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Tue, 16 Oct 2018 17:53:19 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
Message-ID: <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>

Is your concern something like the following?

with Pool(8) as p:
    gen = p.imap_unordered(func, ls)
    first_elem = next(gen)
    p.apply_async(long_func, x)
    remaining_elems = [elem for elem in gen]

...here, if we store "func" on each worker Process as a global, and execute
this pattern above, we will likely alter state of one of the worker
processes s.t. it stores "long_func" in place of the initial "func".

So yes, this could break things. *A potential solution*:

Replace "func" in the task tuple with an identifier (maybe, *perhaps
> naively*, func.__qualname__), and store the "identifier => func map"
> somewhere globally accessible, maybe as a class attribute on Pool. On any
> call to Pool.map, Pool.apply, etc... this map is updated. Then, in the
> worker process, as each task is processed, we use the "func identifier" on
> the task to recover the globally mapped 'func', and apply it.


This would avoid the weird stateful bug above. We could also do something
slightly different to this if folks are averse to the "Pool class-attribute
func map" (i.e. averse to globals), and store this map as an Instance
Attribute on the Pool object, and wrap the "initializer" func to make the
map globally available in the worker via the "global" keyword.

One note: this isn't a "cache", it's just a global map which has its keys &
values updated *blindly* with every call to Pool.<public_method>. It serves
as a way to bypass repeated serialization of functions in Pool, which can
be large when bound to big objects (like large class instances, or
functools.partial objects).

On Tue, Oct 16, 2018 at 9:27 AM Michael Selik <michael.selik at gmail.com>
wrote:

> Would this change the other pool method behavior in some way if the user,
> for whatever reason, mixed techniques?
>
> imap_unordered will only block when nexting the generator. If the user
> mingles nexting that generator with, say, apply_async, could the change
> you're proposing have some side-effect?
>
> On Tue, Oct 16, 2018, 5:09 AM Sean Harrington <seanharr11 at gmail.com>
> wrote:
>
>> @Nataniel this is what I am suggesting as well. No cacheing - just
>> storing the `fn` on each worker, rather than pickling it for each item in
>> our iterable.
>>
>> As long as we store the `fn` post-fork on the worker process (perhaps as
>> global), subsequent calls to Pool.map shouldn't be effected (referencing
>> Antoine's & Michael's points that "multiprocessing encapsulates each
>> subprocesses globals in a separate namespace").
>>
>> @Antoine - I'm making an effort to take everything you've said into
>> consideration here.  My initial PR and talk
>> <https://www.youtube.com/watch?v=DH0JVSXvxu0> was intended to shed light
>> on a couple of pitfalls that I often see Python end-users encounter with
>> Pool. Moving beyond my naive first attempt, and the onslaught of deserved
>> criticism, it seems that we have an opportunity here: No changes to the
>> interface, just an optimization to reduce the frequency of pickling.
>>
>> Raymond Hettinger may also be interested in this optimization, as he
>> speaks (with great analogies) about different ways you can misuse
>> concurrency in Python <https://www.youtube.com/watch?v=9zinZmE3Ogk>.
>> This would address one of the pitfalls that he outlines: the "size of the
>> serialized/deserialized data".
>>
>> Is this an optimization that either of you would be willing to review,
>> and accept, if I find there is a *reasonable way* to implement it?
>>
>>
>> On Fri, Oct 12, 2018 at 3:40 PM Nathaniel Smith <njs at pobox.com> wrote:
>>
>>> On Fri, Oct 12, 2018, 06:09 Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>
>>>> On Fri, 12 Oct 2018 08:33:32 -0400
>>>> Sean Harrington <seanharr11 at gmail.com> wrote:
>>>> > Hi Nathaniel - this if this solution can be made performant, than I
>>>> would
>>>> > be more than satisfied.
>>>> >
>>>> > I think this would require removing "func" from the "task tuple", and
>>>> > storing the "func" "once per worker" somewhere globally (maybe a class
>>>> > attribute set post-fork?).
>>>> >
>>>> > This also has the beneficial outcome of increasing general
>>>> performance of
>>>> > Pool.map and friends. I've seen MANY folks across the interwebs doing
>>>> > things like passing instance methods to map, resulting in "big"
>>>> tasks, and
>>>> > slower-than-sequential parallelized code. Parallelizing "instance
>>>> methods"
>>>> > by passing them to map, w/o needing to wrangle with staticmethods and
>>>> > globals, would be a GREAT feature! It'd just be as easy as:
>>>> >
>>>> >     Pool.map(self.func, ls)
>>>> >
>>>> > What do you think about this idea? This is something I'd be able to
>>>> take
>>>> > on, assuming I get a few core dev blessings...
>>>>
>>>> Well, I'm not sure how it would work, so it's difficult to give an
>>>> opinion.  How do you plan to avoid passing "self"?  By caching (by
>>>> equality? by identity?)?  Something else?  But what happens if "self"
>>>> changed value (in the case of a mutable object) in the parent?  Do you
>>>> keep using the stale version in the child?  That would break
>>>> compatibility...
>>>>
>>>
>>> I was just suggesting that within a single call to Pool.map, it would be
>>> reasonable optimization to only send the fn once to each worker. So e.g. if
>>> you have 5 workers and 1000 items, you'd only pickle fn 5 times, rather
>>> than 1000 times like we do now. I wouldn't want to get any fancier than
>>> that with caching data between different map calls or anything.
>>>
>>> Of course even this may turn out to be too complicated to implement in a
>>> reasonable way, since it would require managing some extra state on the
>>> workers. But semantically it would be purely an optimization of current
>>> semantics.
>>>
>>> -n
>>>
>>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com
>>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181016/33e7b9f6/attachment.html>

From facundobatista at gmail.com  Wed Oct 17 20:32:14 2018
From: facundobatista at gmail.com (Facundo Batista)
Date: Wed, 17 Oct 2018 21:32:14 -0300
Subject: [Python-Dev] Servicing pypi.python.org
Message-ID: <CAM09pzSJe2Zy6C+FhQRGTSnKLBEpPKuR2HQzLCTeSSRB7HP27Q@mail.gmail.com>

Hola!

tl;dr: can we have a (semi)permanent redirect from pypi.python.org to pypi.org?

I own/maintain a project called `fades`, which is an automatic
virtualenv creator/manager [0].

While it relies on `pip` itself for the heavy work on the virtualenv
itself, `fades` checks if all required dependencies can be served [1]
*before* starting to do the rest of the work, which saves a lot of
time.

I just noticed that this is broken, because `pypi.python.org` is dead,
and the service should be queried under `pypi.org` directly.

We probably should have switched to the new domain months ago, but you
know how it's open source, it frequently happens that one has to live
first and that leaves no time for fun work.

So, can we have a (semi)permanent redirect from pypi.python.org to pypi.org?

This will help `fades` directly (even if we fix this and release soon,
there will be a lot of older fades in the wild hitting the de[recated
URL) and probably other projects using the old URL.

I suspect that having a generic redirect will not be burdensome server side.

Can we? Can we? P-p-please??? :)

Thanks in advance.

[0] fades.readthedocs.io
[1] using URLs like `https://pypi.python.org/pypi/Twisted/json`

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org.ar/
Twitter: @facundobatista

From tjreedy at udel.edu  Wed Oct 17 23:00:54 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 17 Oct 2018 23:00:54 -0400
Subject: [Python-Dev] Servicing pypi.python.org
In-Reply-To: <CAM09pzSJe2Zy6C+FhQRGTSnKLBEpPKuR2HQzLCTeSSRB7HP27Q@mail.gmail.com>
References: <CAM09pzSJe2Zy6C+FhQRGTSnKLBEpPKuR2HQzLCTeSSRB7HP27Q@mail.gmail.com>
Message-ID: <pq8st6$9i2$1@blaine.gmane.org>

On 10/17/2018 8:32 PM, Facundo Batista wrote:
> Hola!
> 
> tl;dr: can we have a (semi)permanent redirect from pypi.python.org to pypi.org?

pypi is run by a different group from pydev core developers.
Maybe someone here know what the current list is.

-- 
Terry Jan Reedy


From donald at stufft.io  Wed Oct 17 23:07:35 2018
From: donald at stufft.io (Donald Stufft)
Date: Wed, 17 Oct 2018 23:07:35 -0400
Subject: [Python-Dev] Servicing pypi.python.org
In-Reply-To: <CAM09pzSJe2Zy6C+FhQRGTSnKLBEpPKuR2HQzLCTeSSRB7HP27Q@mail.gmail.com>
References: <CAM09pzSJe2Zy6C+FhQRGTSnKLBEpPKuR2HQzLCTeSSRB7HP27Q@mail.gmail.com>
Message-ID: <C560340E-7722-49E0-9F98-C7D81BF25375@stufft.io>



> On Oct 17, 2018, at 8:32 PM, Facundo Batista <facundobatista at gmail.com> wrote:
> 
> tl;dr: can we have a (semi)permanent redirect from pypi.python.org <http://pypi.python.org/> to pypi.org <http://pypi.org/>?


This already exists:

$ curl -I https://pypi.python.org/project/Twisted/json
HTTP/2 301
server: Varnish
retry-after: 0
location: https://pypi.org/project/Twisted/json
content-type: text/html; charset=UTF-8
accept-ranges: bytes
date: Thu, 18 Oct 2018 03:06:12 GMT
x-served-by: cache-iad2626-IAD
x-cache: HIT
x-cache-hits: 0
x-timer: S1539831972.243137,VS0,VE0
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
content-length: 122


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181017/554e9bdd/attachment.html>

From michael.selik at gmail.com  Thu Oct 18 00:49:12 2018
From: michael.selik at gmail.com (Michael Selik)
Date: Wed, 17 Oct 2018 21:49:12 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
Message-ID: <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>

If imap_unordered is currently re-pickling and sending func each time it's
called on the worker, I have to suspect there was some reason to do that
and not cache it after the first call. Rather than assuming that's an
opportunity for an optimization, I'd want to be certain it won't have edge
case negative effects.


On Tue, Oct 16, 2018 at 2:53 PM Sean Harrington <seanharr11 at gmail.com>
wrote:

> Is your concern something like the following?
>
> with Pool(8) as p:
>     gen = p.imap_unordered(func, ls)
>     first_elem = next(gen)
>     p.apply_async(long_func, x)
>     remaining_elems = [elem for elem in gen]
>

My concern was passing the same function (or a function with the same
qualname). You're suggesting caching functions and identifying them by
qualname to avoid re-pickling a large stateful object that's shoved into
the function's defaults or closure. Is that a correct summary?

If so, how would the function cache distinguish between two functions with
the same name? Would it need to examine the defaults and closure as well?
If so, that means it's pickling the second one anyway, so there's no
efficiency gain.

In [1]: def foo(a):
   ...:     def bar():
   ...:         print(a)
   ...:     return bar
In [2]: f = foo(1)
In [3]: g = foo(2)
In [4]: f
Out[4]: <function __main__.foo.<locals>.bar()>
In [5]: g
Out[5]: <function __main__.foo.<locals>.bar()>

If we say pool.apply_async(f) and pool.apply_async(g), would you want the
latter one to avoid serialization, letting the worker make a second call
with the first function object?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181017/1d30bec3/attachment.html>

From 1benediktwerner at gmail.com  Thu Oct 18 01:11:45 2018
From: 1benediktwerner at gmail.com (Benedikt Werner)
Date: Thu, 18 Oct 2018 07:11:45 +0200
Subject: [Python-Dev] Servicing pypi.python.org
In-Reply-To: <C560340E-7722-49E0-9F98-C7D81BF25375@stufft.io>
References: <CAM09pzSJe2Zy6C+FhQRGTSnKLBEpPKuR2HQzLCTeSSRB7HP27Q@mail.gmail.com>
 <C560340E-7722-49E0-9F98-C7D81BF25375@stufft.io>
Message-ID: <de5a7105-c282-9d66-882a-9d808c6bafbd@gmail.com>

> This already exists:
>
> $ curl -I https://pypi.python.org/project/Twisted/json
> HTTP/2 301
> server: Varnish
> retry-after: 0
> location: https://pypi.org/project/Twisted/json
> content-type: text/html; charset=UTF-8
> accept-ranges: bytes
> date: Thu, 18 Oct 2018 03:06:12 GMT
> x-served-by: cache-iad2626-IAD
> x-cache: HIT
> x-cache-hits: 0
> x-timer: S1539831972.243137,VS0,VE0
> strict-transport-security: max-age=31536000; includeSubDomains; preload
> x-frame-options: deny
> x-xss-protection: 1; mode=block
> x-content-type-options: nosniff
> x-permitted-cross-domain-policies: none
> content-length: 122
Especially if you use a project that actually exists like 
https://pypi.python.org/pypi/numpy ;)
Then you don't get a 404 after the redirect which probably was your problem.

From van.lindberg at gmail.com  Thu Oct 18 10:07:13 2018
From: van.lindberg at gmail.com (VanL)
Date: Thu, 18 Oct 2018 09:07:13 -0500
Subject: [Python-Dev] wininst-*.exe files in Lib/distutils/command
Message-ID: <CAFQvZEPmGVscWNYF+Bng61LMdwpEgkjLqyhG3nogBedkG_u2KQ@mail.gmail.com>

Hi all,

I am looking into an issue associated with the wininst-*.exe files in the
distutils/command subdirectory. It looks like these are the executable
stubs used to create self-extracting zips for installation - but I am not
100% sure. It also looks like they include the calls to standard Windows
functions to display the installer window.

I have a couple questions I need help with:
1) Am I correct about the function, and if not, what are they?
2) Where did these come from, and where is their source code?

Thanks,
Van
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181018/09802cfd/attachment.html>

From p.f.moore at gmail.com  Thu Oct 18 10:36:23 2018
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 18 Oct 2018 15:36:23 +0100
Subject: [Python-Dev] wininst-*.exe files in Lib/distutils/command
In-Reply-To: <CAFQvZEPmGVscWNYF+Bng61LMdwpEgkjLqyhG3nogBedkG_u2KQ@mail.gmail.com>
References: <CAFQvZEPmGVscWNYF+Bng61LMdwpEgkjLqyhG3nogBedkG_u2KQ@mail.gmail.com>
Message-ID: <CACac1F_ddGsvGHun9V9VJcyFJ=aaGXT2bpMNi-J7U=tHNUBCyQ@mail.gmail.com>

They are for the distutils bdist_wininst command (mostly obsolete now,
wheels are the preferred binary distribution format these days). The
source appears to be in PC\bdist_wininst in the CPython repository.

Paul
On Thu, 18 Oct 2018 at 15:10, VanL <van.lindberg at gmail.com> wrote:
>
> Hi all,
>
> I am looking into an issue associated with the wininst-*.exe files in the distutils/command subdirectory. It looks like these are the executable stubs used to create self-extracting zips for installation - but I am not 100% sure. It also looks like they include the calls to standard Windows functions to display the installer window.
>
> I have a couple questions I need help with:
> 1) Am I correct about the function, and if not, what are they?
> 2) Where did these come from, and where is their source code?
>
> Thanks,
> Van
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/p.f.moore%40gmail.com

From zachary.ware+pydev at gmail.com  Thu Oct 18 10:40:53 2018
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Thu, 18 Oct 2018 09:40:53 -0500
Subject: [Python-Dev] wininst-*.exe files in Lib/distutils/command
In-Reply-To: <CAFQvZEPmGVscWNYF+Bng61LMdwpEgkjLqyhG3nogBedkG_u2KQ@mail.gmail.com>
References: <CAFQvZEPmGVscWNYF+Bng61LMdwpEgkjLqyhG3nogBedkG_u2KQ@mail.gmail.com>
Message-ID: <CAKJDb-ODjg6YkXOooz2crKKaz4n9u6gxgEVp_Z=qEf8U2+xLCQ@mail.gmail.com>

On Thu, Oct 18, 2018 at 9:09 AM VanL <van.lindberg at gmail.com> wrote:
> Hi all,
>
> I am looking into an issue associated with the wininst-*.exe files in the distutils/command subdirectory. It looks like these are the executable stubs used to create self-extracting zips for installation - but I am not 100% sure. It also looks like they include the calls to standard Windows functions to display the installer window.
>
> I have a couple questions I need help with:
> 1) Am I correct about the function, and if not, what are they?

You are correct.  IIUC, they are checked in to allow creating those
installers from non-Windows platforms.

> 2) Where did these come from, and where is their source code?

Source can be found here:
https://github.com/python/cpython/tree/master/PC/bdist_wininst

The individual checked-in .exe files were each originally built by
whoever updated the Windows toolchain to a new version of MSVC
(Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
ones were added by Thomas Heller, presumably using whatever the
current toolchain(s) was (were) at the time).  A few of them have been
rebuilt after bug fixes in the source since they were added, mostly by
the same people, though I also see Mark Hammond and Raymond Hettinger
in the history (and Georg Brandl via svnmerge).  I notice that there
are a few very minor code cleanups (the three latest commits here
https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c)
that have not made it into a rebuilt exe yet.

FTR, we really ought to remove all but the 14.0 version from the
master branch.  We don't support building Python with any toolchain
older than 14.0 anymore, and the older toolchains are nigh impossible
to find anymore anyway.

--
Zach

From van.lindberg at gmail.com  Thu Oct 18 10:48:10 2018
From: van.lindberg at gmail.com (VanL)
Date: Thu, 18 Oct 2018 09:48:10 -0500
Subject: [Python-Dev] wininst-*.exe files in Lib/distutils/command
In-Reply-To: <CAKJDb-ODjg6YkXOooz2crKKaz4n9u6gxgEVp_Z=qEf8U2+xLCQ@mail.gmail.com>
References: <CAFQvZEPmGVscWNYF+Bng61LMdwpEgkjLqyhG3nogBedkG_u2KQ@mail.gmail.com>
 <CAKJDb-ODjg6YkXOooz2crKKaz4n9u6gxgEVp_Z=qEf8U2+xLCQ@mail.gmail.com>
Message-ID: <CAFQvZEM05+0a5hmyJAX_A2+sfahrNUDJhpi1UM_kf4hRhSOikw@mail.gmail.com>

Thank you all, this gives me what I need. Sorry I missed the source in the
the PC/ directory.

Thanks,
Van

On Thu, Oct 18, 2018 at 9:41 AM Zachary Ware <zachary.ware+pydev at gmail.com>
wrote:

> On Thu, Oct 18, 2018 at 9:09 AM VanL <van.lindberg at gmail.com> wrote:
> > Hi all,
> >
> > I am looking into an issue associated with the wininst-*.exe files in
> the distutils/command subdirectory. It looks like these are the executable
> stubs used to create self-extracting zips for installation - but I am not
> 100% sure. It also looks like they include the calls to standard Windows
> functions to display the installer window.
> >
> > I have a couple questions I need help with:
> > 1) Am I correct about the function, and if not, what are they?
>
> You are correct.  IIUC, they are checked in to allow creating those
> installers from non-Windows platforms.
>
> > 2) Where did these come from, and where is their source code?
>
> Source can be found here:
> https://github.com/python/cpython/tree/master/PC/bdist_wininst
>
> The individual checked-in .exe files were each originally built by
> whoever updated the Windows toolchain to a new version of MSVC
> (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
> ones were added by Thomas Heller, presumably using whatever the
> current toolchain(s) was (were) at the time).  A few of them have been
> rebuilt after bug fixes in the source since they were added, mostly by
> the same people, though I also see Mark Hammond and Raymond Hettinger
> in the history (and Georg Brandl via svnmerge).  I notice that there
> are a few very minor code cleanups (the three latest commits here
> https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c
> )
> that have not made it into a rebuilt exe yet.
>
> FTR, we really ought to remove all but the 14.0 version from the
> master branch.  We don't support building Python with any toolchain
> older than 14.0 anymore, and the older toolchains are nigh impossible
> to find anymore anyway.
>
> --
> Zach
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181018/78b0b0f4/attachment-0001.html>

From seanharr11 at gmail.com  Thu Oct 18 11:35:16 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Thu, 18 Oct 2018 11:35:16 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
Message-ID: <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>

You have correctly identified the summary of my intentions, and I agree
with your reasoning & concern - however there is a somewhat reasonable
answer as to why this optimization has never been implemented:

In Pool, the `task` tuple consists of (result_job, func, (x,), {}) .  This
is the object that is serialized/deserialized b/t processes.  The only
thing we really care about here is the tuple `(x,)`, confusingly, not
`func` (func is ACTUALLY either mapstar() or starmapstar(), which is called
with (x,) as its *args). Our element of interest is `(x,)` - a tuple of
(func, iterable). Because we need to temper the size of the `iterable`
bundled in each task, to avoid de/serialization slowness, we usually end up
with multiple tasks per worker, and thus multiple `func`s per worker. Thus,
this is really only an optimization in the case of really big
functions/closures/partials (or REALLY big iterables with an unreasonably
small chunksize passed to map()). The most common use case comes up when
passing instance methods (of really big objects!) to Pool.map().

This post
<https://thelaziestprogrammer.com/python/a-multiprocessing-pool-pickle#stuck-in-a-pickle>
may
color in the above with more details.

Further, let me pivot on my idea of __qualname__...we can use the `id` of
`func` as the cache key to address your concern, and store this `id` on the
`task` tuple (i.e. an integer in-lieu of the `func` previously stored
there).


On Thu, Oct 18, 2018 at 12:49 AM Michael Selik <michael.selik at gmail.com>
wrote:

> If imap_unordered is currently re-pickling and sending func each time it's
> called on the worker, I have to suspect there was some reason to do that
> and not cache it after the first call. Rather than assuming that's an
> opportunity for an optimization, I'd want to be certain it won't have edge
> case negative effects.
>
>
> On Tue, Oct 16, 2018 at 2:53 PM Sean Harrington <seanharr11 at gmail.com>
> wrote:
>
>> Is your concern something like the following?
>>
>> with Pool(8) as p:
>>     gen = p.imap_unordered(func, ls)
>>     first_elem = next(gen)
>>     p.apply_async(long_func, x)
>>     remaining_elems = [elem for elem in gen]
>>
>
> My concern was passing the same function (or a function with the same
> qualname). You're suggesting caching functions and identifying them by
> qualname to avoid re-pickling a large stateful object that's shoved into
> the function's defaults or closure. Is that a correct summary?
>
> If so, how would the function cache distinguish between two functions with
> the same name? Would it need to examine the defaults and closure as well?
> If so, that means it's pickling the second one anyway, so there's no
> efficiency gain.
>
> In [1]: def foo(a):
>    ...:     def bar():
>    ...:         print(a)
>    ...:     return bar
> In [2]: f = foo(1)
> In [3]: g = foo(2)
> In [4]: f
> Out[4]: <function __main__.foo.<locals>.bar()>
> In [5]: g
> Out[5]: <function __main__.foo.<locals>.bar()>
>
> If we say pool.apply_async(f) and pool.apply_async(g), would you want the
> latter one to avoid serialization, letting the worker make a second call
> with the first function object?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181018/999d3aa6/attachment.html>

From michael.selik at gmail.com  Thu Oct 18 12:00:11 2018
From: michael.selik at gmail.com (Michael Selik)
Date: Thu, 18 Oct 2018 09:00:11 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
Message-ID: <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>

On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington <seanharr11 at gmail.com>
wrote:

> The most common use case comes up when passing instance methods (of really
> big objects!) to Pool.map().
>

This reminds me of that old joke: "A patient says to the doctor, 'Doctor,
it hurts when I ...!' The doctor replies, 'Well, don't do that.'"

Further, let me pivot on my idea of __qualname__...we can use the `id` of
> `func` as the cache key to address your concern, and store this `id` on the
> `task` tuple (i.e. an integer in-lieu of the `func` previously stored
> there).
>

Possible. Does the Pool keep a reference to the passed function in the main
process? If not, couldn't the garbage collector free that memory location
and a new function could replace it? Then it could have the same qualname
and id in CPython. Edge case, for sure. Worse, it'd be hard to reproduce as
it'd be dependent on the vagaries of memory allocation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181018/d98e0307/attachment.html>

From encukou at gmail.com  Thu Oct 18 12:26:34 2018
From: encukou at gmail.com (Petr Viktorin)
Date: Thu, 18 Oct 2018 18:26:34 +0200
Subject: [Python-Dev] wininst-*.exe files in Lib/distutils/command
In-Reply-To: <CAKJDb-ODjg6YkXOooz2crKKaz4n9u6gxgEVp_Z=qEf8U2+xLCQ@mail.gmail.com>
References: <CAFQvZEPmGVscWNYF+Bng61LMdwpEgkjLqyhG3nogBedkG_u2KQ@mail.gmail.com>
 <CAKJDb-ODjg6YkXOooz2crKKaz4n9u6gxgEVp_Z=qEf8U2+xLCQ@mail.gmail.com>
Message-ID: <5e5e14f2-b4f2-d904-ffc4-b825794d1600@gmail.com>

On 10/18/18 4:40 PM, Zachary Ware wrote:
> On Thu, Oct 18, 2018 at 9:09 AM VanL <van.lindberg at gmail.com> wrote:
>> Hi all,
>>
>> I am looking into an issue associated with the wininst-*.exe files in the distutils/command subdirectory. It looks like these are the executable stubs used to create self-extracting zips for installation - but I am not 100% sure. It also looks like they include the calls to standard Windows functions to display the installer window.
>>
>> I have a couple questions I need help with:
>> 1) Am I correct about the function, and if not, what are they?
> 
> You are correct.  IIUC, they are checked in to allow creating those
> installers from non-Windows platforms.

Is that the only reason for them?
At least on Linux, bdist_wininst does not work since at least Python 
3.2, as it tries to use a Windows-only encoding internally.
https://bugs.python.org/issue10945

If they're only there for non-Windows platforms, they're useless.

>> 2) Where did these come from, and where is their source code?
> 
> Source can be found here:
> https://github.com/python/cpython/tree/master/PC/bdist_wininst
> 
> The individual checked-in .exe files were each originally built by
> whoever updated the Windows toolchain to a new version of MSVC
> (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
> ones were added by Thomas Heller, presumably using whatever the
> current toolchain(s) was (were) at the time).  A few of them have been
> rebuilt after bug fixes in the source since they were added, mostly by
> the same people, though I also see Mark Hammond and Raymond Hettinger
> in the history (and Georg Brandl via svnmerge).  I notice that there
> are a few very minor code cleanups (the three latest commits here
> https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c)
> that have not made it into a rebuilt exe yet.
> 
> FTR, we really ought to remove all but the 14.0 version from the
> master branch.  We don't support building Python with any toolchain
> older than 14.0 anymore, and the older toolchains are nigh impossible
> to find anymore anyway.


From python at mrabarnett.plus.com  Thu Oct 18 12:41:29 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 18 Oct 2018 17:41:29 +0100
Subject: [Python-Dev] LibSSH Vulnerability
Message-ID: <503eca89-f6c9-cf81-9c47-34e45abd6cff@mrabarnett.plus.com>

I wondered if any of you have heard of this:

Hacker: I'm logged in. New LibSSH Vulnerability: OK! I believe you.
https://www.bleepingcomputer.com/news/security/hacker-im-logged-in-new-libssh-vulnerability-ok-i-believe-you/

From michael.selik at gmail.com  Thu Oct 18 12:39:34 2018
From: michael.selik at gmail.com (Michael Selik)
Date: Thu, 18 Oct 2018 09:39:34 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
 <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
Message-ID: <CAGgTfkObGn5L=oX9-K6-JjLfQX4b9ZGWsaoVm5xo6R3j7viUKg@mail.gmail.com>

One idea would be for the Pool method to generate a uuid and slap it on the
function as an attribute. If a function being passed in doesn't have one,
generate one. If it already has one, just pass that instead of pickling.
The child process will keep a cache mapping uuids to functions.

I'm still worried about unintended consequences.


On Thu, Oct 18, 2018 at 9:00 AM Michael Selik <michael.selik at gmail.com>
wrote:

> On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington <seanharr11 at gmail.com>
> wrote:
>
>> The most common use case comes up when passing instance methods (of
>> really big objects!) to Pool.map().
>>
>
> This reminds me of that old joke: "A patient says to the doctor, 'Doctor,
> it hurts when I ...!' The doctor replies, 'Well, don't do that.'"
>
> Further, let me pivot on my idea of __qualname__...we can use the `id` of
>> `func` as the cache key to address your concern, and store this `id` on the
>> `task` tuple (i.e. an integer in-lieu of the `func` previously stored
>> there).
>>
>
> Possible. Does the Pool keep a reference to the passed function in the
> main process? If not, couldn't the garbage collector free that memory
> location and a new function could replace it? Then it could have the same
> qualname and id in CPython. Edge case, for sure. Worse, it'd be hard to
> reproduce as it'd be dependent on the vagaries of memory allocation.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181018/4d2b7599/attachment.html>

From solipsis at pitrou.net  Thu Oct 18 12:51:58 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 18 Oct 2018 18:51:58 +0200
Subject: [Python-Dev] LibSSH Vulnerability
References: <503eca89-f6c9-cf81-9c47-34e45abd6cff@mrabarnett.plus.com>
Message-ID: <20181018185158.30749a9c@fsol>

On Thu, 18 Oct 2018 17:41:29 +0100
MRAB <python at mrabarnett.plus.com> wrote:
> I wondered if any of you have heard of this:
> 
> Hacker: I'm logged in. New LibSSH Vulnerability: OK! I believe you.
> https://www.bleepingcomputer.com/news/security/hacker-im-logged-in-new-libssh-vulnerability-ok-i-believe-you/

AFAIK, this doesn't have any to do with OpenSSH, which practically
everyone uses on Unix.

Regards

Antoine.



From van.lindberg at gmail.com  Thu Oct 18 13:44:19 2018
From: van.lindberg at gmail.com (VanL)
Date: Thu, 18 Oct 2018 12:44:19 -0500
Subject: [Python-Dev] wininst-*.exe files in Lib/distutils/command
In-Reply-To: <5e5e14f2-b4f2-d904-ffc4-b825794d1600@gmail.com>
References: <CAFQvZEPmGVscWNYF+Bng61LMdwpEgkjLqyhG3nogBedkG_u2KQ@mail.gmail.com>
 <CAKJDb-ODjg6YkXOooz2crKKaz4n9u6gxgEVp_Z=qEf8U2+xLCQ@mail.gmail.com>
 <5e5e14f2-b4f2-d904-ffc4-b825794d1600@gmail.com>
Message-ID: <CAFQvZEPRFqhia_y0f9CGpRgTBPsRiRVkiK=Ek4MphBYPe-aX9g@mail.gmail.com>

Primarily for non-windows platforms, but I also think for Windows users
without any compilers or similar tools installed. There is also some
discussion of removing some of the older toolchain-specific versions
(leaving only -14), but that is a subject for another day.

Also I am not sure that bug report applies to 3.5/3.6/3.7.

Thanks,
Van


On Thu, Oct 18, 2018 at 11:26 AM Petr Viktorin <encukou at gmail.com> wrote:

> On 10/18/18 4:40 PM, Zachary Ware wrote:
> > On Thu, Oct 18, 2018 at 9:09 AM VanL <van.lindberg at gmail.com> wrote:
> >> Hi all,
> >>
> >> I am looking into an issue associated with the wininst-*.exe files in
> the distutils/command subdirectory. It looks like these are the executable
> stubs used to create self-extracting zips for installation - but I am not
> 100% sure. It also looks like they include the calls to standard Windows
> functions to display the installer window.
> >>
> >> I have a couple questions I need help with:
> >> 1) Am I correct about the function, and if not, what are they?
> >
> > You are correct.  IIUC, they are checked in to allow creating those
> > installers from non-Windows platforms.
>
> Is that the only reason for them?
> At least on Linux, bdist_wininst does not work since at least Python
> 3.2, as it tries to use a Windows-only encoding internally.
> https://bugs.python.org/issue10945
>
> If they're only there for non-Windows platforms, they're useless.
>
> >> 2) Where did these come from, and where is their source code?
> >
> > Source can be found here:
> > https://github.com/python/cpython/tree/master/PC/bdist_wininst
> >
> > The individual checked-in .exe files were each originally built by
> > whoever updated the Windows toolchain to a new version of MSVC
> > (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
> > ones were added by Thomas Heller, presumably using whatever the
> > current toolchain(s) was (were) at the time).  A few of them have been
> > rebuilt after bug fixes in the source since they were added, mostly by
> > the same people, though I also see Mark Hammond and Raymond Hettinger
> > in the history (and Georg Brandl via svnmerge).  I notice that there
> > are a few very minor code cleanups (the three latest commits here
> >
> https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c
> )
> > that have not made it into a rebuilt exe yet.
> >
> > FTR, we really ought to remove all but the 14.0 version from the
> > master branch.  We don't support building Python with any toolchain
> > older than 14.0 anymore, and the older toolchains are nigh impossible
> > to find anymore anyway.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181018/59f057b4/attachment.html>

From jmcs at jsantos.eu  Thu Oct 18 15:16:29 2018
From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=)
Date: Thu, 18 Oct 2018 21:16:29 +0200
Subject: [Python-Dev] LibSSH Vulnerability
In-Reply-To: <20181018185158.30749a9c@fsol>
References: <503eca89-f6c9-cf81-9c47-34e45abd6cff@mrabarnett.plus.com>
 <20181018185158.30749a9c@fsol>
Message-ID: <CAH_XWH1BT4pdNoSSv9Gxt5hv-fy_YQKJ5QbxTyUVT1W+Wopn1Q@mail.gmail.com>

Slightly Off-Topic, but more relevant for python developers. Paramiko also
had a very similar vulnerability:
https://github.com/paramiko/paramiko/issues/1283.

On Thu, 18 Oct 2018 at 18:56, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Thu, 18 Oct 2018 17:41:29 +0100
> MRAB <python at mrabarnett.plus.com> wrote:
> > I wondered if any of you have heard of this:
> >
> > Hacker: I'm logged in. New LibSSH Vulnerability: OK! I believe you.
> >
> https://www.bleepingcomputer.com/news/security/hacker-im-logged-in-new-libssh-vulnerability-ok-i-believe-you/
>
> AFAIK, this doesn't have any to do with OpenSSH, which practically
> everyone uses on Unix.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jmcs%40jsantos.eu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181018/5a2cf06f/attachment.html>

From chris.jerdonek at gmail.com  Thu Oct 18 16:16:57 2018
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Thu, 18 Oct 2018 13:16:57 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
 <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
Message-ID: <CAOTb1wc_gTuTC2Tmi6NYYLpzrtH6V=gKOjK5eF25QLC-kPkWgA@mail.gmail.com>

On Thu, Oct 18, 2018 at 9:11 AM Michael Selik <michael.selik at gmail.com> wrote:
> On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington <seanharr11 at gmail.com> wrote:
>> Further, let me pivot on my idea of __qualname__...we can use the `id` of `func` as the cache key to address your concern, and store this `id` on the `task` tuple (i.e. an integer in-lieu of the `func` previously stored there).
>
>
> Possible. Does the Pool keep a reference to the passed function in the main process? If not, couldn't the garbage collector free that memory location and a new function could replace it? Then it could have the same qualname and id in CPython. Edge case, for sure. Worse, it'd be hard to reproduce as it'd be dependent on the vagaries of memory allocation.

I'm not following this thread closely, but I just wanted to point out
that __qualname__ won't necessarily be an attribute of the object if
the API accepts any callable. (I happen to be following an issue on
the tracker where this came up.)

--Chris

From thomas.moreau.2010 at gmail.com  Fri Oct 19 02:57:35 2018
From: thomas.moreau.2010 at gmail.com (Thomas Moreau)
Date: Fri, 19 Oct 2018 08:57:35 +0200
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAOTb1wc_gTuTC2Tmi6NYYLpzrtH6V=gKOjK5eF25QLC-kPkWgA@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
 <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
 <CAOTb1wc_gTuTC2Tmi6NYYLpzrtH6V=gKOjK5eF25QLC-kPkWgA@mail.gmail.com>
Message-ID: <CAOh-wgBc91w_D5OmryTdkp3tUk7BqQ=HgDEiUF99LG9fvCLoqA@mail.gmail.com>

Hello,

I have been working on the concurent.futures module lately and I think this
optimization should be avoided in the context of python Pools.

This is an interesting idea, however its implementation will bring many
complicated issues as it breaks the basic paradigm of a Pool: the tasks are
independent and you don't know which worker is going to run which task.

The function is serialized with each task because of this paradigm. This
ensure that any worker picking the task will be able to perform it
independently from the tasks it has run before, given that it as been
initialized correctly at the beginning. This makes it simple to run each
task.

As the Pool comes with no scheduler, with your idea, you would need a
synchronization step to send the function to all workers before you can
launch your task. But if there is already one worker performing a long
running task, does the Pool wait for it to be done before it sends the
function? If the Pool doesn't wait, how does it ensure that this worker
will be able to get the definition of the function before running it?
Also, the multiprocessing.Pool has some features where a worker can shut
itself down after a given number of tasks or a timeout. How does it ensure
that the new worker will have the definition of the function?
It is unsafe to try such a feature (sending only once an object) anywhere
else than in the initializer which is guaranteed to be run once per worker.

On the other hand, you mentioned an interesting point being that making
globals available in the workers could be made simpler. A possible solution
would be to add a "globals" argument in the Pool which would instanciate
global variables in the workers. I have no specific idea but on the
implementation of such features but it would be safer as it would be an
initialization feature.

Regards,
Thomas Moreau

On Thu, Oct 18, 2018, 22:20 Chris Jerdonek <chris.jerdonek at gmail.com> wrote:

> On Thu, Oct 18, 2018 at 9:11 AM Michael Selik <michael.selik at gmail.com>
> wrote:
> > On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington <seanharr11 at gmail.com>
> wrote:
> >> Further, let me pivot on my idea of __qualname__...we can use the `id`
> of `func` as the cache key to address your concern, and store this `id` on
> the `task` tuple (i.e. an integer in-lieu of the `func` previously stored
> there).
> >
> >
> > Possible. Does the Pool keep a reference to the passed function in the
> main process? If not, couldn't the garbage collector free that memory
> location and a new function could replace it? Then it could have the same
> qualname and id in CPython. Edge case, for sure. Worse, it'd be hard to
> reproduce as it'd be dependent on the vagaries of memory allocation.
>
> I'm not following this thread closely, but I just wanted to point out
> that __qualname__ won't necessarily be an attribute of the object if
> the API accepts any callable. (I happen to be following an issue on
> the tracker where this came up.)
>
> --Chris
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/thomas.moreau.2010%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181019/ef763de8/attachment-0001.html>

From j.orponen at 4teamwork.ch  Fri Oct 19 07:31:46 2018
From: j.orponen at 4teamwork.ch (Joni Orponen)
Date: Fri, 19 Oct 2018 13:31:46 +0200
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAOh-wgBc91w_D5OmryTdkp3tUk7BqQ=HgDEiUF99LG9fvCLoqA@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
 <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
 <CAOTb1wc_gTuTC2Tmi6NYYLpzrtH6V=gKOjK5eF25QLC-kPkWgA@mail.gmail.com>
 <CAOh-wgBc91w_D5OmryTdkp3tUk7BqQ=HgDEiUF99LG9fvCLoqA@mail.gmail.com>
Message-ID: <CA+LW3i0uoUhZvW9DeGCo9TT73JgManuf9kBmC67HHksNkwRavg@mail.gmail.com>

On Fri, Oct 19, 2018 at 9:09 AM Thomas Moreau <thomas.moreau.2010 at gmail.com>
wrote:

> Hello,
>
> I have been working on the concurent.futures module lately and I think
> this optimization should be avoided in the context of python Pools.
>
> This is an interesting idea, however its implementation will bring many
> complicated issues as it breaks the basic paradigm of a Pool: the tasks are
> independent and you don't know which worker is going to run which task.
>
> The function is serialized with each task because of this paradigm. This
> ensure that any worker picking the task will be able to perform it
> independently from the tasks it has run before, given that it as been
> initialized correctly at the beginning. This makes it simple to run each
> task.
>

I would not mind if there would be a subtype of Pool where you can only
apply one kind of task to. This is a very common use mode.

Though the question there is 'should this live in Python itself'? I'd be
fine with a package on PyPi.

As the Pool comes with no scheduler, with your idea, you would need a
> synchronization step to send the function to all workers before you can
> launch your task. But if there is already one worker performing a long
> running task, does the Pool wait for it to be done before it sends the
> function? If the Pool doesn't wait, how does it ensure that this worker
> will be able to get the definition of the function before running it?
> Also, the multiprocessing.Pool has some features where a worker can shut
> itself down after a given number of tasks or a timeout. How does it ensure
> that the new worker will have the definition of the function?
> It is unsafe to try such a feature (sending only once an object) anywhere
> else than in the initializer which is guaranteed to be run once per worker.
>
> On the other hand, you mentioned an interesting point being that making
> globals available in the workers could be made simpler. A possible solution
> would be to add a "globals" argument in the Pool which would instanciate
> global variables in the workers. I have no specific idea but on the
> implementation of such features but it would be safer as it would be an
> initialization feature.
>

Would this also mean one could use a Pool in a context where threading is
used? Currently using threading side effects unpicklables into the globals.

Also being able to pass in globals=None would be optimal for a lot of use
cases.

-- 
Joni Orponen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181019/3a30c4bc/attachment.html>

From seanharr11 at gmail.com  Fri Oct 19 07:58:27 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Fri, 19 Oct 2018 07:58:27 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CA+LW3i0uoUhZvW9DeGCo9TT73JgManuf9kBmC67HHksNkwRavg@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
 <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
 <CAOTb1wc_gTuTC2Tmi6NYYLpzrtH6V=gKOjK5eF25QLC-kPkWgA@mail.gmail.com>
 <CAOh-wgBc91w_D5OmryTdkp3tUk7BqQ=HgDEiUF99LG9fvCLoqA@mail.gmail.com>
 <CA+LW3i0uoUhZvW9DeGCo9TT73JgManuf9kBmC67HHksNkwRavg@mail.gmail.com>
Message-ID: <CAFbyWc0X95an_kOC-QQ7g_VhUKUioM=kMu+=2N1ZLXhWXThhyw@mail.gmail.com>

On Fri, Oct 19, 2018 at 7:32 AM Joni Orponen <j.orponen at 4teamwork.ch> wrote:

> On Fri, Oct 19, 2018 at 9:09 AM Thomas Moreau <
> thomas.moreau.2010 at gmail.com> wrote:
>
>> Hello,
>>
>> I have been working on the concurent.futures module lately and I think
>> this optimization should be avoided in the context of python Pools.
>>
>> This is an interesting idea, however its implementation will bring many
>> complicated issues as it breaks the basic paradigm of a Pool: the tasks are
>> independent and you don't know which worker is going to run which task.
>>
>> The function is serialized with each task because of this paradigm. This
>> ensure that any worker picking the task will be able to perform it
>> independently from the tasks it has run before, given that it as been
>> initialized correctly at the beginning. This makes it simple to run each
>> task.
>>
>
> I would not mind if there would be a subtype of Pool where you can only
> apply one kind of task to. This is a very common use mode.
>
>
Though the question there is 'should this live in Python itself'? I'd be
> fine with a package on PyPi.
>

Thomas makes a good point: despite the common user mode of calling
Pool.map() once, blocking, and returning, the need for serialization of
functions within tasks arises when calling Pool.map() (and friends) while
workers are still running (i.e. there was a previous call to
Pool.async_map()).

However this is an uncommon user mode, as Joni points out. The most common
user mode is that your Pool workers are only ever executing one type of
task at a given time.  This opens up optimization opportunities, so long as
we store state on the subclassed Pool object of whether or not a SingleTask
is running, or has been completed(SingleTaskPool?), to prevent the user
from getting in this funky state above.

I would rather see this included in the multiprocessing stdlib, as it
seemingly will not introduce a lot of new code, would benefit from existing
tests. Most importantly it optimizes (in my opinion) the most common user
mode of Pool.


> As the Pool comes with no scheduler, with your idea, you would need a
>> synchronization step to send the function to all workers before you can
>> launch your task. But if there is already one worker performing a long
>> running task, does the Pool wait for it to be done before it sends the
>> function? If the Pool doesn't wait, how does it ensure that this worker
>> will be able to get the definition of the function before running it?
>> Also, the multiprocessing.Pool has some features where a worker can shut
>> itself down after a given number of tasks or a timeout. How does it ensure
>> that the new worker will have the definition of the function?
>> It is unsafe to try such a feature (sending only once an object) anywhere
>> else than in the initializer which is guaranteed to be run once per worker.
>>
>> On the other hand, you mentioned an interesting point being that making
>> globals available in the workers could be made simpler. A possible solution
>> would be to add a "globals" argument in the Pool which would instanciate
>> global variables in the workers. I have no specific idea but on the
>> implementation of such features but it would be safer as it would be an
>> initialization feature.
>>
>
> Would this also mean one could use a Pool in a context where threading is
> used? Currently using threading side effects unpicklables into the globals.
>
> Also being able to pass in globals=None would be optimal for a lot of use
> cases.
>

We could do this - but we can easily get the same behavior by declaring a
"global" in "initializer" (albeit a pattern which I do not like).  I like
the idea to extend the Pool class a bit more, but this is also my opinion.


>
> --
> Joni Orponen
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181019/220eb360/attachment.html>

From status at bugs.python.org  Fri Oct 19 12:10:06 2018
From: status at bugs.python.org (Python tracker)
Date: Fri, 19 Oct 2018 18:10:06 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20181019161006.4AEE6116829@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2018-10-12 - 2018-10-19)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6835 (+11)
  closed 39943 (+50)
  total  46778 (+61)

Open issues with patches: 2737 


Issues opened (41)
==================

#34783: [3.7] segmentation-fault/core dump when try to run non-existin
https://bugs.python.org/issue34783  reopened by ned.deily

#34909: StrEnum subclasses cannot be created
https://bugs.python.org/issue34909  reopened by ned.deily

#34968: loop.call_soon_threadsafe should be documented to be re-entran
https://bugs.python.org/issue34968  opened by njs

#34969: Add --fast, --best to the gzip CLI
https://bugs.python.org/issue34969  opened by matrixise

#34970: Protect tasks weak set manipulation in asyncio.all_tasks()
https://bugs.python.org/issue34970  opened by asvetlov

#34971: add support for tls/ssl sessions in asyncio
https://bugs.python.org/issue34971  opened by RemiCardona

#34973: Crash in bytes constructor with mutating list
https://bugs.python.org/issue34973  opened by serhiy.storchaka

#34975: start_tls() difficult when using asyncio.start_server()
https://bugs.python.org/issue34975  opened by icgood

#34976: IDLE: Replace the search dialog with a search bar
https://bugs.python.org/issue34976  opened by taleinat

#34977: Release Windows Store app containing Python
https://bugs.python.org/issue34977  opened by steve.dower

#34978: check type of object in fix_dict.py in 2to3
https://bugs.python.org/issue34978  opened by devarakondapranav

#34979: Python throws ???SyntaxError: Non-UTF-8 code start with \xe8..
https://bugs.python.org/issue34979  opened by ausaki

#34980: KillPython target doesn't detect 64-bit processes
https://bugs.python.org/issue34980  opened by jkloth

#34981: Unable to install Python from web-based installer and executab
https://bugs.python.org/issue34981  opened by skycraper

#34983: expose symtable.Symbol.is_nonlocal()
https://bugs.python.org/issue34983  opened by pablogsal

#34984: Improve error messages in bytes and bytearray constructors
https://bugs.python.org/issue34984  opened by serhiy.storchaka

#34985: python finds test modules from the wrong directory during PGO 
https://bugs.python.org/issue34985  opened by Kal Sze2

#34987: A possible null pointer dereference in _pickle.c's save_reduce
https://bugs.python.org/issue34987  opened by ZackerySpytz

#34990: year 2038 problem in compileall.py
https://bugs.python.org/issue34990  opened by bmwiedemann

#34991: variable type list [] referential integrity data loss
https://bugs.python.org/issue34991  opened by alan.pan

#34993: asyncio.streams.FlowControlMixin should be part of the API
https://bugs.python.org/issue34993  opened by xitop

#34995: functools.cached_property does not maintain the wrapped method
https://bugs.python.org/issue34995  opened by mwilbz

#34996: Add name to process and thread pool
https://bugs.python.org/issue34996  opened by Raz Manor

#35000: aexit called after loop close
https://bugs.python.org/issue35000  opened by pdxjohnny

#35003: Provide an option to venv to put files in a bin/ directory on 
https://bugs.python.org/issue35003  opened by brett.cannon

#35004: Odd behavior when using datetime.timedelta under cProfile
https://bugs.python.org/issue35004  opened by beaugunderson

#35005: argparse should accept json and yaml argument types
https://bugs.python.org/issue35005  opened by derelbenkoenig

#35007: Minor change to weakref docs
https://bugs.python.org/issue35007  opened by frankmillman

#35009: argparse throws UnicodeEncodeError for printing help with unic
https://bugs.python.org/issue35009  opened by xtreak

#35012: [3.7] test_multiprocessing_spawn hangs randomly on AppVeyor
https://bugs.python.org/issue35012  opened by vstinner

#35015: availability directive breaks po files
https://bugs.python.org/issue35015  opened by mdk

#35017: socketserver accept a last request after shutdown
https://bugs.python.org/issue35017  opened by beledouxdenis

#35018: Sax parser provides no user access to lexical handlers
https://bugs.python.org/issue35018  opened by Jonathan.Gossage

#35019: Minor Bug found in asyncio - Python 3.5.3
https://bugs.python.org/issue35019  opened by bassford

#35020: Add multisort recipe to sorting docs
https://bugs.python.org/issue35020  opened by xtreak

#35021: Assertion failures in datetimemodule.c.
https://bugs.python.org/issue35021  opened by twouters

#35022: MagicMock should support `__fspath__`
https://bugs.python.org/issue35022  opened by Maxime Belanger

#35024: Incorrect logging in importlib when '.pyc' file creation fails
https://bugs.python.org/issue35024  opened by qagren

#35025: Compiling `timemodule.c` can fail on macOS due to availability
https://bugs.python.org/issue35025  opened by Maxime Belanger

#35026: Winreg's documentation lacks mentioning required permission at
https://bugs.python.org/issue35026  opened by georgefischhof

#35027: distutils.core.setup does not raise TypeError when if classifi
https://bugs.python.org/issue35027  opened by TilmanKrummeck



Most recent 15 issues with no replies (15)
==========================================

#35027: distutils.core.setup does not raise TypeError when if classifi
https://bugs.python.org/issue35027

#35026: Winreg's documentation lacks mentioning required permission at
https://bugs.python.org/issue35026

#35025: Compiling `timemodule.c` can fail on macOS due to availability
https://bugs.python.org/issue35025

#35024: Incorrect logging in importlib when '.pyc' file creation fails
https://bugs.python.org/issue35024

#35022: MagicMock should support `__fspath__`
https://bugs.python.org/issue35022

#35020: Add multisort recipe to sorting docs
https://bugs.python.org/issue35020

#35018: Sax parser provides no user access to lexical handlers
https://bugs.python.org/issue35018

#35017: socketserver accept a last request after shutdown
https://bugs.python.org/issue35017

#35009: argparse throws UnicodeEncodeError for printing help with unic
https://bugs.python.org/issue35009

#35007: Minor change to weakref docs
https://bugs.python.org/issue35007

#35003: Provide an option to venv to put files in a bin/ directory on 
https://bugs.python.org/issue35003

#35000: aexit called after loop close
https://bugs.python.org/issue35000

#34984: Improve error messages in bytes and bytearray constructors
https://bugs.python.org/issue34984

#34983: expose symtable.Symbol.is_nonlocal()
https://bugs.python.org/issue34983

#34958: urllib.error.HTTPError.fp is not closed when error is finalize
https://bugs.python.org/issue34958



Most recent 15 issues waiting for review (15)
=============================================

#35025: Compiling `timemodule.c` can fail on macOS due to availability
https://bugs.python.org/issue35025

#35022: MagicMock should support `__fspath__`
https://bugs.python.org/issue35022

#35021: Assertion failures in datetimemodule.c.
https://bugs.python.org/issue35021

#35020: Add multisort recipe to sorting docs
https://bugs.python.org/issue35020

#35019: Minor Bug found in asyncio - Python 3.5.3
https://bugs.python.org/issue35019

#35017: socketserver accept a last request after shutdown
https://bugs.python.org/issue35017

#34996: Add name to process and thread pool
https://bugs.python.org/issue34996

#34995: functools.cached_property does not maintain the wrapped method
https://bugs.python.org/issue34995

#34990: year 2038 problem in compileall.py
https://bugs.python.org/issue34990

#34987: A possible null pointer dereference in _pickle.c's save_reduce
https://bugs.python.org/issue34987

#34984: Improve error messages in bytes and bytearray constructors
https://bugs.python.org/issue34984

#34983: expose symtable.Symbol.is_nonlocal()
https://bugs.python.org/issue34983

#34980: KillPython target doesn't detect 64-bit processes
https://bugs.python.org/issue34980

#34979: Python throws ???SyntaxError: Non-UTF-8 code start with \xe8..
https://bugs.python.org/issue34979

#34976: IDLE: Replace the search dialog with a search bar
https://bugs.python.org/issue34976



Top 10 most discussed issues (10)
=================================

#34970: Protect tasks weak set manipulation in asyncio.all_tasks()
https://bugs.python.org/issue34970  10 msgs

#35005: argparse should accept json and yaml argument types
https://bugs.python.org/issue35005  10 msgs

#34953: Implement `mmap.mmap.__repr__`
https://bugs.python.org/issue34953   7 msgs

#34979: Python throws ???SyntaxError: Non-UTF-8 code start with \xe8..
https://bugs.python.org/issue34979   7 msgs

#34990: year 2038 problem in compileall.py
https://bugs.python.org/issue34990   7 msgs

#35015: availability directive breaks po files
https://bugs.python.org/issue35015   7 msgs

#34919: Crash caused by certain characters in a string
https://bugs.python.org/issue34919   6 msgs

#34995: functools.cached_property does not maintain the wrapped method
https://bugs.python.org/issue34995   6 msgs

#34095: [2.7] Seg fault on archlinux 32 when run tests with xvfb-run
https://bugs.python.org/issue34095   5 msgs

#34783: [3.7] segmentation-fault/core dump when try to run non-existin
https://bugs.python.org/issue34783   5 msgs



Issues closed (48)
==================

#12572: HP/UX compiler workarounds
https://bugs.python.org/issue12572  closed by terry.reedy

#16965: 2to3 should rewrite execfile() to open in 'rb' mode
https://bugs.python.org/issue16965  closed by serhiy.storchaka

#17837: Support for building on ppc64p7
https://bugs.python.org/issue17837  closed by vstinner

#22851: 2.7 core crashes with generator.gi_frame.f_restricted
https://bugs.python.org/issue22851  closed by serhiy.storchaka

#22872: multiprocessing.Queue raises AssertionError
https://bugs.python.org/issue22872  closed by serhiy.storchaka

#23420: python -m cProfile -s fails with non informative message
https://bugs.python.org/issue23420  closed by vstinner

#23554: EchoServerClientProtocol class should be called EchoServerProt
https://bugs.python.org/issue23554  closed by yselivanov

#23831: tkinter canvas lacks of moveto method.
https://bugs.python.org/issue23831  closed by serhiy.storchaka

#26441: email.charset: to_splittable and from_splittable are not there
https://bugs.python.org/issue26441  closed by r.david.murray

#30220: Why are custom messages for ValueError, TypeError suppressed i
https://bugs.python.org/issue30220  closed by paul.j3

#32789: Note missing from logging.debug() docs
https://bugs.python.org/issue32789  closed by vinay.sajip

#32912: Raise non-silent warning for invalid escape sequences
https://bugs.python.org/issue32912  closed by serhiy.storchaka

#34158: Documentation of datetime  '%z' format code is odd
https://bugs.python.org/issue34158  closed by ned.deily

#34475: functools.partial objects have no __qualname__ attribute
https://bugs.python.org/issue34475  closed by serhiy.storchaka

#34565: Launcher does not validate major versions
https://bugs.python.org/issue34565  closed by ned.deily

#34590: "Logging HOWTO" should share an example of best practices for 
https://bugs.python.org/issue34590  closed by vinay.sajip

#34741: Get rid of tp_getattro and tp_setattro in pyexpat.xmlparser
https://bugs.python.org/issue34741  closed by serhiy.storchaka

#34765: Update install-sh
https://bugs.python.org/issue34765  closed by vstinner

#34844: logging.Formatter enhancement - Checking on style and fmt fiel
https://bugs.python.org/issue34844  closed by vinay.sajip

#34918: Python 3 tkinter measurement problem
https://bugs.python.org/issue34918  closed by serhiy.storchaka

#34920: PYTHONWARNINGS entries are escaped
https://bugs.python.org/issue34920  closed by martin.panter

#34927: Tkinter-related segfault on macOS (regression between 3.7.0 an
https://bugs.python.org/issue34927  closed by ned.deily

#34930: sha1module: Switch sha1 implementation to sha1dc/hardened sha1
https://bugs.python.org/issue34930  closed by antoine.pietri

#34939: Possibly spurious SyntaxError: annotated name can't be global
https://bugs.python.org/issue34939  closed by pablogsal

#34941: xml.etree.ElementTree findall() fails when using custom TreeBu
https://bugs.python.org/issue34941  closed by serhiy.storchaka

#34961: Global scoping when shadowing local names in class definitions
https://bugs.python.org/issue34961  closed by multun

#34967: Sphinx is deprecating add_description_unit
https://bugs.python.org/issue34967  closed by mdk

#34972: json dump silently converts int keys to string
https://bugs.python.org/issue34972  closed by steve.dower

#34974: bytes and bytearray constructors replace unexpected exceptions
https://bugs.python.org/issue34974  closed by serhiy.storchaka

#34982: re.sub() different behavior in 3.7
https://bugs.python.org/issue34982  closed by serhiy.storchaka

#34986: python finds test modules from the wrong directory during PGO 
https://bugs.python.org/issue34986  closed by Kal Sze2

#34988: Rc2 candidates: "gcc" not found on AIX
https://bugs.python.org/issue34988  closed by ned.deily

#34989: python-gdb.py fails with TypeError("'FakeRepr' object is not s
https://bugs.python.org/issue34989  closed by vstinner

#34992: many warnings with f28 and gcc 8.1.1
https://bugs.python.org/issue34992  closed by vstinner

#34994: sysconfig returns the git commands for GITBRANCH, GITVERSION a
https://bugs.python.org/issue34994  closed by ned.deily

#34997: test.test_logging.ConfigDictTest.test_out_of_order fails in x8
https://bugs.python.org/issue34997  closed by pablogsal

#34998: Logging formatter validation breaks backward ducktyping
https://bugs.python.org/issue34998  closed by vinay.sajip

#34999: copy.copy and deepcopy do return same logger objects in 3.7
https://bugs.python.org/issue34999  closed by vinay.sajip

#35001: ImportFrom level cannot be optional
https://bugs.python.org/issue35001  closed by brett.cannon

#35002: Potential bug in ConfigParser.read() in python3.6, before os.P
https://bugs.python.org/issue35002  closed by BNMetrics

#35006: itertools.combinations has wrong type when using the typing pa
https://bugs.python.org/issue35006  closed by brett.cannon

#35008: Leaks xml.etree.ElementTree.Element.__setsate__()
https://bugs.python.org/issue35008  closed by serhiy.storchaka

#35010: sort by partially reversed key tuple
https://bugs.python.org/issue35010  closed by rhettinger

#35011: expat: Restore the use of pyexpatns.h to avoid link time confl
https://bugs.python.org/issue35011  closed by gregory.p.smith

#35013: Add more type checks for children of xml.etree.ElementTree.Ele
https://bugs.python.org/issue35013  closed by serhiy.storchaka

#35014: asyncio subprocess accepts string as parameter which lead to U
https://bugs.python.org/issue35014  closed by natim

#35016: \r to match add into address header with not-ascii character
https://bugs.python.org/issue35016  closed by r.david.murray

#35023: Missed a key when iterating over dictionary
https://bugs.python.org/issue35023  closed by ammar2

From stephane at wirtel.be  Fri Oct 19 12:57:54 2018
From: stephane at wirtel.be (Stephane Wirtel)
Date: Fri, 19 Oct 2018 18:57:54 +0200
Subject: [Python-Dev] Some PRs to merge?
Message-ID: <20181019165754.GA17454@xps>

Hi all,

How are you? I am fine ;-) and you?

So, on this morning I was playing with the github interface and the
pull requests of CPython and I have discovered the advanced search of
Github and I think this one is really useful for us and certainly for
the core-dev.

So, I was interested by somes PRs.

PRs with this status:
* open
* review is approved
* status of the CI is 'success'
* has labels "awaiting merge", "CLA signed" and -"DO-NOT-MERGE"

total: 49 PRs

In the GitHub interface, here is the criteria

is:open is:pr review:approved status:success label:"awaiting merge" -label:"DO-NOT-MERGE" label:""LA signed""

But if you want to see the result in your browser, just click on this link.
https://github.com/python/cpython/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+review%3Aapproved+status%3Asuccess+label%3A%22awaiting+merge%22+-label%3A%22DO-NOT-MERGE%22+label%3A%22CLA+signed%22

Here are the numbers:

* just open: 959
    is:open
* and with label "CLA signed": 900
    label:"CLA signed"
* and with label "awaiting merge": 169
    label:"awaiting merge"
* and without label "DO-NOT-MERGE": 152
    -label:"DO-NOT-MERGE"
* with CI is happy ;-): 112
    status:success
* with review is approved: 49
    review:approved

total: 49 PRs could be merged.

I was really surprised by this tool, (doc:
https://help.github.com/articles/searching-issues-and-pull-requests/)

But I was thinking about one thing, how can I help the core-devs to
merge these PRs?

Each week, I can send a report to this ML with the "mergeable" PRs.
This kind of report could be useful for you?


Have a nice day,

St?phane

-- 
St?phane Wirtel - https://wirtel.be - @matrixise

From brett at python.org  Fri Oct 19 14:42:00 2018
From: brett at python.org (Brett Cannon)
Date: Fri, 19 Oct 2018 11:42:00 -0700
Subject: [Python-Dev] Some PRs to merge?
In-Reply-To: <20181019165754.GA17454@xps>
References: <20181019165754.GA17454@xps>
Message-ID: <CAP1=2W5p51UJiY8z2UjP3T5qWUan_hL9a9xAsAAn-_VZX+pzxA@mail.gmail.com>

Of those 49 PRs, 18 are by core developers themselves, so 31 PRs are by
external contributors that seem ready to be merged.

There was a discussion at one point on core-workflow about changing the
default "needs" label for PRs by core devs which in this instance would
help with providing a search for PRs that are possibly very close to being
ready to be merged.

On Fri, 19 Oct 2018 at 09:58, Stephane Wirtel <stephane at wirtel.be> wrote:

> Hi all,
>
> How are you? I am fine ;-) and you?
>
> So, on this morning I was playing with the github interface and the
> pull requests of CPython and I have discovered the advanced search of
> Github and I think this one is really useful for us and certainly for
> the core-dev.
>
> So, I was interested by somes PRs.
>
> PRs with this status:
> * open
> * review is approved
> * status of the CI is 'success'
> * has labels "awaiting merge", "CLA signed" and -"DO-NOT-MERGE"
>
> total: 49 PRs
>
> In the GitHub interface, here is the criteria
>
> is:open is:pr review:approved status:success label:"awaiting merge"
> -label:"DO-NOT-MERGE" label:""LA signed""
>
> But if you want to see the result in your browser, just click on this link.
>
> https://github.com/python/cpython/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+review%3Aapproved+status%3Asuccess+label%3A%22awaiting+merge%22+-label%3A%22DO-NOT-MERGE%22+label%3A%22CLA+signed%22
>
> Here are the numbers:
>
> * just open: 959
>     is:open
> * and with label "CLA signed": 900
>     label:"CLA signed"
> * and with label "awaiting merge": 169
>     label:"awaiting merge"
> * and without label "DO-NOT-MERGE": 152
>     -label:"DO-NOT-MERGE"
> * with CI is happy ;-): 112
>     status:success
> * with review is approved: 49
>     review:approved
>
> total: 49 PRs could be merged.
>
> I was really surprised by this tool, (doc:
> https://help.github.com/articles/searching-issues-and-pull-requests/)
>
> But I was thinking about one thing, how can I help the core-devs to
> merge these PRs?
>
> Each week, I can send a report to this ML with the "mergeable" PRs.
> This kind of report could be useful for you?
>
>
> Have a nice day,
>
> St?phane
>
> --
> St?phane Wirtel - https://wirtel.be - @matrixise
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181019/be5a36c1/attachment.html>

From mike at selik.org  Fri Oct 19 16:05:56 2018
From: mike at selik.org (Michael Selik)
Date: Fri, 19 Oct 2018 13:05:56 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc0X95an_kOC-QQ7g_VhUKUioM=kMu+=2N1ZLXhWXThhyw@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
 <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
 <CAOTb1wc_gTuTC2Tmi6NYYLpzrtH6V=gKOjK5eF25QLC-kPkWgA@mail.gmail.com>
 <CAOh-wgBc91w_D5OmryTdkp3tUk7BqQ=HgDEiUF99LG9fvCLoqA@mail.gmail.com>
 <CA+LW3i0uoUhZvW9DeGCo9TT73JgManuf9kBmC67HHksNkwRavg@mail.gmail.com>
 <CAFbyWc0X95an_kOC-QQ7g_VhUKUioM=kMu+=2N1ZLXhWXThhyw@mail.gmail.com>
Message-ID: <CAGgTfkMoMrURkDNM9Vz=naeX+SJWNr=uknY91kdYwFqUCS49BQ@mail.gmail.com>

On Fri, Oct 19, 2018 at 5:01 AM Sean Harrington <seanharr11 at gmail.com>
wrote:

> I like the idea to extend the Pool class [to optimize the case when only
> one function is passed to the workers].
>

Why would this keep the same interface as the Pool class? If its workers
are restricted to calling only one function, that should be passed into the
Pool constructor. The map and apply methods would then only receive that
function's args and not the function itself. You're also trying to avoid
the initializer/globals pattern, so you could eliminate that parameter from
the Pool constructor. In fact, it sounds more like you'd want a function
than a class. You can call it "procmap" or similar. That's code I've
written more than once.

    results = poolmap(func, iterable, processes=cpu_count())

The nuance is that, since there's no explicit context manager, you'll want
to ensure the pool is shut down after all the tasks are finished, even if
the results generator hasn't been fully consumed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181019/1e3d63d7/attachment.html>

From vstinner at redhat.com  Fri Oct 19 21:08:35 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Sat, 20 Oct 2018 03:08:35 +0200
Subject: [Python-Dev] Some PRs to merge?
In-Reply-To: <20181019165754.GA17454@xps>
References: <20181019165754.GA17454@xps>
Message-ID: <CA+3bQGGaseAZ5KAZwn2ytQYjf4mFbVS-e9ic7nJyA5uXmLYLrA@mail.gmail.com>

Le ven. 19 oct. 2018 ? 19:01, Stephane Wirtel <stephane at wirtel.be> a ?crit :
> total: 49 PRs
> is:open is:pr review:approved status:success label:"awaiting merge" -label:"DO-NOT-MERGE" label:""LA signed""

I merged many PRs and closed a few (2 if I recall correctly). Your
query now counts 24 PRs.

Victor

From stephane at wirtel.be  Sat Oct 20 05:32:11 2018
From: stephane at wirtel.be (Stephane Wirtel)
Date: Sat, 20 Oct 2018 11:32:11 +0200
Subject: [Python-Dev] Some PRs to merge?
In-Reply-To: <CA+3bQGGaseAZ5KAZwn2ytQYjf4mFbVS-e9ic7nJyA5uXmLYLrA@mail.gmail.com>
References: <20181019165754.GA17454@xps>
 <CA+3bQGGaseAZ5KAZwn2ytQYjf4mFbVS-e9ic7nJyA5uXmLYLrA@mail.gmail.com>
Message-ID: <20181020093211.GA29503@xps>

On 10/20, Victor Stinner wrote:
>Le ven. 19 oct. 2018 ? 19:01, Stephane Wirtel <stephane at wirtel.be> a ?crit :
>> total: 49 PRs
>> is:open is:pr review:approved status:success label:"awaiting merge" -label:"DO-NOT-MERGE" label:""LA signed""
>
>I merged many PRs and closed a few (2 if I recall correctly). Your
>query now counts 24 PRs.
Really nice for your job, I think the contributors will appreciate a
lot.
>
>Victor

I would like to know if you are interested by this kind of reports. I
can interact with the GraphQL api of GitHub and just provide the report
via email to python-dev at python.org

Have a nice day and thank you for your merges.

St?phane

-- 
St?phane Wirtel - https://wirtel.be - @matrixise

From steve at holdenweb.com  Sat Oct 20 06:25:33 2018
From: steve at holdenweb.com (Steve Holden)
Date: Sat, 20 Oct 2018 11:25:33 +0100
Subject: [Python-Dev] Some PRs to merge?
In-Reply-To: <20181020093211.GA29503@xps>
References: <20181019165754.GA17454@xps>
 <CA+3bQGGaseAZ5KAZwn2ytQYjf4mFbVS-e9ic7nJyA5uXmLYLrA@mail.gmail.com>
 <20181020093211.GA29503@xps>
Message-ID: <CAMofdRDqoXnhP15fU+XMA0R7Bpocdm_rZhqOxB6vY1hkx7UjNg@mail.gmail.com>

This is terrific work. We all know that the best way to encourage
contributors is to use their usable contributions. Thank you very much,
Stephane and Victor (again)!

Steve Holden

On Sat, Oct 20, 2018 at 10:32 AM, Stephane Wirtel <stephane at wirtel.be>
wrote:

> On 10/20, Victor Stinner wrote:
>
>> Le ven. 19 oct. 2018 ? 19:01, Stephane Wirtel <stephane at wirtel.be> a
>> ?crit :
>>
>>> total: 49 PRs
>>> is:open is:pr review:approved status:success label:"awaiting merge"
>>> -label:"DO-NOT-MERGE" label:""LA signed""
>>>
>>
>> I merged many PRs and closed a few (2 if I recall correctly). Your
>> query now counts 24 PRs.
>>
> Really nice for your job, I think the contributors will appreciate a
> lot.
>
>>
>> Victor
>>
>
> I would like to know if you are interested by this kind of reports. I
> can interact with the GraphQL api of GitHub and just provide the report
> via email to python-dev at python.org
>
> Have a nice day and thank you for your merges.
>
> St?phane
>
> --
> St?phane Wirtel - https://wirtel.be - @matrixise
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve%
> 40holdenweb.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181020/77a339f9/attachment-0001.html>

From storchaka at gmail.com  Sat Oct 20 07:06:49 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 20 Oct 2018 14:06:49 +0300
Subject: [Python-Dev] The future of the wchar_t cache
Message-ID: <pqf246$tv3$1@blaine.gmane.org>

Currently the PyUnicode object contains two caches: for UTF-8 
representation and for wchar_t representation. They are needed not for 
optimization but for supporting C API which returns borrowed references 
for such representations.

The UTF-8 cache always was in unicode objects (but in Python 2 it was 
not a UTF-8 cache, but a 8-bit representation cache). Initially it was 
needed for compatibility with 8-bit str, for implementing the "s" and 
"z" format units in PyArg_Parse(). Now it is used also for 
PyUnicode_AsUTF8() and PyUnicode_AsUTF8AndSize().

The wchar_t cache was added with PEP 393 in 3.3 as a replacement for the 
former Py_UNICODE representation. Now Py_UNICODE is defined as an alias 
of wchar_t, and the C API which returned a pointer to Py_UNICODE content 
returns now a pointer to the cached wchar_t representation. It is the 
"u" and "Z" format units in PyArg_Parse(), PyUnicode_AsUnicode(), 
PyUnicode_AsUnicodeAndSize(), PyUnicode_GET_SIZE(), 
PyUnicode_GET_DATA_SIZE(), PyUnicode_AS_UNICODE(), PyUnicode_AS_DATA().

All this increase the size of the unicode object. It includes the 
constant overhead of additional pointer and size fields, and the 
overhead of the cached representation proportional to the string length. 
The following table contains number of bytes per character for different 
kinds, with and without filling specified caches.

        raw  +utf8     +wchar_t       +utf8+wchar_t
                    Windows  Linux   Windows  Linux
ASCII   1     1       3       5        3       5
UCS1    1    2-3      3       5       4-5     6-7
UCS2    2    3-5      2       6       3-5     7-9
UCS4    4    5-8     6-8      4       7-12    5-8

There is also a new C API added in 3.3 for getting wchar_t 
representation without using the cache: PyUnicode_AsWideChar() and 
PyUnicode_AsWideCharString(). Currently it uses the cache, this has 
benefits and disadvantages.

Old Py_UNICODE based API is deprecated, and will be removed eventually.
I want to ask about the future of the wchar_t cache. Is the benefit of 
caching the wchar_t representation larger the disadvantage of spending 
more memory? The wchar_t representation is so natural for Windows API as 
the UTF8 representation for POSIX API. But in all other cases it is just 
waste of memory. Are there reasons of keeping the wchar_t cache after 
removing the deprecated API?

I have rewrote PyUnicode_AsWideChar() and PyUnicode_AsWideCharString(). 
They were implemented via the old Py_UNICODE based API, and now they 
don't use deprecated functions. They still use the wchar_t cache if it 
was created by previous use of the deprecated API, but don't create it 
themselves. Is this the correct decision?

https://bugs.python.org/issue30863


From storchaka at gmail.com  Sat Oct 20 07:12:29 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 20 Oct 2018 14:12:29 +0300
Subject: [Python-Dev] Some PRs to merge?
In-Reply-To: <CA+3bQGGaseAZ5KAZwn2ytQYjf4mFbVS-e9ic7nJyA5uXmLYLrA@mail.gmail.com>
References: <20181019165754.GA17454@xps>
 <CA+3bQGGaseAZ5KAZwn2ytQYjf4mFbVS-e9ic7nJyA5uXmLYLrA@mail.gmail.com>
Message-ID: <pqf2eq$hlf$1@blaine.gmane.org>

20.10.18 04:08, Victor Stinner ????:
> Le ven. 19 oct. 2018 ? 19:01, Stephane Wirtel <stephane at wirtel.be> a ?crit :
>> total: 49 PRs
>> is:open is:pr review:approved status:success label:"awaiting merge" -label:"DO-NOT-MERGE" label:""LA signed""
> 
> I merged many PRs and closed a few (2 if I recall correctly). Your
> query now counts 24 PRs.

Thank you Victor! I prefer to merge my PRs and PRs assigned to me 
myself, but I am not sure that I would merge all PRs that can be merged 
in the nearest future. ;)


From songofacandy at gmail.com  Sat Oct 20 08:32:20 2018
From: songofacandy at gmail.com (INADA Naoki)
Date: Sat, 20 Oct 2018 21:32:20 +0900
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <pqf246$tv3$1@blaine.gmane.org>
References: <pqf246$tv3$1@blaine.gmane.org>
Message-ID: <CAEfz+TwHGah1HibVxOfabRE0D=etv6DENTEDEF1om8b6L4Wv6g@mail.gmail.com>

+1 to remove wchar_t cache.

I hope we can remove it at Python 3.9 or 3.10.

From stefan_ml at behnel.de  Sat Oct 20 09:01:52 2018
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 20 Oct 2018 15:01:52 +0200
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <pqf246$tv3$1@blaine.gmane.org>
References: <pqf246$tv3$1@blaine.gmane.org>
Message-ID: <pqf8rt$iat$1@blaine.gmane.org>

Serhiy Storchaka schrieb am 20.10.2018 um 13:06:
> Currently the PyUnicode object contains two caches: for UTF-8
> representation and for wchar_t representation. They are needed not for
> optimization but for supporting C API which returns borrowed references for
> such representations.
> 
> The UTF-8 cache always was in unicode objects (but in Python 2 it was not a
> UTF-8 cache, but a 8-bit representation cache). Initially it was needed for
> compatibility with 8-bit str, for implementing the "s" and "z" format units
> in PyArg_Parse(). Now it is used also for PyUnicode_AsUTF8() and
> PyUnicode_AsUTF8AndSize().
> 
> The wchar_t cache was added with PEP 393 in 3.3 as a replacement for the
> former Py_UNICODE representation. Now Py_UNICODE is defined as an alias of
> wchar_t, and the C API which returned a pointer to Py_UNICODE content
> returns now a pointer to the cached wchar_t representation. It is the "u"
> and "Z" format units in PyArg_Parse(), PyUnicode_AsUnicode(),
> PyUnicode_AsUnicodeAndSize(), PyUnicode_GET_SIZE(),
> PyUnicode_GET_DATA_SIZE(), PyUnicode_AS_UNICODE(), PyUnicode_AS_DATA().
> 
> All this increase the size of the unicode object. It includes the constant
> overhead of additional pointer and size fields, and the overhead of the
> cached representation proportional to the string length. The following
> table contains number of bytes per character for different kinds, with and
> without filling specified caches.
> 
> ?????? raw? +utf8???? +wchar_t?????? +utf8+wchar_t
> ?????????????????? Windows? Linux?? Windows? Linux
> ASCII?? 1???? 1?????? 3?????? 5??????? 3?????? 5
> UCS1??? 1??? 2-3????? 3?????? 5?????? 4-5???? 6-7
> UCS2??? 2??? 3-5????? 2?????? 6?????? 3-5???? 7-9
> UCS4??? 4??? 5-8???? 6-8????? 4?????? 7-12??? 5-8
> 
> There is also a new C API added in 3.3 for getting wchar_t representation
> without using the cache: PyUnicode_AsWideChar() and
> PyUnicode_AsWideCharString(). Currently it uses the cache, this has
> benefits and disadvantages.
> 
> Old Py_UNICODE based API is deprecated, and will be removed eventually.
> I want to ask about the future of the wchar_t cache. Is the benefit of
> caching the wchar_t representation larger the disadvantage of spending more
> memory? The wchar_t representation is so natural for Windows API as the
> UTF8 representation for POSIX API. But in all other cases it is just waste
> of memory. Are there reasons of keeping the wchar_t cache after removing
> the deprecated API?

I'd be happy to get rid of it. But regarding the use under Windows, I
wonder if there's interest in keeping it as a special Windows-only feature,
e.g. to speed up the data exchange with the Win32 APIs. I guess it would
have to provide a visible (performance?) advantage to justify such special
casing over the code removal.

Stefan


From steve.dower at python.org  Sat Oct 20 11:58:58 2018
From: steve.dower at python.org (Steve Dower)
Date: Sat, 20 Oct 2018 11:58:58 -0400
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <pqf8rt$iat$1@blaine.gmane.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
Message-ID: <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>

On 20Oct2018 0901, Stefan Behnel wrote:
> I'd be happy to get rid of it. But regarding the use under Windows, I
> wonder if there's interest in keeping it as a special Windows-only feature,
> e.g. to speed up the data exchange with the Win32 APIs. I guess it would
> have to provide a visible (performance?) advantage to justify such special
> casing over the code removal.

I think these cases would be just as well served by maintaining the 
original UCS-2 representation even if the maximum character fits into 
UCS-1, and only do the conversion when Python copies the string into a 
new location.

I don't have numbers, but my instinct says the most impacted operations 
would be retrieving collections of strings from the OS (avoiding a 
scan/conversion for each one), comparisons against these collections 
(in-memory handling for hash/comparison of mismatched KIND), and passing 
some of these strings back to the OS (conversion back into UCS-2). This 
is basically a glob/fnmatch/stat sequence, and is the main real scenario 
I can think of where Python's overhead might become noticeable.

Another option that might be useful is some way to allow the UCS-1/4 <-> 
UCS-2 conversion to occur outside the GIL. Most of the time when we need 
to convert we're about to release the GIL (or have just recovered it), 
so even without the cache we could probably recover some of the 
performance impact in parallelism. (That said, these are often tied up 
in conditions and generated code, so it may not be as easy to do this as 
retaining the original format.)

Some sort of tracing to see how often the cache is reused after being 
generated would be interesting, as well as how often the cache is being 
generated for a string that was originally in UCS-2 but we changed it to 
UCS-1.

Cheers,
Steve

From nad at python.org  Sat Oct 20 13:37:57 2018
From: nad at python.org (Ned Deily)
Date: Sat, 20 Oct 2018 13:37:57 -0400
Subject: [Python-Dev] [RELEASE] Python 3.7.1 and 3.6.7 are now available
Message-ID: <F1F2105E-707C-46E5-981E-52685CFAC658@python.org>

On behalf of the Python development community and the Python 3.7 release
team, we are pleased to announce the availability of Python 3.7.1. Python
3.7.1 is the first maintenance release of the newest feature release of
the Python language. You can find Python 3.7.1 here:
    https://www.python.org/downloads/release/python-371/

See the What?s New In Python 3.7 document for more information about the
many new features and optimizations included in the 3.7 series. Detailed
information about the changes made in 3.7.1 can be found in its change log:
    https://docs.python.org/3.7/whatsnew/3.7.html
    https://docs.python.org/3.7/whatsnew/changelog.html#python-3-7-1-final

We are also happy to announce the availability of Python 3.6.7, the next
maintenance release of Python 3.6:
    https://www.python.org/downloads/release/python-367/

Thanks to all of the many volunteers who help make Python Development and
these releases possible! Please consider supporting our efforts by
volunteering yourself or through organization contributions to the Python
Software Foundation.

--
  Ned Deily
  nad at python.org -- []


From tjreedy at udel.edu  Sat Oct 20 15:54:44 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 20 Oct 2018 15:54:44 -0400
Subject: [Python-Dev] [RELEASE] Python 3.7.1 and 3.6.7 are now available
In-Reply-To: <F1F2105E-707C-46E5-981E-52685CFAC658__23386.542502749$1540058215$gmane$org@python.org>
References: <F1F2105E-707C-46E5-981E-52685CFAC658__23386.542502749$1540058215$gmane$org@python.org>
Message-ID: <pqg122$m75$1@blaine.gmane.org>

On 10/20/2018 1:37 PM, Ned Deily wrote:

> We are also happy to announce the availability of Python 3.6.7, the next
> maintenance release of Python 3.6:
>      https://www.python.org/downloads/release/python-367/

Am I correct in thinking that there will be just one more maintenance 
release before going on security-only status?

-- 
Terry Jan Reedy


From nad at python.org  Sat Oct 20 17:24:27 2018
From: nad at python.org (Ned Deily)
Date: Sat, 20 Oct 2018 17:24:27 -0400
Subject: [Python-Dev] [RELEASE] Python 3.7.1 and 3.6.7 are now available
In-Reply-To: <pqg122$m75$1@blaine.gmane.org>
References: <F1F2105E-707C-46E5-981E-52685CFAC658__23386.542502749$1540058215$gmane$org@python.org>
 <pqg122$m75$1@blaine.gmane.org>
Message-ID: <753138D5-CEE8-419E-ACA7-84C56F6579AE@python.org>

On Oct 20, 2018, at 15:54, Terry Reedy <tjreedy at udel.edu> wrote:
> 
> On 10/20/2018 1:37 PM, Ned Deily wrote:
> 
>> We are also happy to announce the availability of Python 3.6.7, the next
>> maintenance release of Python 3.6:
>>     https://www.python.org/downloads/release/python-367/
> 
> Am I correct in thinking that there will be just one more maintenance release before going on security-only status?

Yes, that's the published plan:
    https://www.python.org/dev/peps/pep-0494/

    3.6.8 schedule (2018-12)
    Final maintenance mode release, final binary releases.


    3.6.9 and beyond schedule
    Security fixes only, as needed, until 2021-12

--
  Ned Deily
  nad at python.org -- []


From facundobatista at gmail.com  Sat Oct 20 17:57:11 2018
From: facundobatista at gmail.com (Facundo Batista)
Date: Sat, 20 Oct 2018 18:57:11 -0300
Subject: [Python-Dev] Servicing pypi.python.org
In-Reply-To: <C560340E-7722-49E0-9F98-C7D81BF25375@stufft.io>
References: <CAM09pzSJe2Zy6C+FhQRGTSnKLBEpPKuR2HQzLCTeSSRB7HP27Q@mail.gmail.com>
 <C560340E-7722-49E0-9F98-C7D81BF25375@stufft.io>
Message-ID: <CAM09pzS8r0ZpYPuM4g5eY191uXyKXzxYRd804M9FDRx90qRRgA@mail.gmail.com>

El jue., 18 de oct. de 2018 a la(s) 00:07, Donald Stufft
(donald at stufft.io) escribi?:

> tl;dr: can we have a (semi)permanent redirect from pypi.python.org to pypi.org?
>
>
>
> This already exists:

Indeed, it works! I was in a place with a very crappy internet, but I
never suspected the issue I was getting was part of that.

Anyway, everything is fine! Thanks for the help and sorry for the noise :)

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org.ar/
Twitter: @facundobatista

From vstinner at redhat.com  Mon Oct 22 03:59:56 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 22 Oct 2018 09:59:56 +0200
Subject: [Python-Dev] Some PRs to merge?
In-Reply-To: <pqf2eq$hlf$1@blaine.gmane.org>
References: <20181019165754.GA17454@xps>
 <CA+3bQGGaseAZ5KAZwn2ytQYjf4mFbVS-e9ic7nJyA5uXmLYLrA@mail.gmail.com>
 <pqf2eq$hlf$1@blaine.gmane.org>
Message-ID: <CA+3bQGGHvtJ+0Djycp_-mRNjpYWUQso_xo=A8c3CLfuz2wZgvQ@mail.gmail.com>

Le sam. 20 oct. 2018 ? 13:15, Serhiy Storchaka <storchaka at gmail.com> a ?crit :
> Thank you Victor! I prefer to merge my PRs and PRs assigned to me
> myself, but I am not sure that I would merge all PRs that can be merged
> in the nearest future. ;)

Some PRs were blocked by me because I was nitpicking on something. I
decided that, nah, it's fine. It's better to merge these "not perfect"
PRs rather than leaving them die in review.

Many PRs were written by core developers but still not merged 6 months
after they have been approved, I'm not sure why. I decided to merge
them to reduce the queue of open pull requests. Pressing the [Merge]
button also means that I approve a PR. I tested manually some of these
PRs before merging them, to make sure that they work as expected ;-)

Victor

From vstinner at redhat.com  Mon Oct 22 04:09:43 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 22 Oct 2018 10:09:43 +0200
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <pqf8rt$iat$1@blaine.gmane.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
Message-ID: <CA+3bQGGtnJ233rxSfBQra0qqG3B48V58uNd9KNH0eHz42zf7Og@mail.gmail.com>

Hi Serhiy,

+1 to remove wchar_t cache. IMHO it wastes memory for no real performance gain.

By the way, can we start to schedule the *removal* of the Py_UNICODE
API? For example, decide when Py_DEPRECATED is used in the C API?
Should we start to deprecate when Python 2 reachs its end of life? Or
can we *remove* this API as soon as Python 2 is dead? (Please, don't
use "Python 4" as a milestone to introduce such backward incompatible
change.)


Le sam. 20 oct. 2018 ? 15:05, Stefan Behnel <stefan_ml at behnel.de> a ?crit :
> I'd be happy to get rid of it. But regarding the use under Windows, I
> wonder if there's interest in keeping it as a special Windows-only feature,
> e.g. to speed up the data exchange with the Win32 APIs. I guess it would
> have to provide a visible (performance?) advantage to justify such special
> casing over the code removal.

If someone wants to *keep* the cache, I would like to see:

* statistics on the cache usage: how many strings are converted to
wchar_t*? What is the percentage on all strings?
* what is the cache hit?
* what is the overhead of removing the cache? (on length 10, 100,
1000, 1 MB, 10 MB?)

When you look at the performance of the str type, it's common that
timings are smaller than 1 us. Each str function has been highly
optimized.

IMHO the conversion of a string to wchar_t* is cheap, especially
because I expect that all strings used with the Windows API are
shorter than 1,000 characters.

Victor

From vstinner at redhat.com  Mon Oct 22 04:13:06 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 22 Oct 2018 10:13:06 +0200
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
Message-ID: <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>

Le sam. 20 oct. 2018 ? 18:02, Steve Dower <steve.dower at python.org> a ?crit :
> I don't have numbers, but my instinct says the most impacted operations
> would be retrieving collections of strings from the OS (avoiding a
> scan/conversion for each one), comparisons against these collections
> (in-memory handling for hash/comparison of mismatched KIND), and passing
> some of these strings back to the OS (conversion back into UCS-2). This
> is basically a glob/fnmatch/stat sequence, and is the main real scenario
> I can think of where Python's overhead might become noticeable.

Use os.scandir() to avoid stat :-)

For code like "for name in os.listdir(): open(name): ...." (replace
listdir with scandir if you want to get file metadata), the cache is
useless, since the fresh string has to be converted to wchar_t*
anyway, and the cache is destroyed at the end of the loop iteration,
whereas the cache has never been used...

I'm not saying that the cache is useless. I just doubt that it's so
common that it really provide any performance benefit.

Victor

From encukou at gmail.com  Mon Oct 22 07:41:50 2018
From: encukou at gmail.com (Petr Viktorin)
Date: Mon, 22 Oct 2018 13:41:50 +0200
Subject: [Python-Dev] wininst-*.exe files in Lib/distutils/command
In-Reply-To: <CAFQvZEPRFqhia_y0f9CGpRgTBPsRiRVkiK=Ek4MphBYPe-aX9g@mail.gmail.com>
References: <CAFQvZEPmGVscWNYF+Bng61LMdwpEgkjLqyhG3nogBedkG_u2KQ@mail.gmail.com>
 <CAKJDb-ODjg6YkXOooz2crKKaz4n9u6gxgEVp_Z=qEf8U2+xLCQ@mail.gmail.com>
 <5e5e14f2-b4f2-d904-ffc4-b825794d1600@gmail.com>
 <CAFQvZEPRFqhia_y0f9CGpRgTBPsRiRVkiK=Ek4MphBYPe-aX9g@mail.gmail.com>
Message-ID: <52759e82-1fea-0953-439c-4fc4018d0bc0@gmail.com>

On 10/18/18 7:44 PM, VanL wrote:
> Primarily for non-windows platforms, but I also think for Windows users 
> without any compilers or similar tools installed. There is also some 
> discussion of removing some of the older toolchain-specific versions 
> (leaving only -14), but that is a subject for another day.
> 
> Also I am not sure that bug report applies to 3.5/3.6/3.7.

It does.



> On Thu, Oct 18, 2018 at 11:26 AM Petr Viktorin <encukou at gmail.com 
> <mailto:encukou at gmail.com>> wrote:
> 
>     On 10/18/18 4:40 PM, Zachary Ware wrote:
>      > On Thu, Oct 18, 2018 at 9:09 AM VanL <van.lindberg at gmail.com
>     <mailto:van.lindberg at gmail.com>> wrote:
>      >> Hi all,
>      >>
>      >> I am looking into an issue associated with the wininst-*.exe
>     files in the distutils/command subdirectory. It looks like these are
>     the executable stubs used to create self-extracting zips for
>     installation - but I am not 100% sure. It also looks like they
>     include the calls to standard Windows functions to display the
>     installer window.
>      >>
>      >> I have a couple questions I need help with:
>      >> 1) Am I correct about the function, and if not, what are they?
>      >
>      > You are correct.? IIUC, they are checked in to allow creating those
>      > installers from non-Windows platforms.
> 
>     Is that the only reason for them?
>     At least on Linux, bdist_wininst does not work since at least Python
>     3.2, as it tries to use a Windows-only encoding internally.
>     https://bugs.python.org/issue10945
> 
>     If they're only there for non-Windows platforms, they're useless.
> 
>      >> 2) Where did these come from, and where is their source code?
>      >
>      > Source can be found here:
>      > https://github.com/python/cpython/tree/master/PC/bdist_wininst
>      >
>      > The individual checked-in .exe files were each originally built by
>      > whoever updated the Windows toolchain to a new version of MSVC
>      > (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
>      > ones were added by Thomas Heller, presumably using whatever the
>      > current toolchain(s) was (were) at the time).? A few of them have
>     been
>      > rebuilt after bug fixes in the source since they were added,
>     mostly by
>      > the same people, though I also see Mark Hammond and Raymond Hettinger
>      > in the history (and Georg Brandl via svnmerge).? I notice that there
>      > are a few very minor code cleanups (the three latest commits here
>      >
>     https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c)
>      > that have not made it into a rebuilt exe yet.
>      >
>      > FTR, we really ought to remove all but the 14.0 version from the
>      > master branch.? We don't support building Python with any toolchain
>      > older than 14.0 anymore, and the older toolchains are nigh impossible
>      > to find anymore anyway.
> 

From seanharr11 at gmail.com  Mon Oct 22 08:28:08 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Mon, 22 Oct 2018 08:28:08 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAGgTfkMoMrURkDNM9Vz=naeX+SJWNr=uknY91kdYwFqUCS49BQ@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
 <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
 <CAOTb1wc_gTuTC2Tmi6NYYLpzrtH6V=gKOjK5eF25QLC-kPkWgA@mail.gmail.com>
 <CAOh-wgBc91w_D5OmryTdkp3tUk7BqQ=HgDEiUF99LG9fvCLoqA@mail.gmail.com>
 <CA+LW3i0uoUhZvW9DeGCo9TT73JgManuf9kBmC67HHksNkwRavg@mail.gmail.com>
 <CAFbyWc0X95an_kOC-QQ7g_VhUKUioM=kMu+=2N1ZLXhWXThhyw@mail.gmail.com>
 <CAGgTfkMoMrURkDNM9Vz=naeX+SJWNr=uknY91kdYwFqUCS49BQ@mail.gmail.com>
Message-ID: <CAFbyWc1SLZTTqMgdttQJnU5+q3zLOujeHapOtVEQR9LWzGddHQ@mail.gmail.com>

Michael - the initializer/globals pattern still might be necessary if you
need to create an object AFTER a worker process has been instantiated (i.e.
a database connection). Further, the user may want to access all of the
niceties of Pool, like imap, imap_unordered, etc.  The goal (IMO) would be
to preserve an interface which many Python users have grown accustomed to,
and to allow them to access this optimization out-of-the-bag.

Having talked to folks at the Boston Python meetup, folks on my dev team,
and perusing stack overflow, this "instance method parallelization" is a
pretty common pattern that is often times a negative return on investment
for the developer, due to the implicit implementation detail of pickling
the function (and object) once per task.

Is anyone open to reviewing a PR concerning this optimization of Pool,
delivered as a subclass? This feature restricts the number of unique tasks
being executed by workers at once to 1, while allowing aggressive
subprocess-level function cacheing to prevent repeated
serialization/deserialization of large functions/closures. The use case is
s.t. the user only ever needs 1 call to Pool.map(func, ls) (or friends)
executing at once, when `func` has a non-trivial memory footprint.


On Fri, Oct 19, 2018 at 4:06 PM Michael Selik <mike at selik.org> wrote:

> On Fri, Oct 19, 2018 at 5:01 AM Sean Harrington <seanharr11 at gmail.com>
> wrote:
>
>> I like the idea to extend the Pool class [to optimize the case when only
>> one function is passed to the workers].
>>
>
> Why would this keep the same interface as the Pool class? If its workers
> are restricted to calling only one function, that should be passed into the
> Pool constructor. The map and apply methods would then only receive that
> function's args and not the function itself. You're also trying to avoid
> the initializer/globals pattern, so you could eliminate that parameter from
> the Pool constructor. In fact, it sounds more like you'd want a function
> than a class. You can call it "procmap" or similar. That's code I've
> written more than once.
>
>     results = poolmap(func, iterable, processes=cpu_count())
>
> The nuance is that, since there's no explicit context manager, you'll want
> to ensure the pool is shut down after all the tasks are finished, even if
> the results generator hasn't been fully consumed.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181022/039b9535/attachment.html>

From steve.dower at python.org  Mon Oct 22 09:08:33 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 22 Oct 2018 09:08:33 -0400
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
Message-ID: <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>

On 22Oct2018 0413, Victor Stinner wrote:
> For code like "for name in os.listdir(): open(name): ...." (replace
> listdir with scandir if you want to get file metadata), the cache is
> useless, since the fresh string has to be converted to wchar_t*
> anyway, and the cache is destroyed at the end of the loop iteration,
> whereas the cache has never been used...

Agreed the cache is useless here, but since the listdir() result came in 
as wchar_t we could keep it that way (assuming we'd only be changing it 
to char), and then there wouldn't have to be a conversion when we 
immediately pass it back to open().

That said, I spent some time yesterday converting the importlib cache to 
use scandir and separate caches for dir/file (to avoid the stat calls) 
and it made very little overall difference. I have to assume the string 
manipulation dominates. (Making DirEntry lazily calculate its .path had 
a bigger impact. Also, I didn't try to make Windows flush its own stat 
cache, and accessing warm files is much faster than cold ones.)

> I'm not saying that the cache is useless. I just doubt that it's so
> common that it really provide any performance benefit.

I think that it is mostly useless, but if we can transparently keep many 
strings "native" size, that will handle many of the useful cases such as 
the single-use pass-through scenario like above.

Cheers,
Steve

From vstinner at redhat.com  Mon Oct 22 09:13:12 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 22 Oct 2018 15:13:12 +0200
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
 <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
Message-ID: <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>

Le lun. 22 oct. 2018 ? 15:08, Steve Dower <steve.dower at python.org> a ?crit :
> Agreed the cache is useless here, but since the listdir() result came in
> as wchar_t we could keep it that way (assuming we'd only be changing it
> to char), and then there wouldn't have to be a conversion when we
> immediately pass it back to open().

Serhiy wants to remove the cache which should *reduce* Python memory
footprint on Windows.

You are proposing to fill the cache eagierly, that would increase the
Python memory footprint :-/ Your proposed change is an optimisation, a
benchmark is needed to see the benefit. I expect no significant
difference on benchmarks of https://pyperformance.readthedocs.io/ ...


> That said, I spent some time yesterday converting the importlib cache to
> use scandir and separate caches for dir/file (to avoid the stat calls)
> and it made very little overall difference. I have to assume the string
> manipulation dominates. (Making DirEntry lazily calculate its .path had
> a bigger impact. Also, I didn't try to make Windows flush its own stat
> cache, and accessing warm files is much faster than cold ones.)

I helped Ben Hoyt to design and implement his PEP 471 (os.scandir).
When the kernel filesystem cache is filled, the speedup of
os.scandir() is hard to notice. But when you work on a network
filesystem like NFS, the speedup is like 5x faster. NFS doesn't cache
stat() by default.

Victor

From steve.dower at python.org  Mon Oct 22 09:24:22 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 22 Oct 2018 09:24:22 -0400
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
 <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
 <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>
Message-ID: <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>

On 22Oct2018 0913, Victor Stinner wrote:
> Le lun. 22 oct. 2018 ? 15:08, Steve Dower <steve.dower at python.org> a ?crit :
>> Agreed the cache is useless here, but since the listdir() result came in
>> as wchar_t we could keep it that way (assuming we'd only be changing it
>> to char), and then there wouldn't have to be a conversion when we
>> immediately pass it back to open().
> 
> Serhiy wants to remove the cache which should *reduce* Python memory
> footprint on Windows.
> 
> You are proposing to fill the cache eagierly, that would increase the
> Python memory footprint :-/ Your proposed change is an optimisation, a
> benchmark is needed to see the benefit. I expect no significant
> difference on benchmarks of https://pyperformance.readthedocs.io/ ...

Yes, that's true. But "should reduce ... footprint" is also an 
optimisation that deserves a benchmark by that standard. Also, I'm 
proposing keeping the 'kind' as UCS-2 when the string is created from 
UCS-2 data that is likely to be used as UCS-2. We would not create the 
UCS-1 version in this case, so it's not the same as prefilling the 
cache, but it would cost a bit of memory in exchange for CPU. If slicing 
and concatentation between matching kinds also preserved the kind, a lot 
of path handling code could avoid back-and-forth conversions.

The import benchmarks ought to be improved on Windows by this new 
optimisation, as this is a prime case where we regularly convert strings 
from what the OS gave us into UCS-1 and back into what the OS expects. 
But if you don't run the benchmarks on all OS's, then sure, you won't 
see any difference :)

Cheers,
Steve

From vstinner at redhat.com  Mon Oct 22 09:28:32 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 22 Oct 2018 15:28:32 +0200
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
 <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
 <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>
 <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>
Message-ID: <CA+3bQGFejkXLvS2LhO_fyYb0eSHAZSuYQjtb2Y6tM1H6QCmY1Q@mail.gmail.com>

Le lun. 22 oct. 2018 ? 15:24, Steve Dower <steve.dower at python.org> a ?crit :
> Yes, that's true. But "should reduce ... footprint" is also an
> optimisation that deserves a benchmark by that standard.

pyperformance has a mode to mesure the memory usage (mostly the memory
peak) if someone wants to have a look.

> Also, I'm
> proposing keeping the 'kind' as UCS-2 when the string is created from
> UCS-2 data that is likely to be used as UCS-2.

Oh. That's a major change in the PEP 393 design. You would have to
modify many functions in CPython. Currently, the PEP 393 requires that
a string always use the most efficient storage, and many optimizations
and code paths rely on that assumptions.

I'm against this change.

Moreover, it's hard to guess how a string will be used later...

Victor

From storchaka at gmail.com  Mon Oct 22 09:34:29 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 22 Oct 2018 16:34:29 +0300
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <pqf8rt$iat$1@blaine.gmane.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
Message-ID: <pqkjh3$mub$1@blaine.gmane.org>

20.10.18 16:01, Stefan Behnel ????:
> But regarding the use under Windows, I
> wonder if there's interest in keeping it as a special Windows-only feature,
> e.g. to speed up the data exchange with the Win32 APIs. I guess it would
> have to provide a visible (performance?) advantage to justify such special
> casing over the code removal.

This is an interesting question, and we should found the answer right 
now. Should PyUnicode_AsWideChar() and PyUnicode_AsWideCharString() 
continue to attach the wchar_t representation to the unicode object on 
Windows? The cost is higher memory consumption and slower first time 
call. The benefit is faster following calls for the same object. 
Although they still need to copy the content, so the time is still O(N), 
just with smaller constant multiplier.


From steve.dower at python.org  Mon Oct 22 09:48:06 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 22 Oct 2018 09:48:06 -0400
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <CA+3bQGFejkXLvS2LhO_fyYb0eSHAZSuYQjtb2Y6tM1H6QCmY1Q@mail.gmail.com>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
 <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
 <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>
 <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>
 <CA+3bQGFejkXLvS2LhO_fyYb0eSHAZSuYQjtb2Y6tM1H6QCmY1Q@mail.gmail.com>
Message-ID: <43b60910-a654-acc6-0384-60fdfecd9278@python.org>

On 22Oct2018 0928, Victor Stinner wrote:
>> Also, I'm
>> proposing keeping the 'kind' as UCS-2 when the string is created from
>> UCS-2 data that is likely to be used as UCS-2.
> 
> Oh. That's a major change in the PEP 393 design. You would have to
> modify many functions in CPython. Currently, the PEP 393 requires that
> a string always use the most efficient storage, and many optimizations
> and code paths rely on that assumptions.

I don't know that it requires that many modifications - those functions 
already have to handle UCS-2 content anyway (e.g. if I get a path from 
scandir() that includes a non-ASCII character), and they're only using 
the assumption of most efficient storage to determine the resulting 
storage size of a string operation (which I'm proposing should also be 
UCS-2 when the source strings are UCS-2, since that's the best indicator 
we have that it'll be used as UCS-2 later, as well as being the current 
implementation :) ).

> I'm against this change.
> 
> Moreover, it's hard to guess how a string will be used later...

Agreed. There are some heuristics we can use, but it's definitely only a 
guess. That's the nature of this problem - guessing that it *won't* be 
used as UCS-2 later on is also a guess.

Cheers,
Steve

From storchaka at gmail.com  Mon Oct 22 09:57:40 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 22 Oct 2018 16:57:40 +0300
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <CA+3bQGGtnJ233rxSfBQra0qqG3B48V58uNd9KNH0eHz42zf7Og@mail.gmail.com>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <CA+3bQGGtnJ233rxSfBQra0qqG3B48V58uNd9KNH0eHz42zf7Og@mail.gmail.com>
Message-ID: <pqkksh$hi$1@blaine.gmane.org>

22.10.18 11:09, Victor Stinner ????:
> +1 to remove wchar_t cache. IMHO it wastes memory for no real performance gain.
> 
> By the way, can we start to schedule the *removal* of the Py_UNICODE
> API? For example, decide when Py_DEPRECATED is used in the C API?
> Should we start to deprecate when Python 2 reachs its end of life? Or
> can we *remove* this API as soon as Python 2 is dead? (Please, don't
> use "Python 4" as a milestone to introduce such backward incompatible
> change.)

Such removal is scheduled on 4.0. But since currently there are no any 
plans for Python 4, I think we should schedule some concrete date.

Removing the C API is a major breakage, we should prepare it carefully.

1. Make Py_DEPRECATED working on Windows. [1] Unfortunately this 
requires breaking its interface. It needs to be inserted before the 
function declaration, not after it.

[1] https://bugs.python.org/issue33407

2. Add Py_DEPRECATED to the rest of Py_UNICODE based C API 
(PyUnicode_AsUnicode(), PyUnicode_AsUnicodeAndSize(), etc).

3. Some macros like PyUnicode_AS_UNICODE() should be converted to 
functions for using compile-time warnings.

3. Would be nice to make the "u" format unit in PyArg_ParseTuple() and 
similar functions producing a compile time warning. Don't know if it is 
possible, but compilers are able to check format strings for printf-like 
functions.

4. Add a configuration option for removing the cache at compile time. 
This will help us to find all C API that depends on it.

5. Add tests for all C API (I'm working on this).

... two or more releases later ...

6. Make all deprecated C API functions emitting a DeprecationWarning at 
runtime.

... two or more releases later ...

7. Make all deprecated C API functions raising an error and remove their 
declarations from headers. Remove the wchar_t cache and legacy 
representations. Make PyUnicode_Ready() a no-op.

So if we implement items 1-5 in 3.8, we could get rid of this legacy in 
3.12.


From storchaka at gmail.com  Mon Oct 22 10:07:10 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 22 Oct 2018 17:07:10 +0300
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
 <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
 <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>
 <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>
Message-ID: <pqklec$ec8$1@blaine.gmane.org>

22.10.18 16:24, Steve Dower ????:
> Yes, that's true. But "should reduce ... footprint" is also an 
> optimisation that deserves a benchmark by that standard. Also, I'm 
> proposing keeping the 'kind' as UCS-2 when the string is created from 
> UCS-2 data that is likely to be used as UCS-2. We would not create the 
> UCS-1 version in this case, so it's not the same as prefilling the 
> cache, but it would cost a bit of memory in exchange for CPU. If slicing 
> and concatentation between matching kinds also preserved the kind, a lot 
> of path handling code could avoid back-and-forth conversions.

Oh, I afraid this will complicate the whole code of unicodeobject.c (and 
several other files) a much and can introduce a lot of subtle bugs.

For example, when you search a UCS2 string in a UCS1 string, the current 
code returns the result fast, because a UCS1 string can't contain codes 
 > 0xff, and a UCS2 string should contain codes > 0xff. And there are 
many such assumptions.


From steve.dower at python.org  Mon Oct 22 10:47:15 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 22 Oct 2018 10:47:15 -0400
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <pqklec$ec8$1@blaine.gmane.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
 <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
 <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>
 <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>
 <pqklec$ec8$1@blaine.gmane.org>
Message-ID: <c1007ef1-0557-1880-e404-7d5e433b6def@python.org>

On 22Oct2018 1007, Serhiy Storchaka wrote:
> 22.10.18 16:24, Steve Dower ????:
>> Yes, that's true. But "should reduce ... footprint" is also an 
>> optimisation that deserves a benchmark by that standard. Also, I'm 
>> proposing keeping the 'kind' as UCS-2 when the string is created from 
>> UCS-2 data that is likely to be used as UCS-2. We would not create the 
>> UCS-1 version in this case, so it's not the same as prefilling the 
>> cache, but it would cost a bit of memory in exchange for CPU. If 
>> slicing and concatentation between matching kinds also preserved the 
>> kind, a lot of path handling code could avoid back-and-forth conversions.
> 
> Oh, I afraid this will complicate the whole code of unicodeobject.c (and 
> several other files) a much and can introduce a lot of subtle bugs.
> 
> For example, when you search a UCS2 string in a UCS1 string, the current 
> code returns the result fast, because a UCS1 string can't contain codes 
>  > 0xff, and a UCS2 string should contain codes > 0xff. And there are 
> many such assumptions.

That doesn't change though, as we're only ever expanding the range. So 
searching a UCS2 string in a UCS2 string that doesn't contain any actual 
UCS2 characters is the only case that would be affected, and whether 
that case occurs more than the UCS2->UCS1->UCS2 conversion case is 
something we can measure (but I'd be surprised if substring searches 
occur more frequently than OS conversions).

Currently, unicode_compare_eq exits early when the kinds do not match, 
and that would be a problem (but is also easily fixable). But other 
string operations already handle mismatched kinds.

Cheers,
Steve

From mike at selik.org  Mon Oct 22 14:00:42 2018
From: mike at selik.org (Michael Selik)
Date: Mon, 22 Oct 2018 11:00:42 -0700
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAFbyWc1SLZTTqMgdttQJnU5+q3zLOujeHapOtVEQR9LWzGddHQ@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
 <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
 <CAOTb1wc_gTuTC2Tmi6NYYLpzrtH6V=gKOjK5eF25QLC-kPkWgA@mail.gmail.com>
 <CAOh-wgBc91w_D5OmryTdkp3tUk7BqQ=HgDEiUF99LG9fvCLoqA@mail.gmail.com>
 <CA+LW3i0uoUhZvW9DeGCo9TT73JgManuf9kBmC67HHksNkwRavg@mail.gmail.com>
 <CAFbyWc0X95an_kOC-QQ7g_VhUKUioM=kMu+=2N1ZLXhWXThhyw@mail.gmail.com>
 <CAGgTfkMoMrURkDNM9Vz=naeX+SJWNr=uknY91kdYwFqUCS49BQ@mail.gmail.com>
 <CAFbyWc1SLZTTqMgdttQJnU5+q3zLOujeHapOtVEQR9LWzGddHQ@mail.gmail.com>
Message-ID: <CAGgTfkPCM5ws2fLPYOOFgfw5UQ91PB7=HHtFL5kqkB=PnexBug@mail.gmail.com>

This thread seems more appropriate for python-ideas than python-dev.


On Mon, Oct 22, 2018 at 5:28 AM Sean Harrington <seanharr11 at gmail.com>
wrote:

> Michael - the initializer/globals pattern still might be necessary if you
> need to create an object AFTER a worker process has been instantiated (i.e.
> a database connection).
>

You said you wanted to avoid the initializer/globals pattern and have such
things as database connections in the defaults or closure of the
task-function, or the bound instance, no? Did I misunderstand?


Further, the user may want to access all of the niceties of Pool, like
> imap, imap_unordered, etc.  The goal (IMO) would be to preserve an
> interface which many Python users have grown accustomed to, and to allow
> them to access this optimization out-of-the-bag.
>

You just said that the dominant use-case was mapping a single
task-function. It sounds like we're talking past each other in some way.
It'll help to have a concrete example of a case that satisfies all the
characteristics you've described: (1) no globals used for communication
between initializer and task-functions; (2) single task-function, mapped
once; (3) an instance-method as task-function, causing a large
serialization burden; and (4) did I miss anything?



> Having talked to folks at the Boston Python meetup, folks on my dev team,
> and perusing stack overflow, this "instance method parallelization" is a
> pretty common pattern that is often times a negative return on investment
> for the developer, due to the implicit implementation detail of pickling
> the function (and object) once per task.
>

I believe you.


> Is anyone open to reviewing a PR concerning this optimization of Pool,
> delivered as a subclass? This feature restricts the number of unique tasks
> being executed by workers at once to 1, while allowing aggressive
> subprocess-level function cacheing to prevent repeated
> serialization/deserialization of large functions/closures. The use case is
> s.t. the user only ever needs 1 call to Pool.map(func, ls) (or friends)
> executing at once, when `func` has a non-trivial memory footprint.
>

You're quite eager to have this PR merged. I understand that. However, it's
reasonable to take some time to discuss the design of what you're
proposing. You don't need it in the stdlib to get your own work done, nor
to share it with others.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181022/2bbb6341/attachment.html>

From seanharr11 at gmail.com  Mon Oct 22 15:12:45 2018
From: seanharr11 at gmail.com (Sean Harrington)
Date: Mon, 22 Oct 2018 15:12:45 -0400
Subject: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension -
 Pass Data to Workers w/o Globals
In-Reply-To: <CAGgTfkPCM5ws2fLPYOOFgfw5UQ91PB7=HHtFL5kqkB=PnexBug@mail.gmail.com>
References: <CAFbyWc38KtWJ8jP2-KXDhoLsaPYTLCgiuVNrMGOVRyuqnvDt7w@mail.gmail.com>
 <CAGgTfkNt0JZD3UPpQRA7U9m75Y_OCZMJb+S+3mVxF=DGZGYwKg@mail.gmail.com>
 <CAFbyWc2w09SSCktQZzG_MYawdH+zYd4GufPrmShS58tN97WTNQ@mail.gmail.com>
 <CAGgTfkPYmfjHfHOvTF=-=wyjBHhou3=EsxHmCa6NB+pHHhiiZQ@mail.gmail.com>
 <CAFbyWc1wORX7kLUfNTxct8SQ-eQcH4Vt2s_xegZNf1M8e5=P+w@mail.gmail.com>
 <CAPJVwBkkyKvbWX6kiiqA=hw9hc3NaoEbUuzk7E2faLqLPNte0w@mail.gmail.com>
 <CAFbyWc0GW5_h6pCRJwh1c99x17piazf6t-ZRuCGcWqWaohmv6Q@mail.gmail.com>
 <20181012144847.7280967b@fsol>
 <CAPJVwBm6qnm6LU_1PZz-LS0kBbbVtxhGCPU8o7r9t7jw8hJWww@mail.gmail.com>
 <CAFbyWc0J2ean7d-JbkFHu9SOEATc5s9i2XJ6vTTxxm2A_NZ1Qw@mail.gmail.com>
 <CAGgTfkNP3aGvW=1w7QgYiNbcNuQumcd5=uTCpqAaqL5Yo=f38A@mail.gmail.com>
 <CAFbyWc0DGJsdZUNLiLWB3WDFUgFjFyczR-V6KB1ikF5wwrO3Pw@mail.gmail.com>
 <CAGgTfkMGPh7Xw-eqOjKYHyhuNg29E35D5Fz7jwhhgm7QC_7tFQ@mail.gmail.com>
 <CAFbyWc1Fx0NwKcX=BwVxO_wHyP8J=DbtEYna90S1CjjACvTMaQ@mail.gmail.com>
 <CAGgTfkPoFXV3h5BqhvfwP-TeMPB8xWG_LURKQSnxToQgm6di0w@mail.gmail.com>
 <CAOTb1wc_gTuTC2Tmi6NYYLpzrtH6V=gKOjK5eF25QLC-kPkWgA@mail.gmail.com>
 <CAOh-wgBc91w_D5OmryTdkp3tUk7BqQ=HgDEiUF99LG9fvCLoqA@mail.gmail.com>
 <CA+LW3i0uoUhZvW9DeGCo9TT73JgManuf9kBmC67HHksNkwRavg@mail.gmail.com>
 <CAFbyWc0X95an_kOC-QQ7g_VhUKUioM=kMu+=2N1ZLXhWXThhyw@mail.gmail.com>
 <CAGgTfkMoMrURkDNM9Vz=naeX+SJWNr=uknY91kdYwFqUCS49BQ@mail.gmail.com>
 <CAFbyWc1SLZTTqMgdttQJnU5+q3zLOujeHapOtVEQR9LWzGddHQ@mail.gmail.com>
 <CAGgTfkPCM5ws2fLPYOOFgfw5UQ91PB7=HHtFL5kqkB=PnexBug@mail.gmail.com>
Message-ID: <CAFbyWc1wy9JnzesE438W42g7MkNE3e6AZ1EDKBgka=zLmetLwg@mail.gmail.com>

On Mon, Oct 22, 2018 at 2:01 PM Michael Selik <mike at selik.org> wrote:

> This thread seems more appropriate for python-ideas than python-dev.
>
>

> On Mon, Oct 22, 2018 at 5:28 AM Sean Harrington <seanharr11 at gmail.com>
> wrote:
>
>> Michael - the initializer/globals pattern still might be necessary if you
>> need to create an object AFTER a worker process has been instantiated (i.e.
>> a database connection).
>>
>
> You said you wanted to avoid the initializer/globals pattern and have such
> things as database connections in the defaults or closure of the
> task-function, or the bound instance, no? Did I misunderstand?
>
>
> Further, the user may want to access all of the niceties of Pool, like
>> imap, imap_unordered, etc.  The goal (IMO) would be to preserve an
>> interface which many Python users have grown accustomed to, and to allow
>> them to access this optimization out-of-the-bag.
>>
>
> You just said that the dominant use-case was mapping a single
> task-function. It sounds like we're talking past each other in some way.
> It'll help to have a concrete example of a case that satisfies all the
> characteristics you've described: (1) no globals used for communication
> between initializer and task-functions; (2) single task-function, mapped
> once; (3) an instance-method as task-function, causing a large
> serialization burden; and (4) did I miss anything?
>

You're right, it's really only use cases (2) and (3) that define this spec.
However, the case for subclassing really boils down to the "free"
inheritance of the public methods of Pool (map, imap, imap_unordered,
etc...).  Why exclude these (by implementing "procmap()") if we get this
great return with such little investment?

>
>
>
>> Having talked to folks at the Boston Python meetup, folks on my dev team,
>> and perusing stack overflow, this "instance method parallelization" is a
>> pretty common pattern that is often times a negative return on investment
>> for the developer, due to the implicit implementation detail of pickling
>> the function (and object) once per task.
>>
>
> I believe you.
>
>
>> Is anyone open to reviewing a PR concerning this optimization of Pool,
>> delivered as a subclass? This feature restricts the number of unique tasks
>> being executed by workers at once to 1, while allowing aggressive
>> subprocess-level function cacheing to prevent repeated
>> serialization/deserialization of large functions/closures. The use case is
>> s.t. the user only ever needs 1 call to Pool.map(func, ls) (or friends)
>> executing at once, when `func` has a non-trivial memory footprint.
>>
>
> You're quite eager to have this PR merged. I understand that. However,
> it's reasonable to take some time to discuss the design of what you're
> proposing. You don't need it in the stdlib to get your own work done, nor
> to share it with others.
>

I am just eager to solve this problem, which is likely evident, given that
this is the 3rd different implementation discussed in detail since my
initial PR.  If the group consensus is that this is best implemented via
"procmap" function in github gist, then the idea will live there, and
likely have a lonely life there.

I contend that multiprocessing.Pool is used most frequently with a single
task. I am proposing a feature that enforces this invariant, optimizes task
memory-footprints & thus serialization time, and preserves the
well-established interface to Pool through subclassing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181022/3bc110b2/attachment.html>

From steve.dower at python.org  Mon Oct 22 16:41:23 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 22 Oct 2018 16:41:23 -0400
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <c1007ef1-0557-1880-e404-7d5e433b6def@python.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
 <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
 <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>
 <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>
 <pqklec$ec8$1@blaine.gmane.org>
 <c1007ef1-0557-1880-e404-7d5e433b6def@python.org>
Message-ID: <b3b764c8-b6c9-32f7-c1a0-108812080332@python.org>

On 22Oct2018 1047, Steve Dower wrote:
> On 22Oct2018 1007, Serhiy Storchaka wrote:
>> 22.10.18 16:24, Steve Dower ????:
>>> Yes, that's true. But "should reduce ... footprint" is also an 
>>> optimisation that deserves a benchmark by that standard. Also, I'm 
>>> proposing keeping the 'kind' as UCS-2 when the string is created from 
>>> UCS-2 data that is likely to be used as UCS-2. We would not create 
>>> the UCS-1 version in this case, so it's not the same as prefilling 
>>> the cache, but it would cost a bit of memory in exchange for CPU. If 
>>> slicing and concatentation between matching kinds also preserved the 
>>> kind, a lot of path handling code could avoid back-and-forth 
>>> conversions.
>>
>> Oh, I afraid this will complicate the whole code of unicodeobject.c 
>> (and several other files) a much and can introduce a lot of subtle bugs.
>>
>> For example, when you search a UCS2 string in a UCS1 string, the 
>> current code returns the result fast, because a UCS1 string can't 
>> contain codes ?> 0xff, and a UCS2 string should contain codes > 0xff. 
>> And there are many such assumptions.
> 
> That doesn't change though, as we're only ever expanding the range. So 
> searching a UCS2 string in a UCS2 string that doesn't contain any actual 
> UCS2 characters is the only case that would be affected, and whether 
> that case occurs more than the UCS2->UCS1->UCS2 conversion case is 
> something we can measure (but I'd be surprised if substring searches 
> occur more frequently than OS conversions).
> 
> Currently, unicode_compare_eq exits early when the kinds do not match, 
> and that would be a problem (but is also easily fixable). But other 
> string operations already handle mismatched kinds.

I made the changes (along with a somewhat expensive update to make 
__hash__ produce the same value for UCS1 and UCS2 strings) and it works 
just fine, but the speed difference seems to be fairly trivial. Equality 
time in particular is slower (highly optimized memcpy vs. plain-old for 
loop).

That said, I didn't remove the wchar_t cache (though I tried some tricks 
to avoid it), so it's possible that once that's gone we'll see an 
avoidable regression here, but on its own this doesn't contribute much.

Cheers,
Steve

From storchaka at gmail.com  Tue Oct 23 03:49:47 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 23 Oct 2018 10:49:47 +0300
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <b3b764c8-b6c9-32f7-c1a0-108812080332@python.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
 <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
 <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>
 <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>
 <pqklec$ec8$1@blaine.gmane.org>
 <c1007ef1-0557-1880-e404-7d5e433b6def@python.org>
 <b3b764c8-b6c9-32f7-c1a0-108812080332@python.org>
Message-ID: <065146f7-20a3-da6e-46a1-0313eb353d57@gmail.com>

22.10.18 23:41, Steve Dower ????:
> That said, I didn't remove the wchar_t cache (though I tried some tricks 
> to avoid it), so it's possible that once that's gone we'll see an 
> avoidable regression here, but on its own this doesn't contribute much.

Could you please test PR 2599 on Windows? It makes 
PyUnicode_AsWideChar() and PyUnicode_AsWideCharString() not filling the 
wchar_t cache. If there is a significant regression in any benchmark or 
small regressions in several benchmarks we can consider to continue to 
fill the cache in these function and to preserve the cache in future. 
Otherwise I'll merge the PR as is.

https://github.com/python/cpython/pull/2599


From vstinner at redhat.com  Tue Oct 23 05:55:56 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 23 Oct 2018 11:55:56 +0200
Subject: [Python-Dev] Python Language Governance Proposals
Message-ID: <CA+3bQGGP3F3d0tLMNogaX7jL27+bfACOVWB8FJ-DmSOKQyaLbw@mail.gmail.com>

Hi,

Last July, Guido van Rossum decided to resign from his role of BDFL.
Python core developers decided to design a new governance/organization
for Python. 6 governance PEPs have been proposed. It has been decided
that discussions are reserved to core developers (everyone can read,
but only core devs can write), since the governance will mostly impact
the life of core developers. I'm writing this email to python-dev to
keep you aware that something is happening :-)

Core developers (of the GitHub team) will vote to decide the new
Python governance in 3 weeks:
"The vote will happen in a 2-week-long window from November 16 2018 to
November 30 (Anywhere-on-Earth)."

See PEP 8001: Python Governance Voting Process
https://www.python.org/dev/peps/pep-8001/


Below are links to the governance PEPs, their abstract, and link to
the Discourse discussions.

Note: a Discourse instance is experimented at discuss.python.org to
maybe replace python-{ideas,dev,committers} mailing lists. See the
"Discourse Feedback" category at https://discuss.python.org/  :-)


(1) PEP 8010: The BDFL Governance Model
https://www.python.org/dev/peps/pep-8010

Abstract:
"""
This PEP proposes a continuation of the singular project leader model,
euphemistically called the Benevolent Dictator For Life (BDFL) model
of Python governance, to be henceforth called in this PEP the Gracious
Umpire Influencing Decisions Officer (GUIDO). This change in name
reflects both the expanded view of the GUIDO as final arbiter for the
Python language decision making process in consultation with the wider
development community, and the recognition that "for life" while
perhaps aspirational, is not necessarily in the best interest of the
well-being of either the language or the GUIDO themselves.

This PEP describes:

* The rationale for maintaining the singular leader model
* The process for how the GUIDO will be selected, elected, retained,
recalled, and succeeded;
* The roles of the GUIDO in the Python language evolution process;
* The term length of service;
* The relationship of the GUIDO with a Council of Pythonistas (CoP)
that advise the GUIDO on technical matters;
* The size, election, and roles of the CoP;
* The decision delegation process;
* Any changes to the PEP process to fit the new governance model;

This PEP does not name a new BDFL. Should this model be adopted, it
will be codified in PEP 13 along with the names of all officeholders
described in this PEP, as voted on per the guidelines in PEP 8001.
"""

Discussion:
https://discuss.python.org/t/pep-8010-the-singular-leader/188


(2) PEP 8011: Python Governance Model Lead by Trio of Pythonistas
https://www.python.org/dev/peps/pep-8011

Abstract:
"""
This PEP proposes a governance model for the Core Python development
community, led by a trio of equally authoritative leaders. The Trio of
Pythonistas (ToP, or simply Trio) is tasked with making final
decisions for the language. It differs from PEP 8010 by specifically
not proposing a central singular leader, but instead a group of three
people as the leaders.

This PEP also proposes a formation of specialized workgroups to assist
the leadership trio in making decisions.

This PEP does not name the members of the Trio. Should this model be
adopted, it will be codified in PEP 13 along with the names of all
officeholders described in this PEP.

This PEP describes:

* The role and responsibilities of the Trio
* Guidelines of how trio members should be formed
* Reasoning of the group of three, instead of a singular leader
* Role and responsibilities of Python core developers to the trio
* Sustainability considerations
* Diversity and inclusivity considerations
"""

Discussion:
https://discuss.python.org/t/pep-8011-leadership-by-trio-of-pythonistas-top-or-simply-trio/199


(3) PEP 8012: The Community Governance Model
https://www.python.org/dev/peps/pep-8012/

Abstract:
"""
This PEP proposes a new model of Python governance based on consensus
and voting by the Python community. This model relies on workgroups to
carry out the governance of the Python language. This governance model
works without the role of a centralized singular leader or a governing
council.

It describes how, when, and why votes are conducted for decisions
affecting the Python language. It also describes the criteria for
voting eligibility.

Should this model be adopted, it will be codified in PEP 13.

This model can be affectionately called "The Least Worst Governance
Model" by its property that while far from ideal, it's still the most
robust one compared to the others. Since avoiding issues inherent to
the other models is a paramount feature of the Community Governance
Model, we start the discussion a bit unusually: by rejecting the other
models.
"""

Discussion:
https://discuss.python.org/t/pep-8012-the-community-model/156


(4) PEP 8013: The External Council Governance Model
https://www.python.org/dev/peps/pep-8013/
and
https://mail.python.org/pipermail/python-committers/2018-September/006141.html

Abstract:
"""
This PEP proposes a new model of Python governance based on a Group of
Unbiased Independent Directors of Order (GUIDO) tasked with making
final decisions for the language. It differs from PEP 8010 by
specifically not proposing a central singular leader, and from PEP
8011 by disallowing core committers from being council members. It
describes the size and role of the council, how the initial group of
council members will be chosen, any term limits of the council
members, and how successors will be elected.

It also spends significant time discussing the intended behaviour of
this model. By design, many processes are not specified here but are
left to the people involved. In order to select people who will make
the best decisions, it is important for those involved to understand
the expectations of GUIDO but it is equally important to allow GUIDO
the freedom to adjust process requirements for varying circumstances.
This only works when process is unspecified, but all participants have
similar expectations.

This PEP does not name the members of GUIDO. Should this model be
adopted, it will be codified in PEP 13 along with the names of all
officeholders described in this PEP.
"""

Discussion:
https://discuss.python.org/t/pep-8013-the-external-council-governance-model/181


(5) PEP 8014 -- The Commons Governance Model
https://www.python.org/dev/peps/pep-8014/

Abstract:
"""
This PEP proposes a governnance model with as few procedures, defined
terms and percentages as possible. It may also be called The Anarchist
Governance Model but uses Commons for now because of possible negative
connotations of the term Anarchist to some audiences.

The basic idea is that all decisions are voted on by a subset of the
community. A subset, because although the whole community is in
principle entitled to vote in practice it will always be only a small
subset that vote on a specific decision. The vote is overseen by an
impartial council that judges whether the decision has passed or not.
The intention is that this council bases its decision not only on the
ratio of yes and no votes but also on the total number of votes, on
the gravity of the proposal being voted on and possibly the individual
voters and how they voted. Thereby this council becomes responsible
for ensuring that each individual decision is carried by a sufficient
majority.
"""

Discussion:
https://discuss.python.org/t/pep-8014-the-commons-model/173



(6) PEP 8015: Organization of the Python community
https://www.python.org/dev/peps/pep-8015/

Abstract:
"""
This PEP formalizes the current organization of the Python community
and proposes 3 main changes:

Formalize the existing concept of "Python teams";
Give more autonomy to Python teams;
Replace the BDFL (Guido van Rossum) with a new "Python Core Board" of
3 members which have limited roles. Their key role is mostly to decide
how a PEP is approved (or rejected or deferred).

Note: the "BDFL-delegate" role is renamed to "PEP delegate".
"""

Discussion:
https://discuss.python.org/t/pep-8015-organization-of-the-python-community/193
and
https://mail.python.org/pipermail/python-committers/2018-October/006250.html


--

See also:

PEP 8000: Python Language Governance Proposal Overview
https://www.python.org/dev/peps/pep-8000/

PEP 8002: Open Source Governance Survey
https://www.python.org/dev/peps/pep-8002/

Victor

From ncoghlan at gmail.com  Tue Oct 23 07:18:28 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Oct 2018 21:18:28 +1000
Subject: [Python-Dev] The future of the wchar_t cache
In-Reply-To: <c1007ef1-0557-1880-e404-7d5e433b6def@python.org>
References: <pqf246$tv3$1@blaine.gmane.org> <pqf8rt$iat$1@blaine.gmane.org>
 <e8c3d857-ffca-8b99-025a-965f70a405a3@python.org>
 <CA+3bQGGNT-Ep-FLyo2yq5fN_D2a+suVk70tz936s2hyfMSqhuw@mail.gmail.com>
 <97f20027-def7-0493-aa60-9ecc5e2c3241@python.org>
 <CA+3bQGEEad2XvoA-TpK=UKyb2-yDA3dbXxrcq1d6gSe_Ev7kxg@mail.gmail.com>
 <0031b829-b3c9-f804-9ae8-4d2c5b904ded@python.org>
 <pqklec$ec8$1@blaine.gmane.org>
 <c1007ef1-0557-1880-e404-7d5e433b6def@python.org>
Message-ID: <CADiSq7ekzG5=SZ6osGczZfEgC+HkOUSU8eoSe-eCUhXLf_iWEw@mail.gmail.com>

On Tue, 23 Oct 2018 at 00:50, Steve Dower <steve.dower at python.org> wrote:
>
> On 22Oct2018 1007, Serhiy Storchaka wrote:
> > 22.10.18 16:24, Steve Dower ????:
> >> Yes, that's true. But "should reduce ... footprint" is also an
> >> optimisation that deserves a benchmark by that standard. Also, I'm
> >> proposing keeping the 'kind' as UCS-2 when the string is created from
> >> UCS-2 data that is likely to be used as UCS-2. We would not create the
> >> UCS-1 version in this case, so it's not the same as prefilling the
> >> cache, but it would cost a bit of memory in exchange for CPU. If
> >> slicing and concatentation between matching kinds also preserved the
> >> kind, a lot of path handling code could avoid back-and-forth conversions.
> >
> > Oh, I afraid this will complicate the whole code of unicodeobject.c (and
> > several other files) a much and can introduce a lot of subtle bugs.
> >
> > For example, when you search a UCS2 string in a UCS1 string, the current
> > code returns the result fast, because a UCS1 string can't contain codes
> >  > 0xff, and a UCS2 string should contain codes > 0xff. And there are
> > many such assumptions.
>
> That doesn't change though, as we're only ever expanding the range. So
> searching a UCS2 string in a UCS2 string that doesn't contain any actual
> UCS2 characters is the only case that would be affected, and whether
> that case occurs more than the UCS2->UCS1->UCS2 conversion case is
> something we can measure (but I'd be surprised if substring searches
> occur more frequently than OS conversions).
>
> Currently, unicode_compare_eq exits early when the kinds do not match,
> and that would be a problem (but is also easily fixable). But other
> string operations already handle mismatched kinds.

If you did allow for denormalised UCS-2 strings, you'd probably want
some kind of flag on the instance to indicate that the real kind was
8-bit.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From sf at fermigier.com  Tue Oct 23 09:21:19 2018
From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=)
Date: Tue, 23 Oct 2018 15:21:19 +0200
Subject: [Python-Dev] Python Language Governance Proposals
In-Reply-To: <CA+3bQGGP3F3d0tLMNogaX7jL27+bfACOVWB8FJ-DmSOKQyaLbw@mail.gmail.com>
References: <CA+3bQGGP3F3d0tLMNogaX7jL27+bfACOVWB8FJ-DmSOKQyaLbw@mail.gmail.com>
Message-ID: <CABuJJj5_q4Yq3YdMiokEPLrPwucTAT41JRA7t1dA-WcjOG0S0g@mail.gmail.com>

+1 (for what it's worth) to any proposal which includes one (or more)
GUIDOs :)

  S.

On Tue, Oct 23, 2018 at 11:57 AM Victor Stinner <vstinner at redhat.com> wrote:

> Hi,
>
> Last July, Guido van Rossum decided to resign from his role of BDFL.
> Python core developers decided to design a new governance/organization
> for Python. 6 governance PEPs have been proposed. It has been decided
> that discussions are reserved to core developers (everyone can read,
> but only core devs can write), since the governance will mostly impact
> the life of core developers. I'm writing this email to python-dev to
> keep you aware that something is happening :-)
>
> Core developers (of the GitHub team) will vote to decide the new
> Python governance in 3 weeks:
> "The vote will happen in a 2-week-long window from November 16 2018 to
> November 30 (Anywhere-on-Earth)."
>
> See PEP 8001: Python Governance Voting Process
> https://www.python.org/dev/peps/pep-8001/
>
>
> Below are links to the governance PEPs, their abstract, and link to
> the Discourse discussions.
>
> Note: a Discourse instance is experimented at discuss.python.org to
> maybe replace python-{ideas,dev,committers} mailing lists. See the
> "Discourse Feedback" category at https://discuss.python.org/  :-)
>
>
> (1) PEP 8010: The BDFL Governance Model
> https://www.python.org/dev/peps/pep-8010
>
> Abstract:
> """
> This PEP proposes a continuation of the singular project leader model,
> euphemistically called the Benevolent Dictator For Life (BDFL) model
> of Python governance, to be henceforth called in this PEP the Gracious
> Umpire Influencing Decisions Officer (GUIDO). This change in name
> reflects both the expanded view of the GUIDO as final arbiter for the
> Python language decision making process in consultation with the wider
> development community, and the recognition that "for life" while
> perhaps aspirational, is not necessarily in the best interest of the
> well-being of either the language or the GUIDO themselves.
>
> This PEP describes:
>
> * The rationale for maintaining the singular leader model
> * The process for how the GUIDO will be selected, elected, retained,
> recalled, and succeeded;
> * The roles of the GUIDO in the Python language evolution process;
> * The term length of service;
> * The relationship of the GUIDO with a Council of Pythonistas (CoP)
> that advise the GUIDO on technical matters;
> * The size, election, and roles of the CoP;
> * The decision delegation process;
> * Any changes to the PEP process to fit the new governance model;
>
> This PEP does not name a new BDFL. Should this model be adopted, it
> will be codified in PEP 13 along with the names of all officeholders
> described in this PEP, as voted on per the guidelines in PEP 8001.
> """
>
> Discussion:
> https://discuss.python.org/t/pep-8010-the-singular-leader/188
>
>
> (2) PEP 8011: Python Governance Model Lead by Trio of Pythonistas
> https://www.python.org/dev/peps/pep-8011
>
> Abstract:
> """
> This PEP proposes a governance model for the Core Python development
> community, led by a trio of equally authoritative leaders. The Trio of
> Pythonistas (ToP, or simply Trio) is tasked with making final
> decisions for the language. It differs from PEP 8010 by specifically
> not proposing a central singular leader, but instead a group of three
> people as the leaders.
>
> This PEP also proposes a formation of specialized workgroups to assist
> the leadership trio in making decisions.
>
> This PEP does not name the members of the Trio. Should this model be
> adopted, it will be codified in PEP 13 along with the names of all
> officeholders described in this PEP.
>
> This PEP describes:
>
> * The role and responsibilities of the Trio
> * Guidelines of how trio members should be formed
> * Reasoning of the group of three, instead of a singular leader
> * Role and responsibilities of Python core developers to the trio
> * Sustainability considerations
> * Diversity and inclusivity considerations
> """
>
> Discussion:
>
> https://discuss.python.org/t/pep-8011-leadership-by-trio-of-pythonistas-top-or-simply-trio/199
>
>
> (3) PEP 8012: The Community Governance Model
> https://www.python.org/dev/peps/pep-8012/
>
> Abstract:
> """
> This PEP proposes a new model of Python governance based on consensus
> and voting by the Python community. This model relies on workgroups to
> carry out the governance of the Python language. This governance model
> works without the role of a centralized singular leader or a governing
> council.
>
> It describes how, when, and why votes are conducted for decisions
> affecting the Python language. It also describes the criteria for
> voting eligibility.
>
> Should this model be adopted, it will be codified in PEP 13.
>
> This model can be affectionately called "The Least Worst Governance
> Model" by its property that while far from ideal, it's still the most
> robust one compared to the others. Since avoiding issues inherent to
> the other models is a paramount feature of the Community Governance
> Model, we start the discussion a bit unusually: by rejecting the other
> models.
> """
>
> Discussion:
> https://discuss.python.org/t/pep-8012-the-community-model/156
>
>
> (4) PEP 8013: The External Council Governance Model
> https://www.python.org/dev/peps/pep-8013/
> and
>
> https://mail.python.org/pipermail/python-committers/2018-September/006141.html
>
> Abstract:
> """
> This PEP proposes a new model of Python governance based on a Group of
> Unbiased Independent Directors of Order (GUIDO) tasked with making
> final decisions for the language. It differs from PEP 8010 by
> specifically not proposing a central singular leader, and from PEP
> 8011 by disallowing core committers from being council members. It
> describes the size and role of the council, how the initial group of
> council members will be chosen, any term limits of the council
> members, and how successors will be elected.
>
> It also spends significant time discussing the intended behaviour of
> this model. By design, many processes are not specified here but are
> left to the people involved. In order to select people who will make
> the best decisions, it is important for those involved to understand
> the expectations of GUIDO but it is equally important to allow GUIDO
> the freedom to adjust process requirements for varying circumstances.
> This only works when process is unspecified, but all participants have
> similar expectations.
>
> This PEP does not name the members of GUIDO. Should this model be
> adopted, it will be codified in PEP 13 along with the names of all
> officeholders described in this PEP.
> """
>
> Discussion:
>
> https://discuss.python.org/t/pep-8013-the-external-council-governance-model/181
>
>
> (5) PEP 8014 -- The Commons Governance Model
> https://www.python.org/dev/peps/pep-8014/
>
> Abstract:
> """
> This PEP proposes a governnance model with as few procedures, defined
> terms and percentages as possible. It may also be called The Anarchist
> Governance Model but uses Commons for now because of possible negative
> connotations of the term Anarchist to some audiences.
>
> The basic idea is that all decisions are voted on by a subset of the
> community. A subset, because although the whole community is in
> principle entitled to vote in practice it will always be only a small
> subset that vote on a specific decision. The vote is overseen by an
> impartial council that judges whether the decision has passed or not.
> The intention is that this council bases its decision not only on the
> ratio of yes and no votes but also on the total number of votes, on
> the gravity of the proposal being voted on and possibly the individual
> voters and how they voted. Thereby this council becomes responsible
> for ensuring that each individual decision is carried by a sufficient
> majority.
> """
>
> Discussion:
> https://discuss.python.org/t/pep-8014-the-commons-model/173
>
>
>
> (6) PEP 8015: Organization of the Python community
> https://www.python.org/dev/peps/pep-8015/
>
> Abstract:
> """
> This PEP formalizes the current organization of the Python community
> and proposes 3 main changes:
>
> Formalize the existing concept of "Python teams";
> Give more autonomy to Python teams;
> Replace the BDFL (Guido van Rossum) with a new "Python Core Board" of
> 3 members which have limited roles. Their key role is mostly to decide
> how a PEP is approved (or rejected or deferred).
>
> Note: the "BDFL-delegate" role is renamed to "PEP delegate".
> """
>
> Discussion:
>
> https://discuss.python.org/t/pep-8015-organization-of-the-python-community/193
> and
>
> https://mail.python.org/pipermail/python-committers/2018-October/006250.html
>
>
> --
>
> See also:
>
> PEP 8000: Python Language Governance Proposal Overview
> https://www.python.org/dev/peps/pep-8000/
>
> PEP 8002: Open Source Governance Survey
> https://www.python.org/dev/peps/pep-8002/
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/sfermigier%2Blists%40gmail.com
>


-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, Free&OSS Group @ Systematic Cluster -
http://www.gt-logiciel-libre.org/
Co-Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ &
http://pydata.fr/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181023/850fccec/attachment-0001.html>

From emmanuelarias30 at gmail.com  Tue Oct 23 12:14:44 2018
From: emmanuelarias30 at gmail.com (eamanu15)
Date: Tue, 23 Oct 2018 13:14:44 -0300
Subject: [Python-Dev] =?utf-8?q?=E2=80=8BRe=3A_Python_Language_Governance?=
 =?utf-8?q?_Proposals?=
In-Reply-To: <mailman.4549.1540308801.2798.python-dev@python.org>
References: <mailman.4549.1540308801.2798.python-dev@python.org>
Message-ID: <CACttXDPLjFjnLxLq7eK0bn+G-MM+HSGzTLhAhLSBuqk8rGYBQQ@mail.gmail.com>

Are there any list of candidates?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181023/0c59d64d/attachment.html>

From vstinner at redhat.com  Tue Oct 23 13:18:53 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 23 Oct 2018 19:18:53 +0200
Subject: [Python-Dev] 
	=?utf-8?q?=E2=80=8BRe=3A_Python_Language_Governance?=
	=?utf-8?q?_Proposals?=
In-Reply-To: <CACttXDPLjFjnLxLq7eK0bn+G-MM+HSGzTLhAhLSBuqk8rGYBQQ@mail.gmail.com>
References: <mailman.4549.1540308801.2798.python-dev@python.org>
 <CACttXDPLjFjnLxLq7eK0bn+G-MM+HSGzTLhAhLSBuqk8rGYBQQ@mail.gmail.com>
Message-ID: <CA+3bQGE1yA-DOf2i35kdQdf1vmE+3iZ5qxQfR=0_cbg6jcjTpw@mail.gmail.com>

Hi,

Le mar. 23 oct. 2018 ? 18:26, eamanu15 <emmanuelarias30 at gmail.com> a ?crit :
> Are there any list of candidates?

No PEP proposes directly candidates. The process to nominate
candidates is part of discussed PEPs. So first core developers must
vote for the governance PEP, and then a new process will be organized
to select candidates and implement the new governance.

Not all PEPs propose to replace the old BDFL with a new single BDFL.
One PEP proposes 3 BDFL, some PEP don't propose any BDFL, etc. I will
not risk me to summarize the six PEPs here, please read them :-)

Victor

From J.Demeyer at UGent.be  Tue Oct 23 16:19:40 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Tue, 23 Oct 2018 22:19:40 +0200
Subject: [Python-Dev] Python Language Governance Proposals
In-Reply-To: <622cf62bf10448b682011baff41ad863@xmail103.UGent.be>
References: <622cf62bf10448b682011baff41ad863@xmail103.UGent.be>
Message-ID: <5BCF825C.70603@UGent.be>

What is the timeframe for the installation of the new governance? In 
other words, when will it be possible to review PEPs?

From stephane at wirtel.be  Wed Oct 24 02:05:21 2018
From: stephane at wirtel.be (Stephane Wirtel)
Date: Wed, 24 Oct 2018 08:05:21 +0200
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
Message-ID: <20181024060521.GA2083@xps>

Good morning/afternoon/evening/night ;-)

In the documentation of os.system [1], there is this paragraph, where we
suggest to use subprocess instead of os.system.

"""
The subprocess module provides more powerful facilities for spawning new
processes and retrieving their results; using that module is preferable
to using this function. See the Replacing Older Functions with the
subprocess Module section in the subprocess documentation for some
helpful recipes.
"""

But I have found some references (between 15 and 23 (if we include
setup.py)) of os.system.


Do we need to keep the code like that or could we start to use
subprocess for these "references" because it is preferable?

If you think we could move to subprocess, I would like to open an issue
and try to fix it.

1. Add the 'deprecated' directive in the doc
2. Use subprocess for these references

What is your opinion?

Thank you,

St?phane

[1] https://docs.python.org/3.8/library/os.html?highlight=os%20system#os.system

-- 
St?phane Wirtel - https://wirtel.be - @matrixise

From solipsis at pitrou.net  Wed Oct 24 04:35:33 2018
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 24 Oct 2018 10:35:33 +0200
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
References: <20181024060521.GA2083@xps>
Message-ID: <20181024103533.34092d0b@fsol>

On Wed, 24 Oct 2018 08:05:21 +0200
Stephane Wirtel <stephane at wirtel.be> wrote:
> Good morning/afternoon/evening/night ;-)
> 
> In the documentation of os.system [1], there is this paragraph, where we
> suggest to use subprocess instead of os.system.
> 
> """
> The subprocess module provides more powerful facilities for spawning new
> processes and retrieving their results; using that module is preferable
> to using this function. See the Replacing Older Functions with the
> subprocess Module section in the subprocess documentation for some
> helpful recipes.
> """
> 
> But I have found some references (between 15 and 23 (if we include
> setup.py)) of os.system.
> 
> 
> Do we need to keep the code like that or could we start to use
> subprocess for these "references" because it is preferable?
> 
> If you think we could move to subprocess, I would like to open an issue
> and try to fix it.
> 
> 1. Add the 'deprecated' directive in the doc
> 2. Use subprocess for these references
> 
> What is your opinion?

I don't think it's useful to deprecate something that works fine for
the intended purpose.

Regards

Antoine.



From steve.dower at python.org  Wed Oct 24 07:55:46 2018
From: steve.dower at python.org (Steve Dower)
Date: Wed, 24 Oct 2018 07:55:46 -0400
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <20181024103533.34092d0b@fsol>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
Message-ID: <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>

On 24Oct2018 0435, Antoine Pitrou wrote:
> On Wed, 24 Oct 2018 08:05:21 +0200
> Stephane Wirtel <stephane at wirtel.be> wrote:
>> 1. Add the 'deprecated' directive in the doc
>> 2. Use subprocess for these references
>>
>> What is your opinion?
> 
> I don't think it's useful to deprecate something that works fine for
> the intended purpose.

Agreed.

There are two different groups of users involved here. People developing 
scripts to run on their own machines (super-shell scripts, if you like), 
and people developing applications/libraries to run on other machines.

The latter case should *definitely* be using subprocess, because they 
shouldn't be making assumptions about what shell is being used.

The former case should be able to use os.system if that satisfies their 
needs, without seeing warnings that make them think they can't write the 
simplest code that works anymore.

If that means changing *some* of the other references in the docs, then 
I'm okay with that. But both uses are valid, so it's really more about 
being clear who the function is intended for.

Cheers,
Steve

From steve.dower at python.org  Wed Oct 24 10:42:28 2018
From: steve.dower at python.org (Steve Dower)
Date: Wed, 24 Oct 2018 10:42:28 -0400
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
 <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
 <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
Message-ID: <e88452db-a56d-3bf2-0352-bdf527ddf39d@python.org>

On 24Oct2018 0934, Calvin Spealman wrote:
> In the spirit of "There should be one-- and preferably only one 
> --obvious way to do it." this makes perfect sense.

To do what? There is one obvious way to run a system command, and one 
obvious way to manage subprocesses. There are also many non-obvious ways 
to run system commands, and many non-obvious ways to manage subprocesses.

> The distinction between "your own machine and other peoples machines" is 
> not always clear, nor planned for, nor understood by developers to be an 
> important distinction to make up-front. So the encouragement should be 
> clear.

Agreed. One good heuristic is whether you're putting the system command 
in a function or not, since functions are explicitly designing for reuse 
while just using it in a script is not.

> Simply put, there is no valid use case for os.system over subprocess by 
> remaining it must be considered redundant.

You have not shown this. Posting quotes followed by an unrelated 
conclusion isn't a very compelling form of argument :)

Cheers,
Steve

From wes.turner at gmail.com  Wed Oct 24 11:30:01 2018
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 24 Oct 2018 11:30:01 -0400
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <e88452db-a56d-3bf2-0352-bdf527ddf39d@python.org>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
 <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
 <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
 <e88452db-a56d-3bf2-0352-bdf527ddf39d@python.org>
Message-ID: <CACfEFw8_QC=wsKQp4+4c36DpY8YuckFJN7T+XqJQAvoaVux6cA@mail.gmail.com>

Sarge is one way to appropriately wrap Process Control (with shell-like
piping of stdin and stdout between multiple processes).

os.system just calls C system(), AFAIU

subprocess.call calls subprocess.Popen, which does far more than just
wrapping C system(); though the return value is also the process returncode.


https://sarge.readthedocs.io/en/latest/

https://docs.python.org/3/library/os.html#os.system
https://github.com/python/cpython/blob/master/Modules/posixmodule.c

https://docs.python.org/3/library/subprocess.html#subprocess.Popen.returncode

https://docs.python.org/3/library/subprocess.html#subprocess.call

https://docs.python.org/3/library/subprocess.html#subprocess.check_call

https://en.wikipedia.org/wiki/System_call
https://en.wikipedia.org/wiki/Exec_(system_call)

...

"ENH: Add call, check_call, check_output, CalledProcessError,
expect_returncode"
https://bitbucket.org/vinay.sajip/sarge/pull-requests/1/enh-add-call-check_call-check_output/diff

On Wednesday, October 24, 2018, Steve Dower <steve.dower at python.org> wrote:

> On 24Oct2018 0934, Calvin Spealman wrote:
>
>> In the spirit of "There should be one-- and preferably only one --obvious
>> way to do it." this makes perfect sense.
>>
>
> To do what? There is one obvious way to run a system command, and one
> obvious way to manage subprocesses. There are also many non-obvious ways to
> run system commands, and many non-obvious ways to manage subprocesses.
>
> The distinction between "your own machine and other peoples machines" is
>> not always clear, nor planned for, nor understood by developers to be an
>> important distinction to make up-front. So the encouragement should be
>> clear.
>>
>
> Agreed. One good heuristic is whether you're putting the system command in
> a function or not, since functions are explicitly designing for reuse while
> just using it in a script is not.
>
> Simply put, there is no valid use case for os.system over subprocess by
>> remaining it must be considered redundant.
>>
>
> You have not shown this. Posting quotes followed by an unrelated
> conclusion isn't a very compelling form of argument :)
>
> Cheers,
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.
> turner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181024/196f8029/attachment.html>

From cspealma at redhat.com  Wed Oct 24 09:34:34 2018
From: cspealma at redhat.com (Calvin Spealman)
Date: Wed, 24 Oct 2018 09:34:34 -0400
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
 <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
Message-ID: <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>

In the spirit of "There should be one-- and preferably only one --obvious
way to do it." this makes perfect sense.

The distinction between "your own machine and other peoples machines" is
not always clear, nor planned for, nor understood by developers to be an
important distinction to make up-front. So the encouragement should be
clear.

Simply put, there is no valid use case for os.system over subprocess by
remaining it must be considered redundant.

On Wed, Oct 24, 2018 at 7:58 AM Steve Dower <steve.dower at python.org> wrote:

> On 24Oct2018 0435, Antoine Pitrou wrote:
> > On Wed, 24 Oct 2018 08:05:21 +0200
> > Stephane Wirtel <stephane at wirtel.be> wrote:
> >> 1. Add the 'deprecated' directive in the doc
> >> 2. Use subprocess for these references
> >>
> >> What is your opinion?
> >
> > I don't think it's useful to deprecate something that works fine for
> > the intended purpose.
>
> Agreed.
>
> There are two different groups of users involved here. People developing
> scripts to run on their own machines (super-shell scripts, if you like),
> and people developing applications/libraries to run on other machines.
>
> The latter case should *definitely* be using subprocess, because they
> shouldn't be making assumptions about what shell is being used.
>
> The former case should be able to use os.system if that satisfies their
> needs, without seeing warnings that make them think they can't write the
> simplest code that works anymore.
>
> If that means changing *some* of the other references in the docs, then
> I'm okay with that. But both uses are valid, so it's really more about
> being clear who the function is intended for.
>
> Cheers,
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/cspealma%40redhat.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181024/e645dd0a/attachment.html>

From vstinner at redhat.com  Wed Oct 24 12:06:27 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 24 Oct 2018 18:06:27 +0200
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <20181024060521.GA2083@xps>
References: <20181024060521.GA2083@xps>
Message-ID: <CA+3bQGHYgZQZL5Yc0=Uicv3=v7k6wGwkL6LbMCVGPCrMjLZrSg@mail.gmail.com>

I like os.system() and use it everyday. I prefer to write
os.system("grep ^Vm /proc/%s/status" % os.getpid()) than... much more
Python code to do the same thing, or subprocess.run("grep ^Vm
/proc/%s/status" % os.getpid(), shell=True):

os.system() is shorter and only requires "import os" :-)

I'm not using os.system() for "production ready" code, but when I
develop, play with the REPL, etc. For production code, I would write a
function which avoids spawning a new process, obviously. But well,
shell commands are shorter and easier to write (for me, at least).

Victor
Le mer. 24 oct. 2018 ? 08:08, Stephane Wirtel <stephane at wirtel.be> a ?crit :
>
> Good morning/afternoon/evening/night ;-)
>
> In the documentation of os.system [1], there is this paragraph, where we
> suggest to use subprocess instead of os.system.
>
> """
> The subprocess module provides more powerful facilities for spawning new
> processes and retrieving their results; using that module is preferable
> to using this function. See the Replacing Older Functions with the
> subprocess Module section in the subprocess documentation for some
> helpful recipes.
> """
>
> But I have found some references (between 15 and 23 (if we include
> setup.py)) of os.system.
>
>
> Do we need to keep the code like that or could we start to use
> subprocess for these "references" because it is preferable?
>
> If you think we could move to subprocess, I would like to open an issue
> and try to fix it.
>
> 1. Add the 'deprecated' directive in the doc
> 2. Use subprocess for these references
>
> What is your opinion?
>
> Thank you,
>
> St?phane
>
> [1] https://docs.python.org/3.8/library/os.html?highlight=os%20system#os.system
>
> --
> St?phane Wirtel - https://wirtel.be - @matrixise
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com

From chris.barker at noaa.gov  Wed Oct 24 15:41:10 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 24 Oct 2018 12:41:10 -0700
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <CA+3bQGHYgZQZL5Yc0=Uicv3=v7k6wGwkL6LbMCVGPCrMjLZrSg@mail.gmail.com>
References: <20181024060521.GA2083@xps>
 <CA+3bQGHYgZQZL5Yc0=Uicv3=v7k6wGwkL6LbMCVGPCrMjLZrSg@mail.gmail.com>
Message-ID: <CALGmxEK=H-n8Mo82BRbzqYpR3wNBrLR7jqb1qfzh4dyvvtqeZA@mail.gmail.com>

On Wed, Oct 24, 2018 at 9:06 AM, Victor Stinner <vstinner at redhat.com> wrote:

> I like os.system() and use it everyday.


me too.

Python has been evolved over the years away from a "scripting language",
and becoming more of a "systems language". Which is mostly a good thing,
but no need to gratuitously make quick scripting more difficult.

That being said, if there are examples in the docs that use os.system()
that have a good reason to use subprocess, then by all means, let's update
those.

Also: a point was made there that using os.system is essentially making an
assumption about teh shell in use, and thsu is not very portable accross
systems. MAybe that point could be explicitly added to the docs, rather
than just:

 "more powerful facilities for spawning new processes and retrieving their
results;"

When I read that, I think "system is working fine, I don't need  more
powerful facilities for spawning new processes or anything special in
retrieving their results;"

So more info, right there, about WHY you would want to use subprocess
instead would be helpful.


-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181024/59198a4b/attachment.html>

From breamoreboy at gmail.com  Wed Oct 24 16:25:56 2018
From: breamoreboy at gmail.com (Mark Lawrence)
Date: Wed, 24 Oct 2018 21:25:56 +0100
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
 <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
 <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
Message-ID: <pqqkch$ggh$1@blaine.gmane.org>

On 24/10/18 14:34, Calvin Spealman wrote:
> In the spirit of "There should be one-- and preferably only one 
> --obvious way to do it." this makes perfect sense.
> 
> The distinction between "your own machine and other peoples machines" is 
> not always clear, nor planned for, nor understood by developers to be an 
> important distinction to make up-front. So the encouragement should be 
> clear.
> 
> Simply put, there is no valid use case for os.system over subprocess by 
> remaining it must be considered redundant.
> 

Really?  From what I've read so far in this thread you've been shot to 
pieces so case closed.  Goodbye.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From steve at pearwood.info  Wed Oct 24 16:27:06 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 25 Oct 2018 07:27:06 +1100
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
 <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
 <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
Message-ID: <20181024202704.GH3817@ando.pearwood.info>

On Wed, Oct 24, 2018 at 09:34:34AM -0400, Calvin Spealman wrote:

> Simply put, there is no valid use case for os.system over subprocess 

That is simply not true. That's your opinion, masquerading as a fact, 
and made in the face of Steve Dower's description of at least one 
appropriate use of os.system.

If you depreciate and then remove os.system, all you will do is force 
people to re-invent it, badly and inefficiently.


> by remaining it must be considered redundant.

And neither is that.

os.system is not redundant, as its purpose is different from that of 
subprocess.



-- 
Steve

From greg at krypto.org  Wed Oct 24 16:45:06 2018
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 24 Oct 2018 13:45:06 -0700
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <20181024202704.GH3817@ando.pearwood.info>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
 <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
 <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
 <20181024202704.GH3817@ando.pearwood.info>
Message-ID: <CAGE7PNLSPszWKhNBc+JaPv6jtRZBGPNbqOU4mdj7FnkTXDxF3Q@mail.gmail.com>

The os module is by definition special.  It exposes libc and platform
APIs.  That there are Python modules that provide similar functionality,
often surpassing it and sometimes being built on top of it, is
intentional.  Random quotes from the Zen don't win arguments.  Although
practicality beats purity.  :P

os.system isn't going away.  That'd be unnecessarily disruptive.

-gps

On Wed, Oct 24, 2018 at 1:29 PM Steven D'Aprano <steve at pearwood.info> wrote:

> On Wed, Oct 24, 2018 at 09:34:34AM -0400, Calvin Spealman wrote:
>
> > Simply put, there is no valid use case for os.system over subprocess
>
> That is simply not true. That's your opinion, masquerading as a fact,
> and made in the face of Steve Dower's description of at least one
> appropriate use of os.system.
>
> If you depreciate and then remove os.system, all you will do is force
> people to re-invent it, badly and inefficiently.
>
>
> > by remaining it must be considered redundant.
>
> And neither is that.
>
> os.system is not redundant, as its purpose is different from that of
> subprocess.
>
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181024/94fd2c26/attachment.html>

From pwang at anaconda.com  Wed Oct 24 17:39:23 2018
From: pwang at anaconda.com (Peter Wang)
Date: Wed, 24 Oct 2018 16:39:23 -0500
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <20181024202704.GH3817@ando.pearwood.info>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
 <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
 <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
 <20181024202704.GH3817@ando.pearwood.info>
Message-ID: <CABGrp+9auPXPYG1htgxnJCKEFO9w7sedz9mkmo5suA3GJzz5xw@mail.gmail.com>

On Wed, Oct 24, 2018 at 3:31 PM Steven D'Aprano <steve at pearwood.info> wrote:

> On Wed, Oct 24, 2018 at 09:34:34AM -0400, Calvin Spealman wrote:
>
> > Simply put, there is no valid use case for os.system over subprocess
>
> If you depreciate and then remove os.system, all you will do is force
> people to re-invent it, badly and inefficiently.


Indeed.  Usability counts, as does not arbitrarily breaking vast amounts of
existing code that Just Works.

Every code-breaking deprecation is an invitation for unseen thousands or
millions to ponder, "Hm, this script doesn't work anymore. Maybe now is the
time to look at Node or Rust."

-Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181024/cdce0159/attachment.html>

From greg.ewing at canterbury.ac.nz  Wed Oct 24 23:24:11 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 25 Oct 2018 16:24:11 +1300
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <CALGmxEK=H-n8Mo82BRbzqYpR3wNBrLR7jqb1qfzh4dyvvtqeZA@mail.gmail.com>
References: <20181024060521.GA2083@xps>
 <CA+3bQGHYgZQZL5Yc0=Uicv3=v7k6wGwkL6LbMCVGPCrMjLZrSg@mail.gmail.com>
 <CALGmxEK=H-n8Mo82BRbzqYpR3wNBrLR7jqb1qfzh4dyvvtqeZA@mail.gmail.com>
Message-ID: <5BD1375B.40104@canterbury.ac.nz>

My take on this is that os.system() is there because it's part
of the C stdlib, and Python generally aims to provide wrappers
for all of the C stdlib facilities. It's not Python's place to
start making value judgements about which things are worthy of
being wrapped and which aren't.

-- 
Greg


From ncoghlan at gmail.com  Thu Oct 25 08:00:19 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Oct 2018 22:00:19 +1000
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
 <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
 <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
Message-ID: <CADiSq7cZxiPXJXZ1VT-e6=ON3fwu+2dMG_eGj0K9AbE4QMsWcg@mail.gmail.com>

On Thu, 25 Oct 2018 at 01:34, Calvin Spealman <cspealma at redhat.com> wrote:
> Simply put, there is no valid use case for os.system over subprocess by remaining it must be considered redundant.

They do different things. The warnings against using os.system are
based on "If you don't know whether or not you have the use case that
this exists to handle it's much safer to assume that you don't", not
"This has no valid use cases" (your use case just has to meet the
criteria that makes os.system safe to use - no exposure to untrusted
input.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Oct 25 08:06:13 2018
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Oct 2018 22:06:13 +1000
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <CADiSq7cZxiPXJXZ1VT-e6=ON3fwu+2dMG_eGj0K9AbE4QMsWcg@mail.gmail.com>
References: <20181024060521.GA2083@xps> <20181024103533.34092d0b@fsol>
 <03456f6c-0c14-57f2-d749-a9bb2e3ca9ad@python.org>
 <CACo5Rz6W9g8v+_SP78CwBRe+q7-=s6RSoUMUf5QmdEn0cOHwZA@mail.gmail.com>
 <CADiSq7cZxiPXJXZ1VT-e6=ON3fwu+2dMG_eGj0K9AbE4QMsWcg@mail.gmail.com>
Message-ID: <CADiSq7cahQxK=q0ont1P0KPeUO2LwagKt0u4w=jTF_8OA5vZ=Q@mail.gmail.com>

On Thu, 25 Oct 2018 at 22:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On Thu, 25 Oct 2018 at 01:34, Calvin Spealman <cspealma at redhat.com> wrote:
> > Simply put, there is no valid use case for os.system over subprocess by remaining it must be considered redundant.
>
> They do different things. The warnings against using os.system are
> based on "If you don't know whether or not you have the use case that
> this exists to handle it's much safer to assume that you don't", not
> "This has no valid use cases" (your use case just has to meet the
> criteria that makes os.system safe to use - no exposure to untrusted
> input.

Whoops, hit send without finishing the sentence: no exposure to
untrusted input, no need for cross-platform compatibility, no need for
assistance with getting string quoting right, no need for significant
interaction with the child process.

os.system is a good thing for linters (especially security linters) to
warn about, since running a linter over something is a decent hint
that you're not writing a throwaway script, and if folks *are* running
a linter against their ad hoc scripts, it's reasonably to expect them
to configure it for their personal preferences. It isn't actively
blocking the development of higher level alternatives though, so there
isn't a good reason to deprecate it and break working code.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stephane at wirtel.be  Thu Oct 25 11:13:08 2018
From: stephane at wirtel.be (Stephane Wirtel)
Date: Thu, 25 Oct 2018 17:13:08 +0200
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <20181024060521.GA2083@xps>
References: <20181024060521.GA2083@xps>
Message-ID: <20181025151308.GA2002@xps>

Hi all,

After your feedback, I have my answer.

I understand the your points of view and I don't want to change any part
of code for os.system and subprocess, I don't want to deprecate
os.system in favor of subprocess. I just wanted to know your opinion
about this point.

+1 to use os.system when we only need a call without portability, just a
quick code.

+1 for subprocess for the portability.

Thank you so much for your patience/time and your explanations.

St?phane

On 10/24, Stephane Wirtel wrote:
>Good morning/afternoon/evening/night ;-)
>
>In the documentation of os.system [1], there is this paragraph, where we
>suggest to use subprocess instead of os.system.
>
>"""
>The subprocess module provides more powerful facilities for spawning new
>processes and retrieving their results; using that module is preferable
>to using this function. See the Replacing Older Functions with the
>subprocess Module section in the subprocess documentation for some
>helpful recipes.
>"""
>
>But I have found some references (between 15 and 23 (if we include
>setup.py)) of os.system.
>
>
>Do we need to keep the code like that or could we start to use
>subprocess for these "references" because it is preferable?
>
>If you think we could move to subprocess, I would like to open an issue
>and try to fix it.
>
>1. Add the 'deprecated' directive in the doc
>2. Use subprocess for these references
>
>What is your opinion?
>
>Thank you,
>
>St?phane
>
>[1] https://docs.python.org/3.8/library/os.html?highlight=os%20system#os.system
>
>-- 
>St?phane Wirtel - https://wirtel.be - @matrixise
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: https://mail.python.org/mailman/options/python-dev/stephane%40wirtel.be

-- 
St?phane Wirtel - https://wirtel.be - @matrixise

From aixtools at felt.demon.nl  Fri Oct 26 02:30:23 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Fri, 26 Oct 2018 08:30:23 +0200
Subject: [Python-Dev] "Deprecation" of os.system in favor of subprocess?
In-Reply-To: <20181025151308.GA2002@xps>
References: <20181024060521.GA2083@xps> <20181025151308.GA2002@xps>
Message-ID: <1fe554b8-d292-516e-bc8f-dcb04088c3c7@felt.demon.nl>

Thanks for asking a question that triggered an enlightening discussion!


On 10/25/2018 5:13 PM, Stephane Wirtel wrote:
> Hi all,
>
> After your feedback, I have my answer.
>
> I understand the your points of view and I don't want to change any part
> of code for os.system and subprocess, I don't want to deprecate
> os.system in favor of subprocess. I just wanted to know your opinion
> about this point.
>
> +1 to use os.system when we only need a call without portability, just a
> quick code.
>
> +1 for subprocess for the portability.
>
> Thank you so much for your patience/time and your explanations.
>
> St?phane


From status at bugs.python.org  Fri Oct 26 12:10:03 2018
From: status at bugs.python.org (Python tracker)
Date: Fri, 26 Oct 2018 18:10:03 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20181026161003.B76BD57418@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2018-10-19 - 2018-10-26)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6823 (-12)
  closed 40005 (+62)
  total  46828 (+50)

Open issues with patches: 2716 


Issues opened (34)
==================

#25750: tp_descr_get(self, obj, type) is called without owning a refer
https://bugs.python.org/issue25750  reopened by serhiy.storchaka

#35028: Off by one error in cgi.FieldStorage(max_num_fields)
https://bugs.python.org/issue35028  opened by Matthew Belisle

#35031: test_asyncio test_start_tls_server_1 fails in AMD64 FreeBSD CU
https://bugs.python.org/issue35031  opened by pablogsal

#35032: Remove the videos from faq/Windows
https://bugs.python.org/issue35032  opened by matrixise

#35033: Column or row spanning cells are not implemented.
https://bugs.python.org/issue35033  opened by mdk

#35035: Documentation for email.utils is named email.util.rst
https://bugs.python.org/issue35035  opened by zmwangx

#35037: PYLONG_BITS_IN_DIGIT differs between MinGW and MSVC
https://bugs.python.org/issue35037  opened by scoder

#35040: [functools] provide an async-compatible version of functools.l
https://bugs.python.org/issue35040  opened by Liran Nuna

#35042: Use the role :pep: for the PEP \d+
https://bugs.python.org/issue35042  opened by matrixise

#35045: test_min_max_version (test.test_ssl.ContextTests) fails on Fed
https://bugs.python.org/issue35045  opened by cstratak

#35047: Better error messages in unittest.mock
https://bugs.python.org/issue35047  opened by Petter S

#35048: Can't reassign __class__ despite the assigned class having ide
https://bugs.python.org/issue35048  opened by bup

#35050: Off-by-one bug in AF_ALG
https://bugs.python.org/issue35050  opened by christian.heimes

#35052: Coverity scan: copy/paste error in Lib/xml/dom/minidom.py
https://bugs.python.org/issue35052  opened by cstratak

#35054: Add more index entries for symbols
https://bugs.python.org/issue35054  opened by serhiy.storchaka

#35055: Error when we try to download the epub archive
https://bugs.python.org/issue35055  opened by matrixise

#35056: Test leaks of memory not managed by Python allocator
https://bugs.python.org/issue35056  opened by serhiy.storchaka

#35058: Unable to Install Python on Windows
https://bugs.python.org/issue35058  opened by alexbach

#35059: Convert Py_INCREF() and PyObject_INIT() to inlined functions
https://bugs.python.org/issue35059  opened by vstinner

#35061: Specify libffi.so soname for ctypes
https://bugs.python.org/issue35061  opened by tturbs

#35062: io.IncrementalNewlineDecoder assign out-of-range value to bitw
https://bugs.python.org/issue35062  opened by xiang.zhang

#35063: Checking for abstractmethod implementation fails to consider M
https://bugs.python.org/issue35063  opened by Antony.Lee

#35064: COUNT_ALLOCS build export inc_count() and dec_count() function
https://bugs.python.org/issue35064  opened by vstinner

#35065: Reading received data from a closed TCP stream using `StreamRe
https://bugs.python.org/issue35065  opened by vxgmichel

#35066: Inconsistency between dangling '%' handling in time.strftime()
https://bugs.python.org/issue35066  opened by mjsaah

#35067: Use vswhere instead of _distutils_findvs
https://bugs.python.org/issue35067  opened by steve.dower

#35068: [2.7] Possible crashes due to incorrect error handling in pyex
https://bugs.python.org/issue35068  opened by ZackerySpytz

#35070: test_posix fails on macOS 10.14 Mojave
https://bugs.python.org/issue35070  opened by barry

#35071: Canot send real string from c api to module (corrupted string)
https://bugs.python.org/issue35071  opened by Yhojann Aguilera

#35072: re.sub does not play nice with chr(92)
https://bugs.python.org/issue35072  opened by Samuel Warfield

#35074: source install [3.7.1] on debian jessie
https://bugs.python.org/issue35074  opened by foxleoly

#35075: Doc: pprint example uses dead URL
https://bugs.python.org/issue35075  opened by inada.naoki

#35076: FAIL: test_min_max_version (test.test_ssl.ContextTests) with l
https://bugs.python.org/issue35076  opened by jean-michel

#35077: Make TypeError message less ambiguous
https://bugs.python.org/issue35077  opened by coyot



Most recent 15 issues with no replies (15)
==========================================

#35076: FAIL: test_min_max_version (test.test_ssl.ContextTests) with l
https://bugs.python.org/issue35076

#35075: Doc: pprint example uses dead URL
https://bugs.python.org/issue35075

#35068: [2.7] Possible crashes due to incorrect error handling in pyex
https://bugs.python.org/issue35068

#35067: Use vswhere instead of _distutils_findvs
https://bugs.python.org/issue35067

#35064: COUNT_ALLOCS build export inc_count() and dec_count() function
https://bugs.python.org/issue35064

#35063: Checking for abstractmethod implementation fails to consider M
https://bugs.python.org/issue35063

#35062: io.IncrementalNewlineDecoder assign out-of-range value to bitw
https://bugs.python.org/issue35062

#35061: Specify libffi.so soname for ctypes
https://bugs.python.org/issue35061

#35048: Can't reassign __class__ despite the assigned class having ide
https://bugs.python.org/issue35048

#35045: test_min_max_version (test.test_ssl.ContextTests) fails on Fed
https://bugs.python.org/issue35045

#35035: Documentation for email.utils is named email.util.rst
https://bugs.python.org/issue35035

#35024: Incorrect logging in importlib when '.pyc' file creation fails
https://bugs.python.org/issue35024

#35018: Sax parser provides no user access to lexical handlers
https://bugs.python.org/issue35018

#35009: argparse throws UnicodeEncodeError for printing help with unic
https://bugs.python.org/issue35009

#35000: aexit called after loop close
https://bugs.python.org/issue35000



Most recent 15 issues waiting for review (15)
=============================================

#35068: [2.7] Possible crashes due to incorrect error handling in pyex
https://bugs.python.org/issue35068

#35067: Use vswhere instead of _distutils_findvs
https://bugs.python.org/issue35067

#35065: Reading received data from a closed TCP stream using `StreamRe
https://bugs.python.org/issue35065

#35059: Convert Py_INCREF() and PyObject_INIT() to inlined functions
https://bugs.python.org/issue35059

#35054: Add more index entries for symbols
https://bugs.python.org/issue35054

#35050: Off-by-one bug in AF_ALG
https://bugs.python.org/issue35050

#35047: Better error messages in unittest.mock
https://bugs.python.org/issue35047

#35042: Use the role :pep: for the PEP \d+
https://bugs.python.org/issue35042

#35035: Documentation for email.utils is named email.util.rst
https://bugs.python.org/issue35035

#35032: Remove the videos from faq/Windows
https://bugs.python.org/issue35032

#35031: test_asyncio test_start_tls_server_1 fails in AMD64 FreeBSD CU
https://bugs.python.org/issue35031

#35028: Off by one error in cgi.FieldStorage(max_num_fields)
https://bugs.python.org/issue35028

#35024: Incorrect logging in importlib when '.pyc' file creation fails
https://bugs.python.org/issue35024

#35022: MagicMock should support `__fspath__`
https://bugs.python.org/issue35022

#35021: Assertion failures in datetimemodule.c.
https://bugs.python.org/issue35021



Top 10 most discussed issues (10)
=================================

#35059: Convert Py_INCREF() and PyObject_INIT() to inlined functions
https://bugs.python.org/issue35059  17 msgs

#33015: Fix function cast warning in thread_pthread.h
https://bugs.python.org/issue33015  13 msgs

#35052: Coverity scan: copy/paste error in Lib/xml/dom/minidom.py
https://bugs.python.org/issue35052  12 msgs

#34576: [EASY doc] http.server, SimpleHTTPServer: warn users on securi
https://bugs.python.org/issue34576  10 msgs

#35066: Inconsistency between dangling '%' handling in time.strftime()
https://bugs.python.org/issue35066   9 msgs

#35042: Use the role :pep: for the PEP \d+
https://bugs.python.org/issue35042   8 msgs

#35070: test_posix fails on macOS 10.14 Mojave
https://bugs.python.org/issue35070   8 msgs

#35055: Error when we try to download the epub archive
https://bugs.python.org/issue35055   7 msgs

#9263: Try to print repr() when an C-level assert fails (in the garba
https://bugs.python.org/issue9263   6 msgs

#35037: PYLONG_BITS_IN_DIGIT differs between MinGW and MSVC
https://bugs.python.org/issue35037   6 msgs



Issues closed (60)
==================

#1621: Do not assume signed integer overflow behavior
https://bugs.python.org/issue1621  closed by vstinner

#8525: Display exceptions' subclasses in help()
https://bugs.python.org/issue8525  closed by ncoghlan

#20216: Misleading docs for sha1, sha256, sha512, md5 modules
https://bugs.python.org/issue20216  closed by vstinner

#21196: Name mangling example in Python tutorial
https://bugs.python.org/issue21196  closed by vstinner

#21967: Interpreter crash upon accessing frame.f_restricted of a frame
https://bugs.python.org/issue21967  closed by serhiy.storchaka

#29843: errors raised by ctypes.Array for invalid _length_ attribute
https://bugs.python.org/issue29843  closed by taleinat

#30863: Rewrite PyUnicode_AsWideChar() and PyUnicode_AsWideCharString(
https://bugs.python.org/issue30863  closed by serhiy.storchaka

#32236: open() shouldn't silently ignore buffering=1 in binary mode
https://bugs.python.org/issue32236  closed by vstinner

#32256: Make patchcheck.py work for out-of-tree builds
https://bugs.python.org/issue32256  closed by izbyshev

#32321: Add pure Python fallback for functools.reduce
https://bugs.python.org/issue32321  closed by vstinner

#32798: mmap.flush() on Linux does not accept the "offset" and "size" 
https://bugs.python.org/issue32798  closed by vstinner

#32890: os: Some functions may report bogus errors on Windows
https://bugs.python.org/issue32890  closed by vstinner

#33594: add deprecation since 3.5 for a few methods of inspect.
https://bugs.python.org/issue33594  closed by vstinner

#33708: Doc: Asyncio's Event documentation typo.
https://bugs.python.org/issue33708  closed by vstinner

#33712: OrderedDict can set an exception in tp_clear
https://bugs.python.org/issue33712  closed by serhiy.storchaka

#33726: Add short descriptions to PEP references in seealso
https://bugs.python.org/issue33726  closed by vstinner

#33947: Dataclasses can raise RecursionError in __repr__
https://bugs.python.org/issue33947  closed by eric.smith

#34070: Superfluous call to isatty in open() when buffering >= 0
https://bugs.python.org/issue34070  closed by vstinner

#34081: Sphinx duplicate label warning in docs
https://bugs.python.org/issue34081  closed by mdk

#34260: shutil.copy2 is not the same as cp -p
https://bugs.python.org/issue34260  closed by vstinner

#34282: Enum._convert shadows members named _convert
https://bugs.python.org/issue34282  closed by orlnub123

#34482: datetime: Tests for potential crashes due to non-UTF-8-encodab
https://bugs.python.org/issue34482  closed by taleinat

#34551: Redundant store can be removed from _PyFunction_FastCallDict
https://bugs.python.org/issue34551  closed by lukasz.langa

#34573: Simplify __reduce__() of set and dict iterators.
https://bugs.python.org/issue34573  closed by vstinner

#34574: OrderedDict iterators are exhausted during pickling
https://bugs.python.org/issue34574  closed by serhiy.storchaka

#34748: Incorrect HTML link in functools.partial
https://bugs.python.org/issue34748  closed by xiang.zhang

#34789: Make xml.sax.make_parser accept iterables
https://bugs.python.org/issue34789  closed by taleinat

#34824: _ssl.c: Possible null pointer dereference
https://bugs.python.org/issue34824  closed by vstinner

#34839: doctest: Change example under warnings section
https://bugs.python.org/issue34839  closed by mdk

#34890: Support functools.partial in inspect.is*function() checks
https://bugs.python.org/issue34890  closed by pablogsal

#34901: Missing isolated (-I) flag in sys.flags table
https://bugs.python.org/issue34901  closed by ned.deily

#34912: Update overflow checks in resize_buffer
https://bugs.python.org/issue34912  closed by vstinner

#34936: tkinter.Spinbox.selection_element() raises TclError
https://bugs.python.org/issue34936  closed by serhiy.storchaka

#34970: Protect tasks weak set manipulation in asyncio.all_tasks()
https://bugs.python.org/issue34970  closed by ned.deily

#34973: Crash in bytes constructor with mutating list
https://bugs.python.org/issue34973  closed by serhiy.storchaka

#34980: KillPython target doesn't detect 64-bit processes
https://bugs.python.org/issue34980  closed by jkloth

#34983: expose symtable.Symbol.is_nonlocal()
https://bugs.python.org/issue34983  closed by pablogsal

#34984: Improve error messages in bytes and bytearray constructors
https://bugs.python.org/issue34984  closed by serhiy.storchaka

#34991: variable type list [] referential integrity data loss
https://bugs.python.org/issue34991  closed by ammar2

#35017: socketserver accept a last request after shutdown
https://bugs.python.org/issue35017  closed by vstinner

#35020: Add multisort recipe to sorting docs
https://bugs.python.org/issue35020  closed by rhettinger

#35025: Compiling `timemodule.c` can fail on macOS due to availability
https://bugs.python.org/issue35025  closed by benjamin.peterson

#35027: distutils.core.setup does not raise TypeError when if classifi
https://bugs.python.org/issue35027  closed by xtreak

#35029: Convert SyntaxWarning exception raised at code generation time
https://bugs.python.org/issue35029  closed by serhiy.storchaka

#35030: Python 2.7 OrderedDict creates circular references
https://bugs.python.org/issue35030  closed by rhettinger

#35034: Add closing and iteration to threading.Queue
https://bugs.python.org/issue35034  closed by rhettinger

#35036: logger failure in suspicious.py
https://bugs.python.org/issue35036  closed by pablogsal

#35038: AttributeError: 'frame' object has no attribute 'f_restricted'
https://bugs.python.org/issue35038  closed by matrixise

#35039: remove unused vars in  Lib/turtledemo module
https://bugs.python.org/issue35039  closed by vstinner

#35041: urllib.parse.quote safe Parameter Not Optional
https://bugs.python.org/issue35041  closed by ammar2

#35043: functools.reduce doesn't work properly with itertools.chain
https://bugs.python.org/issue35043  closed by Vasantha Ganesh Kanniappan

#35044: Use the :exc: role for the exceptions in the doc
https://bugs.python.org/issue35044  closed by vstinner

#35046: logging.StreamHandler performs two syscalls when one would do
https://bugs.python.org/issue35046  closed by vinay.sajip

#35049: argparse.ArgumentParser fails on arguments with leading dash a
https://bugs.python.org/issue35049  closed by paul.j3

#35051: Fix pep8 on  Lib/turtledemo module
https://bugs.python.org/issue35051  closed by brett.cannon

#35053: Enhance tracemalloc to trace properly free lists
https://bugs.python.org/issue35053  closed by vstinner

#35057: Email header refolding adds additional \r in nested parse tree
https://bugs.python.org/issue35057  closed by r.david.murray

#35060: subprocess output seems to depend on size of terminal screen
https://bugs.python.org/issue35060  closed by eryksun

#35069: Unexecuted import in function causes UnboundLocalError
https://bugs.python.org/issue35069  closed by steven.daprano

#35073: 'from app import __init__' behaves differently with native imp
https://bugs.python.org/issue35073  closed by qagren

From brett at python.org  Fri Oct 26 13:17:22 2018
From: brett at python.org (Brett Cannon)
Date: Fri, 26 Oct 2018 10:17:22 -0700
Subject: [Python-Dev] Python Language Governance Proposals
In-Reply-To: <5BCF825C.70603@UGent.be>
References: <622cf62bf10448b682011baff41ad863@xmail103.UGent.be>
 <5BCF825C.70603@UGent.be>
Message-ID: <CAP1=2W4TBjx1wZ0p1UnC02MCn9XnLuiGe9W20KB+80i_ZMiaKg@mail.gmail.com>

On Tue, 23 Oct 2018 at 13:20, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> What is the timeframe for the installation of the new governance? In
> other words, when will it be possible to review PEPs?
>

PEP 8001 outlines the voting for the governance models which includes a
planned schedule for that vote. After that it will vary depending on which
governance model is chosen as some of them require electing people to
positions while others don't. The overall goal is to have the whole ting
resolved no later than probably March (but probably sooner than that).
Basically this should not be a new thing come PyCon US in May.

But since you're asking about wanting to "review PEPs", you can review them
now. There hasn't been much change to them lately so they are pretty stable
at this point.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181026/35b03d37/attachment.html>

From steve at holdenweb.com  Fri Oct 26 14:19:00 2018
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 26 Oct 2018 19:19:00 +0100
Subject: [Python-Dev] Python Language Governance Proposals
In-Reply-To: <CAP1=2W4TBjx1wZ0p1UnC02MCn9XnLuiGe9W20KB+80i_ZMiaKg@mail.gmail.com>
References: <622cf62bf10448b682011baff41ad863@xmail103.UGent.be>
 <5BCF825C.70603@UGent.be>
 <CAP1=2W4TBjx1wZ0p1UnC02MCn9XnLuiGe9W20KB+80i_ZMiaKg@mail.gmail.com>
Message-ID: <CAMofdRB-MNJCxusxxRoyjLXk2id7zc9Ui2GEycohummLfop5Hg@mail.gmail.com>

As a piece of pure experiential data based on some years trying to herd the
PSF cats, if python-dev can find a way of running its activities without
votes needing to be taken I would really emphasise the benefits of the lack
of such administration.

If voting is required, please consider using the PSF to manage such
activity.

Following the debate with interest, but mostly lurking due to my usual
absence of skin in the game. Bonne chance!

regards
 Steve

Steve Holden

On Fri, Oct 26, 2018 at 6:17 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Tue, 23 Oct 2018 at 13:20, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
>
>> What is the timeframe for the installation of the new governance? In
>> other words, when will it be possible to review PEPs?
>>
>
> PEP 8001 outlines the voting for the governance models which includes a
> planned schedule for that vote. After that it will vary depending on which
> governance model is chosen as some of them require electing people to
> positions while others don't. The overall goal is to have the whole ting
> resolved no later than probably March (but probably sooner than that).
> Basically this should not be a new thing come PyCon US in May.
>
> But since you're asking about wanting to "review PEPs", you can review
> them now. There hasn't been much change to them lately so they are pretty
> stable at this point.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> steve%40holdenweb.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181026/0610bc01/attachment.html>

From mathias.laurin at gmail.com  Fri Oct 26 15:34:38 2018
From: mathias.laurin at gmail.com (Mathias Laurin)
Date: Fri, 26 Oct 2018 21:34:38 +0200
Subject: [Python-Dev] PEP 543-conform TLS library
Message-ID: <3A7C8D1D-FA3F-46AF-8EC8-540D407EC825@gmail.com>

Hello Python Dev,


I posted the following to python-ideas but here may be
a more suitable place.  I apologize if cross posting
bothers anyone.


I have implemented an (I believe) PEP 543-conform TLS library
and released TLS support in the latest version yesterday:

https://github.com/Synss/python-mbedtls/tree/0.13.0
https://pypi.org/project/python-mbedtls/0.13.0/


As far as I know, I am the first one to follow PEP 543.  So one
point is that the API works.  However, I have a couple of
questions regarding the PEP:

- I do not know what to do in `TLSWrappedBuffer.do_handshake()`.
 The full TLS handshake requires writing to the server, reading
 back, etc., (ClientHello, ServerHello, KeyExchange, etc.),
 which cannot be accomplished in a single buffer.

 For now, I am doing the handshake in
 `TLSWrappedSocket.do_handshake()`: I set the BIO to using the
 socket directly, then perform the handshake on the socket thus
 entirely bypassing the TLSWrappedBuffer.  Once this is done, I
 swap the BIO to using the buffer and go on encrypting and
 decrypting from the buffer.  That is, the encrypted
 communication is buffered.

- The PEP sometimes mentions an "input buffer" and an "output
 buffer", and some other times just "the buffer".  I believe
 that both implementations are possible.  That is, with two
 different buffers for input and output, or a single one.

 I have implemented it with a single circular buffer (that is a
 stream after all).  What the PEP is expecting is nonetheless
 not clear to me.


So, can anybody clarify these two points from the PEP?


Or should I just address Cory Benfield (who does not seem very
active anymore lately) and Christian Heimes directly?


Cheers,
Mathias

From horos22 at gmail.com  Fri Oct 26 19:18:37 2018
From: horos22 at gmail.com (Ed Peschko)
Date: Fri, 26 Oct 2018 16:18:37 -0700
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in python
 debugger.
Message-ID: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>

all,

I was debugging a very long script that I was not all that familiar
with, and I was doing my familiar routine of being very careful in
evaluating expressions to make sure that I didn't hit such statements
as:

TypeError: unsupported operand type(s) for +: 'int' and 'str'

anyways the script has a runtime of hours, so this was tedious work,
and I hit one-too-many times where I missed a condition and had to
start all over again.

So that got me thinking: the main problem with exceptions and runtime
errors is that they short-circuit the program context. You have this
context, but you can't change it to avoid the failure; ie: with

1. aa = 'a'
2. bb = 1 + a
3. print bb

you'll never get to line 3 if you go through line 2. If you are lucky
you can catch it, modify aa to be an integer to continue, otherwise
your only recourse is to start over again.

So I was wondering if it would be possible to keep that context around
if you are in the debugger and rewind the execution point to before
the statement was triggered. so you could say:

python script.py
(Pdb) c

then hit the exception, say:
(Pdb) aa = 2

then run again
(Pdb) c

to work your way through the exception point. You could then fix the
script inline in an editor.

I can't emphasize exactly how much time and effort this would save. At
best, debugging these types of issues is annoying, at worst it is
excruciating, because they are sometimes intermittent and are not
easily repeatable. Using RemotePdb or Pdb to attach to a long-running
process and having the assurance that the underlying script won't die
because of an inane coding error would do wonders for the reliability
and integrity of scripting.

so - is this possible? just from my experiments it doesn't look so,
but perhaps there is a trick out there that would give this
functionality..

if it isn't possible, how easy would it be to implement?

Thanks much,

Ed

From steve at pearwood.info  Fri Oct 26 19:48:07 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 27 Oct 2018 10:48:07 +1100
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in
 python debugger.
In-Reply-To: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
References: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
Message-ID: <20181026234806.GR3817@ando.pearwood.info>

On Fri, Oct 26, 2018 at 04:18:37PM -0700, Ed Peschko wrote:
> all,
> 
> I was debugging a very long script that I was not all that familiar
> with, and I was doing my familiar routine of being very careful in
> evaluating expressions to make sure that I didn't hit such statements
> as:
> 
> TypeError: unsupported operand type(s) for +: 'int' and 'str'
> 
> anyways the script has a runtime of hours, so this was tedious work,
> and I hit one-too-many times where I missed a condition and had to
> start all over again.

Obviously you weren't careful enough :-)

You know you can set breakpoints in the debugger? You don't have to 
single-step all the way through from the beginning. It isn't clear from 
your post how experienced you are.


[...]
> So I was wondering if it would be possible to keep that context around
> if you are in the debugger and rewind the execution point to before
> the statement was triggered.

I think what you are looking for is a reverse debugger[1] also known as 
a time-travel debugger.

https://softwareengineering.stackexchange.com/questions/181527/why-is-reverse-debugging-rarely-used

There's a Python implementation, but I've never used it:

https://morepypy.blogspot.com/2016/07/reverse-debugging-for-python.html


[...]
> I can't emphasize exactly how much time and effort this would save.

That's what people say. But not everyone is a fan -- there are 
apparently memory consumption and stability issues with reverse 
debugging. There are people who love it, and those who don't.


> so - is this possible? just from my experiments it doesn't look so,
> but perhaps there is a trick out there that would give this
> functionality..
> 
> if it isn't possible, how easy would it be to implement?

If it isn't possible, it would be very difficult to implement :-)




[1] A reverse debugger is not the process of adding bugs to code *wink* 


-- 
Steve

From J.Demeyer at UGent.be  Sat Oct 27 15:45:36 2018
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Sat, 27 Oct 2018 21:45:36 +0200
Subject: [Python-Dev] Python Language Governance Proposals
In-Reply-To: <a51c48266aa7497b8e15bfb64befa192@xmail103.UGent.be>
References: <622cf62bf10448b682011baff41ad863@xmail103.UGent.be>
 <5BCF825C.70603@UGent.be>
 <a51c48266aa7497b8e15bfb64befa192@xmail103.UGent.be>
Message-ID: <5BD4C060.6040404@UGent.be>

On 2018-10-26 19:17, Brett Cannon wrote:
> But since you're asking about wanting to "review PEPs", you can review
> them now.

Unfortunately not everybody agrees on that...

See 
https://mail.python.org/pipermail/python-dev/2018-October/155441.html in 
particular

I really hope that I won't have to wait 5 more months before a decision 
can be made on PEP 580.

From vstinner at redhat.com  Sun Oct 28 12:20:00 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Sun, 28 Oct 2018 17:20:00 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
Message-ID: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>

Hi,

Python C API has no strict separation between the 3 levels of API:

* core: Py_BUILD_CORE define
* stable: Py_LIMITED_API define (it has a value)
* regular: no define

IMHO the current design of header files is done backward: by default,
everything is public. To exclude an API from core or stable, "#ifdef
Py_BUILD_CORE" and "#ifndef Py_LIMITED_API" are used. This design
caused issues in the past: functions, variables or something else
exposed whereas they were supposed to be "private".

I propose a practical solution for that: Include/*.h files would only
be be public API. The "core" API would live in a new subdirectory:
Include/pycore/*.h. Moreover, files of this subdirectory would have
the prefix "pycore_". For example, Include/objimpl.h would have a
twin: Include/pycore/pycore_objimpl.h which extend the public API with
"core" APIs.

I also propose to automatically load the twin: Include/objimpl.h would
load Include/pycore/pycore_objimpl.h if Py_BUILD_CORE is defined:

#ifdef Py_BUILD_CORE
#  include "pycore/pycore_objimpl.h"
#endif

Only in some rare cases, you would have to explicitly use: #include
"pycore/pycore_pygetopt.h". This header is fully private, there is no
public header in Include/pygetopt.h. Or maybe we should modify
Include/Python.h to also include "pycore/pycore_pygetopt.h" if
Py_BUILD_CORE is defined? Well, that's just a detail.

First milestone:

* Create Include/pycore/
* Move Py_BUILD_CORE specific code into Include/pycore/pycore_*.h
* Automatically include pycore files from Include/*.h files (#ifdef
Py_BUILD_CORE)

Second milestone:

* Find a solution for Py_LIMITED_API :-)

Backward compatibility? The overall change is fully backward
compatible. The default API doesn't change. C code (using header fles)
doesn't have to be changed.

Only a specific kinds of applications like debugger may have to be
modified if they really have to access the "core" API, the "most
private" API. Honestly, today it's unclear to me if this API can
technically be used outside CPython.

Victor

From vstinner at redhat.com  Sun Oct 28 12:23:08 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Sun, 28 Oct 2018 17:23:08 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
Message-ID: <CA+3bQGE-WW68wHtF4SfLvohA+nq5TKVC0z2Yzt_MzHmxoXi_Ww@mail.gmail.com>

Oh, I forgot to add a reference to the bugs.python.org issue and my
pull request!

* https://bugs.python.org/issue35081
* https://github.com/python/cpython/pull/10143

My PR more or less implements the first milestone of my plan
(Py_BUILD_CORE): it creates Include/pycore/.

Victor
Le dim. 28 oct. 2018 ? 17:20, Victor Stinner <vstinner at redhat.com> a ?crit :
>
> Hi,
>
> Python C API has no strict separation between the 3 levels of API:
>
> * core: Py_BUILD_CORE define
> * stable: Py_LIMITED_API define (it has a value)
> * regular: no define
>
> IMHO the current design of header files is done backward: by default,
> everything is public. To exclude an API from core or stable, "#ifdef
> Py_BUILD_CORE" and "#ifndef Py_LIMITED_API" are used. This design
> caused issues in the past: functions, variables or something else
> exposed whereas they were supposed to be "private".
>
> I propose a practical solution for that: Include/*.h files would only
> be be public API. The "core" API would live in a new subdirectory:
> Include/pycore/*.h. Moreover, files of this subdirectory would have
> the prefix "pycore_". For example, Include/objimpl.h would have a
> twin: Include/pycore/pycore_objimpl.h which extend the public API with
> "core" APIs.
>
> I also propose to automatically load the twin: Include/objimpl.h would
> load Include/pycore/pycore_objimpl.h if Py_BUILD_CORE is defined:
>
> #ifdef Py_BUILD_CORE
> #  include "pycore/pycore_objimpl.h"
> #endif
>
> Only in some rare cases, you would have to explicitly use: #include
> "pycore/pycore_pygetopt.h". This header is fully private, there is no
> public header in Include/pygetopt.h. Or maybe we should modify
> Include/Python.h to also include "pycore/pycore_pygetopt.h" if
> Py_BUILD_CORE is defined? Well, that's just a detail.
>
> First milestone:
>
> * Create Include/pycore/
> * Move Py_BUILD_CORE specific code into Include/pycore/pycore_*.h
> * Automatically include pycore files from Include/*.h files (#ifdef
> Py_BUILD_CORE)
>
> Second milestone:
>
> * Find a solution for Py_LIMITED_API :-)
>
> Backward compatibility? The overall change is fully backward
> compatible. The default API doesn't change. C code (using header fles)
> doesn't have to be changed.
>
> Only a specific kinds of applications like debugger may have to be
> modified if they really have to access the "core" API, the "most
> private" API. Honestly, today it's unclear to me if this API can
> technically be used outside CPython.
>
> Victor

From benjamin at python.org  Sun Oct 28 16:40:07 2018
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 28 Oct 2018 13:40:07 -0700
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
Message-ID: <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>



On Sun, Oct 28, 2018, at 09:20, Victor Stinner wrote:
> Hi,
> 
> Python C API has no strict separation between the 3 levels of API:
> 
> * core: Py_BUILD_CORE define
> * stable: Py_LIMITED_API define (it has a value)
> * regular: no define
> 
> IMHO the current design of header files is done backward: by default,
> everything is public. To exclude an API from core or stable, "#ifdef
> Py_BUILD_CORE" and "#ifndef Py_LIMITED_API" are used. This design
> caused issues in the past: functions, variables or something else
> exposed whereas they were supposed to be "private".
> 
> I propose a practical solution for that: Include/*.h files would only
> be be public API. The "core" API would live in a new subdirectory:
> Include/pycore/*.h. Moreover, files of this subdirectory would have
> the prefix "pycore_". For example, Include/objimpl.h would have a
> twin: Include/pycore/pycore_objimpl.h which extend the public API with
> "core" APIs.
> 
> I also propose to automatically load the twin: Include/objimpl.h would
> load Include/pycore/pycore_objimpl.h if Py_BUILD_CORE is defined:
> 
> #ifdef Py_BUILD_CORE
> #  include "pycore/pycore_objimpl.h"
> #endif

I don't think more or less API should be magically included based on whether Py_BUILD_CORE is defined or not. If we want to have private headers, we should include them where needed and not install them. Really, Py_BUILD_CORE should go away. We should be moving away from monolithic includes like Python.h to having each C file include exactly what it uses, private or not.

From vstinner at redhat.com  Sun Oct 28 17:30:53 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Sun, 28 Oct 2018 22:30:53 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>
Message-ID: <CA+3bQGGkUMPOfy_7_O3YC0i4tMx-gTwt7tb89fGhvJ3xuVLhjQ@mail.gmail.com>

Le dim. 28 oct. 2018 ? 21:50, Benjamin Peterson <benjamin at python.org> a ?crit :
> I don't think more or less API should be magically included based on whether Py_BUILD_CORE is defined or not. If we want to have private headers, we should include them where needed and not install them. Really, Py_BUILD_CORE should go away. We should be moving away from monolithic includes like Python.h to having each C file include exactly what it uses, private or not.

I would prefer to avoid annoying with the backward compatibility.
Currently, #include <Python.h> more or less provides you "anything"
and I'm fine with that.

I prefer to no put too many changes at once :-)

My overall approach is to make sure that we don't leak functions by
mistakes into the public API or into the stable API anymore. For
example, if a function is really for the core, put it in pycore/. It
will be more explicit when reviewing a change for example.

Py_BUILD_CORE is not only used to select which functions you get.
Py_BUILD_CORE is also commonly used to get a macro instead of a
function call, for best performances.

Victor

From benjamin at python.org  Mon Oct 29 01:32:02 2018
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 28 Oct 2018 22:32:02 -0700
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGGkUMPOfy_7_O3YC0i4tMx-gTwt7tb89fGhvJ3xuVLhjQ@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>
 <CA+3bQGGkUMPOfy_7_O3YC0i4tMx-gTwt7tb89fGhvJ3xuVLhjQ@mail.gmail.com>
Message-ID: <1540791122.3282066.1557851872.38471E47@webmail.messagingengine.com>



On Sun, Oct 28, 2018, at 14:30, Victor Stinner wrote:
> Le dim. 28 oct. 2018 ? 21:50, Benjamin Peterson <benjamin at python.org> a ?crit :
> > I don't think more or less API should be magically included based on whether Py_BUILD_CORE is defined or not. If we want to have private headers, we should include them where needed and not install them. Really, Py_BUILD_CORE should go away. We should be moving away from monolithic includes like Python.h to having each C file include exactly what it uses, private or not.
> 
> I would prefer to avoid annoying with the backward compatibility.
> Currently, #include <Python.h> more or less provides you "anything"
> and I'm fine with that.

It doesn't break backward compatibility if you only make this required for new APIs.

> 
> I prefer to no put too many changes at once :-)
> 
> My overall approach is to make sure that we don't leak functions by
> mistakes into the public API or into the stable API anymore. For
> example, if a function is really for the core, put it in pycore/. It
> will be more explicit when reviewing a change for example.

How does the current Include/internal/ directory fail at accomplishing your goal?

> 
> Py_BUILD_CORE is not only used to select which functions you get.
> Py_BUILD_CORE is also commonly used to get a macro instead of a
> function call, for best performances.

I think the macro and function versions should just have different names then.

From vstinner at redhat.com  Mon Oct 29 08:38:41 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 29 Oct 2018 13:38:41 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <1540791122.3282066.1557851872.38471E47@webmail.messagingengine.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>
 <CA+3bQGGkUMPOfy_7_O3YC0i4tMx-gTwt7tb89fGhvJ3xuVLhjQ@mail.gmail.com>
 <1540791122.3282066.1557851872.38471E47@webmail.messagingengine.com>
Message-ID: <CA+3bQGGJTom5=a0VAe8dHpvUP5KPO219G7zdUKwuDX-VGm4THA@mail.gmail.com>

Le lun. 29 oct. 2018 ? 06:32, Benjamin Peterson <benjamin at python.org> a ?crit :
> > My overall approach is to make sure that we don't leak functions by
> > mistakes into the public API or into the stable API anymore. For
> > example, if a function is really for the core, put it in pycore/. It
> > will be more explicit when reviewing a change for example.
>
> How does the current Include/internal/ directory fail at accomplishing your goal?

Hum, let me understand how I came into this issue. I tried to convert
_PyObject_GC_TRACK() macro to a static inline function. The macro uses
"_PyGC_generation0" which is defined earlier as: "extern PyGC_Head
*_PyGC_generation0;". Problem: this symbol has been removed when Eric
Snow moved it into _PyRuntime which contains "#define
_PyGC_generation0 _PyRuntime.gc.generation0".

Hum, how is possible that _PyObject_GC_TRACK() of objimpl.h works as expected?

It seems like all C file using this macro explicitly uses #include
"internal/pystate.h" which uses #include "internal/mem.h". Oh.

To me, it seems wrong that a function or macro defined in
Include/objimpl.h requires an explicit #include "internal/pystate.h".
objimpl.h should be self-sufficient.

I started to hack Include/internals/*.h, but then I have been traped
by #include which is relative: an include from Include/internals/
first looks for the included file in Include/internals/. It means that
Include/internals/mem.h cannot include Include/objimpl.h if
Include/internals/objimpl.h also exists.

Well, I would like to reorganize all these headers to make them more
consistent, and converting macros to static inline functions force me
to fix dependencies between header files.


> > Py_BUILD_CORE is not only used to select which functions you get.
> > Py_BUILD_CORE is also commonly used to get a macro instead of a
> > function call, for best performances.
>
> I think the macro and function versions should just have different names then.

I don't want to break the backward compatibility. I would like to make
small steps towards a better API. Concrete example with pystate.h:

/* Variable and macro for in-line access to current thread state */

/* Assuming the current thread holds the GIL, this is the
   PyThreadState for the current thread. */
#ifdef Py_BUILD_CORE
#  define _PyThreadState_Current _PyRuntime.gilstate.tstate_current
#  define PyThreadState_GET() \
             ((PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current))
#else
#  define PyThreadState_GET() PyThreadState_Get()
#endif


We cannot remove PyThreadState_GET() from the Python C API. Removing
any function is likely going to break multiple C extensions.

I don't want to change too many things at once.

My first intent is to convert _PyObject_GC_TRACK() into a static
function, not to break the Python C API :-)

Victor

From me at kennethreitz.org  Mon Oct 29 07:04:06 2018
From: me at kennethreitz.org (Kenneth Reitz)
Date: Mon, 29 Oct 2018 07:04:06 -0400
Subject: [Python-Dev] Helping with Documentation
Message-ID: <5bd6e8b7.1c69fb81.65540.052b@mx.google.com>

Hello all,

I?d like to become a core contributor to Python, by contributing polish to its documentation (adding missing pieces, modernize it a bit in spots, add more usage examples (itertools), etc).

Is anyone already working on this, and if so, can I join forces with you?

--
Kenneth Reitz

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181029/7717d154/attachment.html>

From brian at python.org  Mon Oct 29 12:05:45 2018
From: brian at python.org (Brian Curtin)
Date: Mon, 29 Oct 2018 10:05:45 -0600
Subject: [Python-Dev] Helping with Documentation
In-Reply-To: <5bd6e8b7.1c69fb81.65540.052b@mx.google.com>
References: <5bd6e8b7.1c69fb81.65540.052b@mx.google.com>
Message-ID: <CAD+XWwo4AQ9O4FATMh7RVNe2py14pKr448zmXyjWZid1ms0YvA@mail.gmail.com>

On Mon, Oct 29, 2018 at 9:36 AM Kenneth Reitz <me at kennethreitz.org> wrote:

> Hello all,
>
>
>
> I?d like to become a core contributor to Python, by contributing polish to
> its documentation (adding missing pieces, modernize it a bit in spots, add
> more usage examples (itertools), etc).
>
>
>
> Is anyone already working on this, and if so, can I join forces with you?
>

I don't know of any specific or active documentation focus right now (save
for translation, which is mostly what doc-sig has been about lately), but
giving the docs love has been on my todo list for quite a while now with
tiny efforts here or there. Your general idea sounds good enough to just go
for it and put up PRs.

Feel free to add me as a reviewer?@briancurtin?and I'll help get things
merged.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181029/d5cfb593/attachment.html>

From nad at python.org  Mon Oct 29 12:34:36 2018
From: nad at python.org (Ned Deily)
Date: Mon, 29 Oct 2018 12:34:36 -0400
Subject: [Python-Dev] Helping with Documentation
In-Reply-To: <5bd6e8b7.1c69fb81.65540.052b@mx.google.com>
References: <5bd6e8b7.1c69fb81.65540.052b@mx.google.com>
Message-ID: <DDBC4D59-4B5F-4205-896C-49A421596B4D@python.org>

On Oct 29, 2018, at 07:04, Kenneth Reitz <me at kennethreitz.org> wrote:
> I?d like to become a core contributor to Python, by contributing polish to its documentation (adding missing pieces, modernize it a bit in spots, add more usage examples (itertools), etc).
>  
> Is anyone already working on this, and if so, can I join forces with you?

Thanks for the offer, Kenneth, there's certain lots that could be done!  As you may know, Georg Brandl has been the Release Team documentation expert for many years, including providing Sphinx and the migration to it.  More recently, Georg has had to cut back drastically on his Python time and has given up the role.  Julian Palard has been assuming the doc expert role; he has been spearheading the overall documentation translation projects and for some time has been the go-to person for the on-line documentation system (docs.python.org).

As Brian already noted, the Docs Sig is the place for docs related work.  I would suggest you contact Julian and the Docs Sig.  They can also perhaps help with some of the history of the docs and some of the issues involved in changing them.  And, if you haven't seen it already, the Python Developer's Guide contains a lot of information about helping with the documentation and becoming a core contributor.

https://mail.python.org/mailman/listinfo/doc-sig
https://devguide.python.org/docquality/

--
  Ned Deily
  nad at python.org -- []


From armin.rigo at gmail.com  Mon Oct 29 12:49:39 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 29 Oct 2018 17:49:39 +0100
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in
 python debugger.
In-Reply-To: <20181026234806.GR3817@ando.pearwood.info>
References: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
 <20181026234806.GR3817@ando.pearwood.info>
Message-ID: <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>

Hi,

On Sat, 27 Oct 2018 at 01:50, Steven D'Aprano <steve at pearwood.info> wrote:
> [...]
> > So I was wondering if it would be possible to keep that context around
> > if you are in the debugger and rewind the execution point to before
> > the statement was triggered.
>
> I think what you are looking for is a reverse debugger[1] also known as
> a time-travel debugger.

I think it's a bit different.  A reverse debugger is here to debug
complex conditions in a (static) program.  What Ed is looking for is a
way to catch easy failures, fix the obviously faulty line, and
continue running the program.

Of course I can't help but mention to Ed that this is precisely the
kind of easy failures that are found by *testing* your code,
particularly if that's code that only runs after hours of other code
has executed.  *Never* trust yourself to write correct code if you
don't know that it is correct after waiting for hours.

But assuming that you really, really are allergic to tests, then what
you're looking for reminds me of long-ago Python experiments with
resumable exceptions and patching code at runtime.  Both topics are
abandoned now.  Resumable exceptions was a cool hack of the
interpreter that nobody really found a use for (AFAIR); patching code
at runtime comes with a pile of messes---it only works in the simple
cases, but there is no general solution for that.


A bient?t,

Armin.

From aixtools at felt.demon.nl  Mon Oct 29 14:51:42 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Mon, 29 Oct 2018 20:51:42 +0200
Subject: [Python-Dev] AIX build-bots broken for 11 days
Message-ID: <0D71CDC9-262A-4B49-A4AC-0D29888F0B7B@felt.demon.nl>

a) I wish I seen this earlier, but I have been away from OSS for the last weeks as my RL job has priority.

b) to answer a very different question I needed to look at the current (Always FAIL) status of the the two AIX buildbots - and I saw that 11 days ago they both started failing with the following messages:

renaming build/scripts-3.8/2to3 to build/scripts-3.8/2to3-3.8
./install-sh -c -m 644 ./Tools/gdb/libpython.py python-gdb.py
./install-sh: not found
make: 1254-004 The error code from the last command is 1.

renaming build/scripts-3.8/2to3 to build/scripts-3.8/2to3-3.8
./install-sh -c -m 644 ./Tools/gdb/libpython.py python-gdb.py
make: execvp: ./install-sh: The file access permissions do not allow the specified action.
Makefile:662: recipe for target 'python-gdb.py' failed

I am quite surprised, because, afaik, gdb is not used on either of the AIX bots. 
In any case

These two runs failed in the "normal" way

https://buildbot.python.org/all/#/builders/10/builds/1644/steps/4/logs/stdio
https://buildbot.python.org/all/#/builders/161/builds/327/steps/4/logs/stdio
And these fail during the build.

https://buildbot.python.org/all/#/builders/10/builds/1645/steps/2/logs/stdio

https://buildbot.python.org/all/#/builders/161/builds/328/steps/2/logs/stdio
Help appreciated.

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181029/468f4621/attachment.html>

From chris.jerdonek at gmail.com  Mon Oct 29 14:59:10 2018
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 29 Oct 2018 11:59:10 -0700
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in
 python debugger.
In-Reply-To: <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>
References: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
 <20181026234806.GR3817@ando.pearwood.info>
 <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>
Message-ID: <CAOTb1wfVn2Qau94KuaNgkfM5hvMZ-Ssg9S+hvOY=vqTrtd-o7w@mail.gmail.com>

A simpler feature that could possibly help him (assuming there isn't any
external state to deal with) would be the ability to save everything at a
certain point in time, and then resume it later. He could rig things up to
save the state e.g. after every hour: 1 hour, 2 hours, etc. Then if an
error occurs after 2.5 hours, he could at least start resuming after 2
hours. This could be viewed as a cheap form of a reverse debugger, because
a reverse debugger has to save the state at every point in time, not just
at a few select points.

--Chris

On Mon, Oct 29, 2018 at 9:51 AM Armin Rigo <armin.rigo at gmail.com> wrote:

> Hi,
>
> On Sat, 27 Oct 2018 at 01:50, Steven D'Aprano <steve at pearwood.info> wrote:
> > [...]
> > > So I was wondering if it would be possible to keep that context around
> > > if you are in the debugger and rewind the execution point to before
> > > the statement was triggered.
> >
> > I think what you are looking for is a reverse debugger[1] also known as
> > a time-travel debugger.
>
> I think it's a bit different.  A reverse debugger is here to debug
> complex conditions in a (static) program.  What Ed is looking for is a
> way to catch easy failures, fix the obviously faulty line, and
> continue running the program.
>
> Of course I can't help but mention to Ed that this is precisely the
> kind of easy failures that are found by *testing* your code,
> particularly if that's code that only runs after hours of other code
> has executed.  *Never* trust yourself to write correct code if you
> don't know that it is correct after waiting for hours.
>
> But assuming that you really, really are allergic to tests, then what
> you're looking for reminds me of long-ago Python experiments with
> resumable exceptions and patching code at runtime.  Both topics are
> abandoned now.  Resumable exceptions was a cool hack of the
> interpreter that nobody really found a use for (AFAIR); patching code
> at runtime comes with a pile of messes---it only works in the simple
> cases, but there is no general solution for that.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181029/5263aec2/attachment.html>

From storchaka at gmail.com  Mon Oct 29 15:38:49 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 29 Oct 2018 21:38:49 +0200
Subject: [Python-Dev] Standardize error message for non-pickleable types
Message-ID: <pr7ng7$aag$1@blaine.gmane.org>

When you try to to pickle or copy a non-pickleable object, you will get 
an error. In most cases this will be a TypeError with one of few 
similar, but different variants:

   "can't pickle XXX objects" (default)
   "Cannot serialize XXX object" (socket, BZ2Compressor, BZ2Decompressor)
   "can not serialize a 'XXX' object" (buffered files in _pyio)
   "cannot serialize 'XXX' object" (FileIO, TextWrapperIO, WinConsoleIO, 
buffered files in _io, LZMACompressor, LZMADecompressor)
   "cannot serialize {} object" (proposed for SSLContext)

Perhaps some of them where added without deep thinking and then were 
replicated in different places. I'm going to replace all of them with a 
standardized error message. But I'm unsure what variant is better.

1. "pickle" or "serialize"?

2. "can't", "Cannot", "can not" or "cannot"?

3. "object" or "objects"?

4. Use the "a" article or not?

5. Use quotes around type name or not?

Please help me to choose the best variant.


From vstinner at redhat.com  Mon Oct 29 15:51:34 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 29 Oct 2018 20:51:34 +0100
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <pr7ng7$aag$1@blaine.gmane.org>
References: <pr7ng7$aag$1@blaine.gmane.org>
Message-ID: <CA+3bQGFmcyA9PwxK+w7mBFmfPxCBTPm_+WUJ3rQ3jFf7-5k8fQ@mail.gmail.com>

Le lun. 29 oct. 2018 ? 20:42, Serhiy Storchaka <storchaka at gmail.com> a ?crit :
> 1. "pickle" or "serialize"?

serialize

> 2. "can't", "Cannot", "can not" or "cannot"?

cannot

> 3. "object" or "objects"?

object

> 4. Use the "a" article or not?

no: "cannot serialize xxx object" (but i'm not a native english
speaker, so don't trust me :-))

> 5. Use quotes around type name or not?

Use repr() in Python, but use '%s' is C since it would be too complex
to write code to properly implement repr() (decode tp_name from UTF-8,
handle error, call repr, handle error, etc.).

To use repr() on tp_name, I would prefer to have a new formatter, see
the thread of last month.
https://mail.python.org/pipermail/python-dev/2018-September/155150.html

Victor

From chris.jerdonek at gmail.com  Mon Oct 29 16:36:42 2018
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 29 Oct 2018 13:36:42 -0700
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in
 python debugger.
In-Reply-To: <CAOTb1wfVn2Qau94KuaNgkfM5hvMZ-Ssg9S+hvOY=vqTrtd-o7w@mail.gmail.com>
References: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
 <20181026234806.GR3817@ando.pearwood.info>
 <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>
 <CAOTb1wfVn2Qau94KuaNgkfM5hvMZ-Ssg9S+hvOY=vqTrtd-o7w@mail.gmail.com>
Message-ID: <CAOTb1wc_MQ12hPh31=BdYnnFY_tG_nYs3C7SMsc5g4ry4LcVBA@mail.gmail.com>

I have another idea on this. What about the idea of starting the program,
and then a few minutes later, starting the same program a second time. If
the first program errors, you could examine the second one which is a
little bit behind. Before starting the second one, perhaps you could even
make a copy of the second one and pause it for a few minutes before
restarting, in case the second one also errors out, and so on. This would
be more useful only in the case of deterministic errors.

--Chris


On Mon, Oct 29, 2018 at 11:59 AM Chris Jerdonek <chris.jerdonek at gmail.com>
wrote:

> A simpler feature that could possibly help him (assuming there isn't any
> external state to deal with) would be the ability to save everything at a
> certain point in time, and then resume it later. He could rig things up to
> save the state e.g. after every hour: 1 hour, 2 hours, etc. Then if an
> error occurs after 2.5 hours, he could at least start resuming after 2
> hours. This could be viewed as a cheap form of a reverse debugger, because
> a reverse debugger has to save the state at every point in time, not just
> at a few select points.
>
> --Chris
>
> On Mon, Oct 29, 2018 at 9:51 AM Armin Rigo <armin.rigo at gmail.com> wrote:
>
>> Hi,
>>
>> On Sat, 27 Oct 2018 at 01:50, Steven D'Aprano <steve at pearwood.info>
>> wrote:
>> > [...]
>> > > So I was wondering if it would be possible to keep that context around
>> > > if you are in the debugger and rewind the execution point to before
>> > > the statement was triggered.
>> >
>> > I think what you are looking for is a reverse debugger[1] also known as
>> > a time-travel debugger.
>>
>> I think it's a bit different.  A reverse debugger is here to debug
>> complex conditions in a (static) program.  What Ed is looking for is a
>> way to catch easy failures, fix the obviously faulty line, and
>> continue running the program.
>>
>> Of course I can't help but mention to Ed that this is precisely the
>> kind of easy failures that are found by *testing* your code,
>> particularly if that's code that only runs after hours of other code
>> has executed.  *Never* trust yourself to write correct code if you
>> don't know that it is correct after waiting for hours.
>>
>> But assuming that you really, really are allergic to tests, then what
>> you're looking for reminds me of long-ago Python experiments with
>> resumable exceptions and patching code at runtime.  Both topics are
>> abandoned now.  Resumable exceptions was a cool hack of the
>> interpreter that nobody really found a use for (AFAIR); patching code
>> at runtime comes with a pile of messes---it only works in the simple
>> cases, but there is no general solution for that.
>>
>>
>> A bient?t,
>>
>> Armin.
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181029/2777c2c1/attachment.html>

From barry at python.org  Mon Oct 29 16:39:48 2018
From: barry at python.org (Barry Warsaw)
Date: Mon, 29 Oct 2018 13:39:48 -0700
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <CA+3bQGFmcyA9PwxK+w7mBFmfPxCBTPm_+WUJ3rQ3jFf7-5k8fQ@mail.gmail.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <CA+3bQGFmcyA9PwxK+w7mBFmfPxCBTPm_+WUJ3rQ3jFf7-5k8fQ@mail.gmail.com>
Message-ID: <34230461-5D52-4DBA-A824-0876E66681F0@python.org>

On Oct 29, 2018, at 12:51, Victor Stinner <vstinner at redhat.com> wrote:
> 
>> 4. Use the "a" article or not?
> 
> no: "cannot serialize xxx object" (but i'm not a native english
> speaker, so don't trust me :-))

It should be fine to leave off the indefinite article.

> 
>> 5. Use quotes around type name or not?

Ideally yes, if it?s easy to implement.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181029/c4f73a8d/attachment-0001.sig>

From aixtools at felt.demon.nl  Mon Oct 29 16:40:32 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Mon, 29 Oct 2018 22:40:32 +0200
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
Message-ID: <76EEC6C3-1DDD-4AD7-8215-211BDE42F4E9@felt.demon.nl>

My short comment: +1

My longer comment: for someone who is not trying to be caught up in "internals" I find it extremely difficult to work with the "default" approach described below - trying to mentally understand, and remember what those macros mean/do as I got "bug-hunting".

Ultimately, have a clear-cut division between "public" and "internal" make it much much easier for "the rest of us" to know our boundaries and thereby develop code that is based on the "published" aka public API and not a (guess what I found) internal trick. Isn?t that why there is a "public" API to begin with.

Where, or how the separation is provided is not the key issue. But being committed to it is, and this sounds like a step towards commitment and clarity.

> On 10/28/2018 6:20 PM, Victor Stinner wrote:
> IMHO the current design of header files is done backward: by default,
> everything is public. To exclude an API from core or stable, "#ifdef
> Py_BUILD_CORE" and "#ifndef Py_LIMITED_API" are used. This design
> caused issues in the past: functions, variables or something else
> exposed whereas they were supposed to be "private".
> 
> I propose a practical solution for that: Include/*.h files would only
> be be public API. The "core" API would live in a new subdirectory:
> Include/pycore/*.h. Moreover, files of this subdirectory would have
> the prefix "pycore_". For example, Include/objimpl.h would have a
> twin: Include/pycore/pycore_objimpl.h which extend the public API with
> "core" APIs.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181029/ff3916da/attachment.html>

From v+python at g.nevcal.com  Mon Oct 29 16:05:58 2018
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Mon, 29 Oct 2018 13:05:58 -0700
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <CA+3bQGFmcyA9PwxK+w7mBFmfPxCBTPm_+WUJ3rQ3jFf7-5k8fQ@mail.gmail.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <CA+3bQGFmcyA9PwxK+w7mBFmfPxCBTPm_+WUJ3rQ3jFf7-5k8fQ@mail.gmail.com>
Message-ID: <55302dbe-e2fc-dacb-ae03-8c25130ac404@g.nevcal.com>

On 10/29/2018 12:51 PM, Victor Stinner wrote:
> Le lun. 29 oct. 2018 ? 20:42, Serhiy Storchaka <storchaka at gmail.com> a ?crit :
>> 1. "pickle" or "serialize"?
> serialize
>
>> 2. "can't", "Cannot", "can not" or "cannot"?
> cannot
>
>> 3. "object" or "objects"?
> object
>
>> 4. Use the "a" article or not?
> no: "cannot serialize xxx object" (but i'm not a native english
> speaker, so don't trust me :-))

Cannot serialize an object of type 'XXX'

>
>> 5. Use quotes around type name or not?
> Use repr() in Python, but use '%s' is C since it would be too complex
> to write code to properly implement repr() (decode tp_name from UTF-8,
> handle error, call repr, handle error, etc.).
>
> To use repr() on tp_name, I would prefer to have a new formatter, see
> the thread of last month.
> https://mail.python.org/pipermail/python-dev/2018-September/155150.html
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181029/0383adfe/attachment.html>

From python at mrabarnett.plus.com  Mon Oct 29 17:17:12 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 29 Oct 2018 21:17:12 +0000
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <pr7ng7$aag$1@blaine.gmane.org>
References: <pr7ng7$aag$1@blaine.gmane.org>
Message-ID: <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>

On 2018-10-29 19:38, Serhiy Storchaka wrote:
> When you try to to pickle or copy a non-pickleable object, you will get
> an error. In most cases this will be a TypeError with one of few
> similar, but different variants:
> 
>     "can't pickle XXX objects" (default)
>     "Cannot serialize XXX object" (socket, BZ2Compressor, BZ2Decompressor)
>     "can not serialize a 'XXX' object" (buffered files in _pyio)
>     "cannot serialize 'XXX' object" (FileIO, TextWrapperIO, WinConsoleIO,
> buffered files in _io, LZMACompressor, LZMADecompressor)
>     "cannot serialize {} object" (proposed for SSLContext)
> 
> Perhaps some of them where added without deep thinking and then were
> replicated in different places. I'm going to replace all of them with a
> standardized error message. But I'm unsure what variant is better.
> 
> 1. "pickle" or "serialize"?
> 
> 2. "can't", "Cannot", "can not" or "cannot"?
> 
> 3. "object" or "objects"?
> 
> 4. Use the "a" article or not?
> 
> 5. Use quotes around type name or not?
> 
> Please help me to choose the best variant.
> 
1. If you're pickling, then saying "pickle" is more helpful.

2. In English the usual long form is "cannot". Error messages tend to 
avoid abbreviations, and also tend to have lowercase after the colon, e.g.:

     "ZeroDivisionError: division by zero"

     "ValueError: invalid literal for int() with base 10: 'foo'"

3. If it's failing on an object (singular), then it's clearer to say 
"object".

4. Articles tend to be omitted.

5. Error messages tend to have quotes around the type name.

Therefore, my preference is for:

     "cannot pickle 'XXX' object"

From horos22 at gmail.com  Mon Oct 29 17:35:36 2018
From: horos22 at gmail.com (Ed Peschko)
Date: Mon, 29 Oct 2018 14:35:36 -0700
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in the
 python debugger.
Message-ID: <CACQ+YcuBVCWeDLzodahVGkazSqDn-yOLbcFC=sXyhxoMzWq-8A@mail.gmail.com>

Steve,

thanks for the response, and yes, I've experimented with reverse
debugging, and yes for the reasons specified in that article you gave
it isn't really practical with anything but small projects because of
the massive amounts of memory usage.

But that's really not what I'm asking for here. I'm simply asking that
python give developers the opportunity to stop execution and open up a
debugger *before* the exception is hit, and be able to instrument the
code at that point using that debugger.

The way that I mimic this in perl is by wrapping places where die,
confess, croak, AUTOLOAD, etc are hit with something that looks like
this:

sub _confess
{
    $DB::single = 1;
    my $trap = 1;
    if ($trap)
    {
        confess(@_);
    }
}

that way, I can short-circuit evals before they happen and save the
state to both examine and modify.

It's a hack, but it's a manageable hack since there are only so many
places where things can go off the rails in perl, and I cover each
exit. I can set $trap = 0 and lo-and-behold it will continue as if no
trap was issued.

Now, if there is global reverse debugging implemented in python, this
has got to be doable as well. I'd dare say it could be done with very
little overhead in memory.

> You know you can set breakpoints in the debugger? You don't have to
> single-step all the way through from the beginning. It isn't clear from
> your post how experienced you are.

ok just to be on the totally clear side, yes I'm quite aware of
pdb.set_trace(), etc.

They however don't do much to help the pain of debugging and
maintaining code which is *not your own*. For all good they are, you
might as well step through each line because you have no clue where
the mines in a particular script or library reside, and when you are
going to hit them.

And of course depending on how good the debugging implementation is,
when you hit these mines they may be well-nigh impossible to remove
without something like this.

Which btw is the main reason WHY I'm asking for this, and this is why
I think it would be such a productivity booster. Not for my code which
I control, but for others.


I'll take one particular experience here to make my point. I've been
experimenting with an automation framework written in python here that
shall otherwise remain nameless (except it is a fairly popular one).

Its ssh layer is *notoriously* obtuse, it freezes up on multiple
occasions,the support lists are clogged with issues surrounding it,
yet release-after-release things don't get fixed. Why? I'd argue
because the code is hard to parse but even harder to debug. Having
something like this would be a great boon to fixing any such issues.

So that's basically it.

No - I'm not looking for a full-blown reverse debugger for the reasons
you state. I'm looking for something more limited and therefore more
workable.

Ed

From tjreedy at udel.edu  Mon Oct 29 17:58:32 2018
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 29 Oct 2018 17:58:32 -0400
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
Message-ID: <pr7vm7$f5g$1@blaine.gmane.org>

On 10/29/2018 5:17 PM, MRAB wrote:
> On 2018-10-29 19:38, Serhiy Storchaka wrote:
>> When you try to to pickle or copy a non-pickleable object, you will get
>> an error. In most cases this will be a TypeError with one of few
>> similar, but different variants:
>>
>> ??? "can't pickle XXX objects" (default)
>> ??? "Cannot serialize XXX object" (socket, BZ2Compressor, 
>> BZ2Decompressor)
>> ??? "can not serialize a 'XXX' object" (buffered files in _pyio)
>> ??? "cannot serialize 'XXX' object" (FileIO, TextWrapperIO, WinConsoleIO,
>> buffered files in _io, LZMACompressor, LZMADecompressor)
>> ??? "cannot serialize {} object" (proposed for SSLContext)
>>
>> Perhaps some of them where added without deep thinking and then were
>> replicated in different places. I'm going to replace all of them with a
>> standardized error message.

Great idea.

>> But I'm unsure what variant is better.
>>
>> 1. "pickle" or "serialize"?
>>
>> 2. "can't", "Cannot", "can not" or "cannot"?
>>
>> 3. "object" or "objects"?
>>
>> 4. Use the "a" article or not?
>>
>> 5. Use quotes around type name or not?
>>
>> Please help me to choose the best variant.
>>
> 1. If you're pickling, then saying "pickle" is more helpful.
> 
> 2. In English the usual long form is "cannot". Error messages tend to 
> avoid abbreviations,

Agree x 3

> and also tend to have lowercase after the colon, e.g.:
> 
>  ??? "ZeroDivisionError: division by zero"
> 
>  ??? "ValueError: invalid literal for int() with base 10: 'foo'"

I had not noticed, but

IndexError: list index out of range
NameError: name 'sqrt' is not defined

> 3. If it's failing on an object (singular), then it's clearer to say 
> "object".
> 
> 4. Articles tend to be omitted.

Grammatically, the two examples above could/should start with 'The'. 
But that is routinely omitted.  Matching a/an to 'xxx' would be a 
terrible nuisance.  "a 'str'" (a string)?, "an 'str'" (an ess tee ar)?


> 5. Error messages tend to have quotes around the type name.
> 
> Therefore, my preference is for:
> 
>  ??? "cannot pickle 'XXX' object"
+1

-- 
Terry Jan Reedy



From greg.ewing at canterbury.ac.nz  Mon Oct 29 18:07:00 2018
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 30 Oct 2018 11:07:00 +1300
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in
 python debugger.
In-Reply-To: <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>
References: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
 <20181026234806.GR3817@ando.pearwood.info>
 <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>
Message-ID: <5BD78484.3050208@canterbury.ac.nz>

When I have a bug that only happens after hours of run
time, I try to find a much shorter test case that reproduces
it.

-- 
Greg

From steve at pearwood.info  Mon Oct 29 18:08:32 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 30 Oct 2018 09:08:32 +1100
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
Message-ID: <20181029220832.GZ3817@ando.pearwood.info>

On Mon, Oct 29, 2018 at 09:17:12PM +0000, MRAB wrote:

> Therefore, my preference is for:
> 
>     "cannot pickle 'XXX' object"

+1

I completely concur with MRAB. 


-- 
Steve

From steve at pearwood.info  Mon Oct 29 18:21:10 2018
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 30 Oct 2018 09:21:10 +1100
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <CA+3bQGFmcyA9PwxK+w7mBFmfPxCBTPm_+WUJ3rQ3jFf7-5k8fQ@mail.gmail.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <CA+3bQGFmcyA9PwxK+w7mBFmfPxCBTPm_+WUJ3rQ3jFf7-5k8fQ@mail.gmail.com>
Message-ID: <20181029222110.GA3817@ando.pearwood.info>

On Mon, Oct 29, 2018 at 08:51:34PM +0100, Victor Stinner wrote:
> Le lun. 29 oct. 2018 ? 20:42, Serhiy Storchaka <storchaka at gmail.com> a ?crit :
> > 1. "pickle" or "serialize"?
> 
> serialize

-1

Serializing is more general; pickle is merely one form of serializing:

https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats

When practical, error messages should be more specific, not less. We 
don't say "arithmetic operation by zero" for division by zero errors, we 
specify which arithmetic operation failed.

Unlike most serialization formats, "pickle" is both a noun (the name of 
the format) and the verb to convert to that format.


-- 
Steve

From njs at pobox.com  Mon Oct 29 20:09:22 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 29 Oct 2018 17:09:22 -0700
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in
 python debugger.
In-Reply-To: <CAOTb1wfVn2Qau94KuaNgkfM5hvMZ-Ssg9S+hvOY=vqTrtd-o7w@mail.gmail.com>
References: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
 <20181026234806.GR3817@ando.pearwood.info>
 <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>
 <CAOTb1wfVn2Qau94KuaNgkfM5hvMZ-Ssg9S+hvOY=vqTrtd-o7w@mail.gmail.com>
Message-ID: <CAPJVwBmtVXJ0t5dmdASeReJhHAE7-mn_vvd=zWmXmX3jm=6wfQ@mail.gmail.com>

On Mon, Oct 29, 2018 at 11:59 AM, Chris Jerdonek
<chris.jerdonek at gmail.com> wrote:
> A simpler feature that could possibly help him (assuming there isn't any
> external state to deal with) would be the ability to save everything at a
> certain point in time, and then resume it later. He could rig things up to
> save the state e.g. after every hour: 1 hour, 2 hours, etc. Then if an error
> occurs after 2.5 hours, he could at least start resuming after 2 hours. This
> could be viewed as a cheap form of a reverse debugger, because a reverse
> debugger has to save the state at every point in time, not just at a few
> select points.

That's very difficult to implement without help from the operating
system, but there is prior art...

On Unix, you can use fork() as a hacky way to do this ? fork() off a
process every once and a while, and have the children enter some kind
of loop waiting for instructions (continue / enter debugger / exit /
...).

Or, on Linux, there's: https://www.criu.org/

I also wonder if it would be useful to give pdb the ability to break
when an exception is *raised*, rather than when it's caught?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From rosuav at gmail.com  Mon Oct 29 20:27:02 2018
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 30 Oct 2018 11:27:02 +1100
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in
 python debugger.
In-Reply-To: <CAPJVwBmtVXJ0t5dmdASeReJhHAE7-mn_vvd=zWmXmX3jm=6wfQ@mail.gmail.com>
References: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
 <20181026234806.GR3817@ando.pearwood.info>
 <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>
 <CAOTb1wfVn2Qau94KuaNgkfM5hvMZ-Ssg9S+hvOY=vqTrtd-o7w@mail.gmail.com>
 <CAPJVwBmtVXJ0t5dmdASeReJhHAE7-mn_vvd=zWmXmX3jm=6wfQ@mail.gmail.com>
Message-ID: <CAPTjJmrWxL1JM-pQPmoCxmXJsKFN3C5W5svGY=XD5pt1ryGNhg@mail.gmail.com>

On Tue, Oct 30, 2018 at 11:10 AM Nathaniel Smith <njs at pobox.com> wrote:
> I also wonder if it would be useful to give pdb the ability to break
> when an exception is *raised*, rather than when it's caught?

This is veering into python-ideas territory (or even python-list), but
the first big concern that comes to my mind is that there are a LOT of
places where exceptions are raised, and many of those exceptions end
up being used for perfectly-normal flow control. So this would
potentially add even more overhead to the raising of exceptions -
basically, you have to retain state as if you're suspending a
generator. But it'd be an extremely cool concept. Exceptions already
snapshot all their locals, and this would just expand on that a bit.

ChrisA

From v+python at g.nevcal.com  Mon Oct 29 20:26:42 2018
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Mon, 29 Oct 2018 17:26:42 -0700
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in
 python debugger.
In-Reply-To: <5BD78484.3050208@canterbury.ac.nz>
References: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
 <20181026234806.GR3817@ando.pearwood.info>
 <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>
 <5BD78484.3050208@canterbury.ac.nz>
Message-ID: <09b85aa4-1016-858e-1aff-8ef1b103b005@g.nevcal.com>

On 10/29/2018 3:07 PM, Greg Ewing wrote:
> When I have a bug that only happens after hours of run
> time, I try to find a much shorter test case that reproduces
> it.
>
Mmm. Yeah.? But that's often a guessing game, with a low chance of success.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181029/75476e86/attachment-0001.html>

From python at mrabarnett.plus.com  Mon Oct 29 20:42:55 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 30 Oct 2018 00:42:55 +0000
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <20181029222110.GA3817@ando.pearwood.info>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <CA+3bQGFmcyA9PwxK+w7mBFmfPxCBTPm_+WUJ3rQ3jFf7-5k8fQ@mail.gmail.com>
 <20181029222110.GA3817@ando.pearwood.info>
Message-ID: <b1fc19ce-6c73-fa35-57e1-bd9b6180c971@mrabarnett.plus.com>

On 2018-10-29 22:21, Steven D'Aprano wrote:
> On Mon, Oct 29, 2018 at 08:51:34PM +0100, Victor Stinner wrote:
>> Le lun. 29 oct. 2018 ? 20:42, Serhiy Storchaka <storchaka at gmail.com> a ?crit :
>> > 1. "pickle" or "serialize"?
>> 
>> serialize
> 
> -1
> 
> Serializing is more general; pickle is merely one form of serializing:
> 
> https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats
> 
> When practical, error messages should be more specific, not less. We
> don't say "arithmetic operation by zero" for division by zero errors, we
> specify which arithmetic operation failed.
> 
> Unlike most serialization formats, "pickle" is both a noun (the name of
> the format) and the verb to convert to that format.
> 
And if you're marshalling, then saying "marshal" is more helpful.

From benjamin at python.org  Mon Oct 29 21:59:29 2018
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 29 Oct 2018 18:59:29 -0700
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGGJTom5=a0VAe8dHpvUP5KPO219G7zdUKwuDX-VGm4THA@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>
 <CA+3bQGGkUMPOfy_7_O3YC0i4tMx-gTwt7tb89fGhvJ3xuVLhjQ@mail.gmail.com>
 <1540791122.3282066.1557851872.38471E47@webmail.messagingengine.com>
 <CA+3bQGGJTom5=a0VAe8dHpvUP5KPO219G7zdUKwuDX-VGm4THA@mail.gmail.com>
Message-ID: <1540864769.232050.1559074016.633A5E14@webmail.messagingengine.com>



On Mon, Oct 29, 2018, at 05:38, Victor Stinner wrote:
> Le lun. 29 oct. 2018 ? 06:32, Benjamin Peterson <benjamin at python.org> a ?crit :
> > > My overall approach is to make sure that we don't leak functions by
> > > mistakes into the public API or into the stable API anymore. For
> > > example, if a function is really for the core, put it in pycore/. It
> > > will be more explicit when reviewing a change for example.
> >
> > How does the current Include/internal/ directory fail at accomplishing your goal?
> 
> Hum, let me understand how I came into this issue. I tried to convert
> _PyObject_GC_TRACK() macro to a static inline function. The macro uses
> "_PyGC_generation0" which is defined earlier as: "extern PyGC_Head
> *_PyGC_generation0;". Problem: this symbol has been removed when Eric
> Snow moved it into _PyRuntime which contains "#define
> _PyGC_generation0 _PyRuntime.gc.generation0".
> 
> Hum, how is possible that _PyObject_GC_TRACK() of objimpl.h works as expected?
> 
> It seems like all C file using this macro explicitly uses #include
> "internal/pystate.h" which uses #include "internal/mem.h". Oh.
> 
> To me, it seems wrong that a function or macro defined in
> Include/objimpl.h requires an explicit #include "internal/pystate.h".
> objimpl.h should be self-sufficient.

I agree. I would say nothing in Include/*.h should be including Include/internal/*.h. We should move things into the "public" headers as required to fix this.

> 
> I started to hack Include/internals/*.h, but then I have been traped
> by #include which is relative: an include from Include/internals/
> first looks for the included file in Include/internals/. It means that
> Include/internals/mem.h cannot include Include/objimpl.h if
> Include/internals/objimpl.h also exists.

Yeah, that's unfortunate. I suppose you could use <> includes.

> 
> Well, I would like to reorganize all these headers to make them more
> consistent, and converting macros to static inline functions force me
> to fix dependencies between header files.

How does the your /pycore/ proposal fix this?

From nad at python.org  Mon Oct 29 23:04:23 2018
From: nad at python.org (Ned Deily)
Date: Mon, 29 Oct 2018 23:04:23 -0400
Subject: [Python-Dev] Julien Palard joins the Python Release Team as
 Documentation Expert
Message-ID: <FE7DDD3F-6D96-4552-BF66-282A1057F6B2@python.org>

https://discuss.python.org/t/julien-palard-joins-the-python-release-team-as-documentation-expert/313

--
  Ned Deily
  nad at python.org -- []


From steve.dower at python.org  Mon Oct 29 23:40:36 2018
From: steve.dower at python.org (Steve Dower)
Date: Mon, 29 Oct 2018 20:40:36 -0700
Subject: [Python-Dev] short-circuiting runtime errors/exceptions in
 python debugger.
In-Reply-To: <CAPJVwBmtVXJ0t5dmdASeReJhHAE7-mn_vvd=zWmXmX3jm=6wfQ@mail.gmail.com>
References: <CACQ+Ycu0=YZ695cC_q9iC9MC8SjwRqNXdb+4cYHh9+4czQYqwQ@mail.gmail.com>
 <20181026234806.GR3817@ando.pearwood.info>
 <CAMSv6X1-1VPddp5bczireRioCyyGw6AzZ2qtm1x7mB0YWaJ-tg@mail.gmail.com>
 <CAOTb1wfVn2Qau94KuaNgkfM5hvMZ-Ssg9S+hvOY=vqTrtd-o7w@mail.gmail.com>
 <CAPJVwBmtVXJ0t5dmdASeReJhHAE7-mn_vvd=zWmXmX3jm=6wfQ@mail.gmail.com>
Message-ID: <83f74809-946f-386f-d5ad-a9c5b036e793@python.org>

On 29Oct2018 1709, Nathaniel Smith wrote:
> I also wonder if it would be useful to give pdb the ability to break
> when an exception is *raised*, rather than when it's caught?

This is basically the first feature I implemented as an intern at
Microsoft back in 2011 :) (for Visual Studio's Python debugger)

It should be in the ptvsd and PyDev.Debugger packages, so any IDEs that
use those for debugging will support it (though it's not my
implementation any more, as far as I know).

Specifically, breaking on an exception is trivial, but filtering out the
cases that have handlers on the stack is nearly impossible. We got close
enough with looking at the AST of each caller that we didn't try any
harder than that. If you know *where* you're expecting the exception,
you could even filter on line number and then break when that line is on
the stack but before unwinding.

Cheers,
Steve

From aixtools at felt.demon.nl  Tue Oct 30 02:08:13 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Tue, 30 Oct 2018 08:08:13 +0200
Subject: [Python-Dev] Custom AIX build last week...
Message-ID: <03AAB140-1B66-4FBF-A952-843A2F1BD7C6@felt.demon.nl>

I noticed that there was a "custom" build queued for my AIX build-bot last week. (https://buildbot.python.org/all/#/builders/159/builds/1).

It failed, but I hope that was due to the issue with install.sh.

If it could be run again - and if it fails again, please let me know what the test was, and I'll look at it on my server manually.

If it succeeds - I would also appreciate a heads-up!

Thanks,

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181030/d4d57d4c/attachment.html>

From aixtools at felt.demon.nl  Tue Oct 30 02:39:43 2018
From: aixtools at felt.demon.nl (Michael Felt)
Date: Tue, 30 Oct 2018 08:39:43 +0200
Subject: [Python-Dev] Custom AIX build last week...
In-Reply-To: <03AAB140-1B66-4FBF-A952-843A2F1BD7C6@felt.demon.nl>
References: <03AAB140-1B66-4FBF-A952-843A2F1BD7C6@felt.demon.nl>
Message-ID: <CFF73706-3697-42FB-B646-0A71E97B5145@felt.demon.nl>

I did a bit of digging and it seems this is related to issue35059, and something about "inline" processing.

Besides learning how to clone a PR manually (which I will need to do) - what are you hoping to find? As my bot is not using gcc ( but xlc) I could look at manually compiling a single file and add some extra flags (-qsource or -qlist) which generates a .lst file and shows what the compiler did (macro expansion, etc..)

If you would like me to attempt this - just let me know. Although it may take 24 hours from that request before I can fulfill it.

Regards,

Michael

> On 10/30/2018 8:08 AM, Michael Felt wrote:
> I noticed that there was a "custom" build queued for my AIX build-bot last week. (https://buildbot.python.org/all/#/builders/159/builds/1).
> 
> It failed, but I hope that was due to the issue with install.sh.
> 
> If it could be run again - and if it fails again, please let me know what the test was, and I'll look at it on my server manually.
> 
> If it succeeds - I would also appreciate a heads-up!
> 
> Thanks,
> 
> Michael
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/aixtools%40felt.demon.nl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181030/339eb625/attachment.html>

From christian at python.org  Tue Oct 30 03:48:07 2018
From: christian at python.org (Christian Heimes)
Date: Tue, 30 Oct 2018 08:48:07 +0100
Subject: [Python-Dev] Julien Palard joins the Python Release Team as
 Documentation Expert
In-Reply-To: <FE7DDD3F-6D96-4552-BF66-282A1057F6B2@python.org>
References: <FE7DDD3F-6D96-4552-BF66-282A1057F6B2@python.org>
Message-ID: <24aa8435-c457-da3e-e98a-41f8a75d669a@python.org>

On 30/10/2018 04.04, Ned Deily wrote:
> https://discuss.python.org/t/julien-palard-joins-the-python-release-team-as-documentation-expert/313

Welcome on board, Julien!

Thank you very much for helping out.


From storchaka at gmail.com  Tue Oct 30 04:12:45 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 30 Oct 2018 10:12:45 +0200
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
Message-ID: <ead58bb6-0360-186f-56e4-f165a535d186@gmail.com>

29.10.18 23:17, MRAB ????:
> 1. If you're pickling, then saying "pickle" is more helpful.
> 
> 2. In English the usual long form is "cannot". Error messages tend to 
> avoid abbreviations, and also tend to have lowercase after the colon, e.g.:
> 
>  ??? "ZeroDivisionError: division by zero"
> 
>  ??? "ValueError: invalid literal for int() with base 10: 'foo'"
> 
> 3. If it's failing on an object (singular), then it's clearer to say 
> "object".
> 
> 4. Articles tend to be omitted.
> 
> 5. Error messages tend to have quotes around the type name.
> 
> Therefore, my preference is for:
> 
>  ??? "cannot pickle 'XXX' object"

Thank you Matthew, I'll use your variant.

Will something change the fact that in all these cases the pickling will 
be failed not just for specific object, but for all instances of the 
specified type?

From storchaka at gmail.com  Tue Oct 30 04:12:45 2018
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 30 Oct 2018 10:12:45 +0200
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
Message-ID: <ead58bb6-0360-186f-56e4-f165a535d186@gmail.com>

29.10.18 23:17, MRAB ????:
> 1. If you're pickling, then saying "pickle" is more helpful.
> 
> 2. In English the usual long form is "cannot". Error messages tend to 
> avoid abbreviations, and also tend to have lowercase after the colon, e.g.:
> 
>  ??? "ZeroDivisionError: division by zero"
> 
>  ??? "ValueError: invalid literal for int() with base 10: 'foo'"
> 
> 3. If it's failing on an object (singular), then it's clearer to say 
> "object".
> 
> 4. Articles tend to be omitted.
> 
> 5. Error messages tend to have quotes around the type name.
> 
> Therefore, my preference is for:
> 
>  ??? "cannot pickle 'XXX' object"

Thank you Matthew, I'll use your variant.

Will something change the fact that in all these cases the pickling will 
be failed not just for specific object, but for all instances of the 
specified type?


From vstinner at redhat.com  Tue Oct 30 05:52:52 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 30 Oct 2018 10:52:52 +0100
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
Message-ID: <CA+3bQGH2i0Y+VOTws++063n=uhmuerZqqBvE-uqCaVS8Jce2fA@mail.gmail.com>

Le lun. 29 oct. 2018 ? 22:20, MRAB <python at mrabarnett.plus.com> a ?crit :
> 1. If you're pickling, then saying "pickle" is more helpful.

I'm not sure that it's really possible to know if the error occurs
while pickle is trying to serialize an object, or if it's a different
serialization protocol. We are talking about the very generic
__getstate__() method. I'm in favor of being more general and say
"cannot serialize".

Victor

From vstinner at redhat.com  Tue Oct 30 07:38:48 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 30 Oct 2018 12:38:48 +0100
Subject: [Python-Dev] Custom AIX build last week...
In-Reply-To: <03AAB140-1B66-4FBF-A952-843A2F1BD7C6@felt.demon.nl>
References: <03AAB140-1B66-4FBF-A952-843A2F1BD7C6@felt.demon.nl>
Message-ID: <CA+3bQGFrDVj_gM1ciTaF5-GtGFv1Q0w-sFQ1eCOqXo2dkV8ZKQ@mail.gmail.com>

I used a custom build to check if a fix repaired a buildbot. It's
unrelated to AIX, but inlining on Visual Studio in Debug mode, so
specific to Windows.

Victor
Le mar. 30 oct. 2018 ? 07:11, Michael Felt <aixtools at felt.demon.nl> a ?crit :
>
> I noticed that there was a "custom" build queued for my AIX build-bot last week. (https://buildbot.python.org/all/#/builders/159/builds/1).
>
> It failed, but I hope that was due to the issue with install.sh.
>
> If it could be run again - and if it fails again, please let me know what the test was, and I'll look at it on my server manually.
>
> If it succeeds - I would also appreciate a heads-up!
>
> Thanks,
>
> Michael
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com

From vstinner at redhat.com  Tue Oct 30 10:45:36 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 30 Oct 2018 15:45:36 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <1540864769.232050.1559074016.633A5E14@webmail.messagingengine.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>
 <CA+3bQGGkUMPOfy_7_O3YC0i4tMx-gTwt7tb89fGhvJ3xuVLhjQ@mail.gmail.com>
 <1540791122.3282066.1557851872.38471E47@webmail.messagingengine.com>
 <CA+3bQGGJTom5=a0VAe8dHpvUP5KPO219G7zdUKwuDX-VGm4THA@mail.gmail.com>
 <1540864769.232050.1559074016.633A5E14@webmail.messagingengine.com>
Message-ID: <CA+3bQGGbwi1Ot81c9DHFFKpZ0NsEc9f3uPXZHrqjR+cBL1oUQQ@mail.gmail.com>

Le mar. 30 oct. 2018 ? 02:59, Benjamin Peterson <benjamin at python.org> a ?crit :
> > To me, it seems wrong that a function or macro defined in
> > Include/objimpl.h requires an explicit #include "internal/pystate.h".
> > objimpl.h should be self-sufficient.
>
> I agree. I would say nothing in Include/*.h should be including Include/internal/*.h. We should move things into the "public" headers as required to fix this.

Hum. It seems like I'm trying to fix too many issues at the same time,
and I'm confused by different constraints which may be incompatible.

I started to cleanup and fix header files with the most
"straightforward" changes:
https://bugs.python.org/issue35081

I will see later if Include/internal/ still has to be rename or moved
(Serhiy Storchaka proposed to move internal/*.h header files out of
the Include/ directory, I'm not sure that it's really needed).

Victor

From v+python at g.nevcal.com  Tue Oct 30 13:49:38 2018
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 30 Oct 2018 10:49:38 -0700
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <ead58bb6-0360-186f-56e4-f165a535d186@gmail.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
 <ead58bb6-0360-186f-56e4-f165a535d186@gmail.com>
Message-ID: <4cbf557c-8d53-f3df-755a-6a289ecd4d84@g.nevcal.com>

On 10/30/2018 1:12 AM, Serhiy Storchaka wrote:
> 29.10.18 23:17, MRAB ????:
>> 1. If you're pickling, then saying "pickle" is more helpful.
>>
>> 2. In English the usual long form is "cannot". Error messages tend to 
>> avoid abbreviations, and also tend to have lowercase after the colon, 
>> e.g.:
>>
>> ???? "ZeroDivisionError: division by zero"
>>
>> ???? "ValueError: invalid literal for int() with base 10: 'foo'"
>>
>> 3. If it's failing on an object (singular), then it's clearer to say 
>> "object".
>>
>> 4. Articles tend to be omitted.
>>
>> 5. Error messages tend to have quotes around the type name.
>>
>> Therefore, my preference is for:
>>
>> ???? "cannot pickle 'XXX' object"
>
> Thank you Matthew, I'll use your variant.
>
> Will something change the fact that in all these cases the pickling 
> will be failed not just for specific object, but for all instances of 
> the specified type?
That's why I suggested "object of type 'XXX'", to leave the type in a 
more prominent position, as it is generally more important to the issue 
than the object.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181030/172d8578/attachment.html>

From python at mrabarnett.plus.com  Tue Oct 30 13:59:29 2018
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 30 Oct 2018 17:59:29 +0000
Subject: [Python-Dev] Standardize error message for non-pickleable types
In-Reply-To: <ead58bb6-0360-186f-56e4-f165a535d186@gmail.com>
References: <pr7ng7$aag$1@blaine.gmane.org>
 <7a0918bb-8ec7-41be-86a9-e58c3d4b9df3@mrabarnett.plus.com>
 <ead58bb6-0360-186f-56e4-f165a535d186@gmail.com>
Message-ID: <57986ccf-8170-d0e4-ad55-c48e569abd17@mrabarnett.plus.com>

On 2018-10-30 08:12, Serhiy Storchaka wrote:
> 29.10.18 23:17, MRAB ????:
>> 1. If you're pickling, then saying "pickle" is more helpful.
>> 
>> 2. In English the usual long form is "cannot". Error messages tend to 
>> avoid abbreviations, and also tend to have lowercase after the colon, e.g.:
>> 
>>  ??? "ZeroDivisionError: division by zero"
>> 
>>  ??? "ValueError: invalid literal for int() with base 10: 'foo'"
>> 
>> 3. If it's failing on an object (singular), then it's clearer to say 
>> "object".
>> 
>> 4. Articles tend to be omitted.
>> 
>> 5. Error messages tend to have quotes around the type name.
>> 
>> Therefore, my preference is for:
>> 
>>  ??? "cannot pickle 'XXX' object"
> 
> Thank you Matthew, I'll use your variant.
> 
> Will something change the fact that in all these cases the pickling will
> be failed not just for specific object, but for all instances of the
> specified type?
> 
Well, the other examples you gave did not say explicitly that all 
instances of that type would fail.

If you look at what 'hash' says:

 >>> hash(())
3527539
 >>> hash(([]))
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

that would suggest "TypeError: unpicklable type: 'list'", but I'm not 
sure I'd like too much of "unpicklable", "unmarshallable", 
"unserializable", etc. :-)

From ericsnowcurrently at gmail.com  Wed Oct 31 14:22:29 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 31 Oct 2018 12:22:29 -0600
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
Message-ID: <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>

On Sun, Oct 28, 2018 at 10:20 AM Victor Stinner <vstinner at redhat.com> wrote:
> Python C API has no strict separation between the 3 levels of API:
>
> * core: Py_BUILD_CORE define
> * stable: Py_LIMITED_API define (it has a value)
> * regular: no define
>
> IMHO the current design of header files is done backward: by default,
> everything is public. To exclude an API from core or stable, "#ifdef
> Py_BUILD_CORE" and "#ifndef Py_LIMITED_API" are used. This design
> caused issues in the past: functions, variables or something else
> exposed whereas they were supposed to be "private".

This is a great point.

> I propose a practical solution for that: Include/*.h files would only
> be be public API.

As we've already discussed, I'm entirely in favor of this. :)  In
fact, I was thinking along those same lines when I originally created
"Include/internal", when working on the runtime state consolidation.
When you originally shared your idea with me (2 years ago?) it made
perfect sense. :)

> The "core" API would live in a new subdirectory:
> Include/pycore/*.h.

I'm mostly -0 on this.  "pycore" is fine I suppose, but I think
"internal" is more obvious to people looking through the repo.  I can
imagine folks getting confused by having "core" in the directory name.

> Moreover, files of this subdirectory would have
> the prefix "pycore_". For example, Include/objimpl.h would have a
> twin: Include/pycore/pycore_objimpl.h which extend the public API with
> "core" APIs.

I'm not sure why this is necessary.  When I created Include/internal I
ran into some issues with relative includes.  However, I solved them
by doing the following (at Benjamin's recommendation, IIRC):  always
explicitly specify "internal/<...>.h" in includes, even in
Include/internal/*.h.

For example:

https://github.com/python/cpython/blob/master/Include/internal/pystate.h#L11
    #include "internal/ceval.h"
Note that a few lines above in that file I include the public header file:
    #include "pystate.h"

Taking this approach there were no issues with "relative" includes.
Incidentally, I originally tried to solve the problem of relative
includes with a prefix ("_" IIRC), but that proved to introduce other
issues.  Switching to using the explicit "internal/" worked great and
was more clear.

> I also propose to automatically load the twin: Include/objimpl.h would
> load Include/pycore/pycore_objimpl.h if Py_BUILD_CORE is defined:
>
> #ifdef Py_BUILD_CORE
> #  include "pycore/pycore_objimpl.h"
> #endif

During the runtime state consolidation I took a similar approach
initially, though at a less granular scale, which proved to be a
headache.  At first I added Include/internal/Python.h and then tried
to conditionally include it in Include/Python.h.  That ended up
confusing, problematic, and unnecessary.  At Benjamin's suggestion I
switched to explicitly including "internal/<...>.h" in .c files (only
where necessary).  The result is simpler and more clear, identifying
dependencies in source files more tightly.  It's also a bit more
idiomatic for well-written C code.

Here's an example of what I did:

https://github.com/python/cpython/blob/master/Python/ceval.c#L13
    #include "internal/pystate.h"

Your proposal is similar, though more granular.  And I suppose it is
reasonable as a short-term step for moving internal API out of the
public header files.  However, I agree with Benjamin that the ideal
would be to not have the public headers rely at all on the internal
ones.  I realize that I somewhat introduced the problem when I added
the internal headers, so I certainly don't have a lot of room to ask
anything here. :)  Regardless, an intermediate step of conditionally
including the private headers in the public ones might be okay if we
have a long-term plan on how to not include the private headers at
all.  I'd be interested in discussing that.

FWIW, in the existing internal headers I include some of the public
header files.  I don't recall if I did anything the other direction
(i.e. include internal headers conditionally in the public ones).

> Only in some rare cases, you would have to explicitly use: #include
> "pycore/pycore_pygetopt.h". This header is fully private, there is no
> public header in Include/pygetopt.h. Or maybe we should modify
> Include/Python.h to also include "pycore/pycore_pygetopt.h" if
> Py_BUILD_CORE is defined? Well, that's just a detail.

As noted above, I tried that and it didn't work out well.

> First milestone:
>
> * Create Include/pycore/

-0 (less clear)

> * Move Py_BUILD_CORE specific code into Include/pycore/pycore_*.h

+1

> * Automatically include pycore files from Include/*.h files (#ifdef Py_BUILD_CORE)

-0 (maybe okay as an intermediate fix)

> Second milestone:
>
> * Find a solution for Py_LIMITED_API :-)

+1 :)

-eric

From vstinner at redhat.com  Wed Oct 31 15:18:39 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 31 Oct 2018 20:18:39 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>
Message-ID: <CA+3bQGFpcUiu_oyZWrU9POKSVhKYh98F-caoFk1xnhQgR6frMA@mail.gmail.com>

Le mer. 31 oct. 2018 ? 19:22, Eric Snow <ericsnowcurrently at gmail.com> a ?crit :
> > I propose a practical solution for that: Include/*.h files would only
> > be be public API.
>
> As we've already discussed, I'm entirely in favor of this. :)  In
> fact, I was thinking along those same lines when I originally created
> "Include/internal", when working on the runtime state consolidation.
> When you originally shared your idea with me (2 years ago?) it made
> perfect sense. :)

I think we discussed that during the CPython sprint in September 2017
:-) But I only formalized a full plan at the sprint in September 2018:
http://pythoncapi.readthedocs.io/

> > The "core" API would live in a new subdirectory:
> > Include/pycore/*.h.
>
> I'm mostly -0 on this.  "pycore" is fine I suppose (...)

Ok.

> > Moreover, files of this subdirectory would have
> > the prefix "pycore_". For example, Include/objimpl.h would have a
> > twin: Include/pycore/pycore_objimpl.h which extend the public API with
> > "core" APIs.
>
> I'm not sure why this is necessary.  (...)
> https://github.com/python/cpython/blob/master/Include/internal/pystate.h#L11
>     #include "internal/ceval.h"
> Note that a few lines above in that file I include the public header file:
>     #include "pystate.h"

The last include doesn't work:
https://bugs.python.org/issue35081#msg328942

"""
Include/internal/pystate.h uses #include "pystate.h" to include
Include/pystate.h, but it tries to include itself
(Include/internal/pystate.h) which does nothing because of "#ifndef
Py_INTERNAL_PYSTATE_H #define Py_INTERNAL_PYSTATE_H ... #endif".

Remove the #ifndef #define to see the bug:
(...)

Compilation fails with:

In file included from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 ...
./Include/internal/pystate.h:5:21: error: #include nested too deeply
 #include "pystate.h"
"""

Using <pystate.h> has no effect, it stills looks for Include/internal/pystate.h.

IMHO the only solution is to use different names in Include/ and
Include/internal/, at least for the header files of Include/internal/
which also exist in Include/.

Rename Include/internal/pystate.h to Include/internal/pyruntime.h,
Include/internal/internal_pystate.h or
Include/internal/pycore_pystate.h.

If we add Include/internal/ to the header search path (gcc -I
Include/internal/), we can avoid redundant
"internal/internal_<name>.h" and just write "internal_<name>.h". I
proposed to rename "internal" to "pycore" to get: #include
"pycore_pystate.h" instead of #include "internal_pystate.h". But I
have no strong preference for "pycore" or "internal", both work for
me.

> > I also propose to automatically load the twin: Include/objimpl.h would
> > load Include/pycore/pycore_objimpl.h if Py_BUILD_CORE is defined:
> >
> > #ifdef Py_BUILD_CORE
> > #  include "pycore/pycore_objimpl.h"
> > #endif
>
> During the runtime state consolidation I took a similar approach
> initially, though at a less granular scale, which proved to be a
> headache.  At first I added Include/internal/Python.h and then tried
> to conditionally include it in Include/Python.h.  That ended up
> confusing, problematic, and unnecessary.  At Benjamin's suggestion I
> switched to explicitly including "internal/<...>.h" in .c files (only
> where necessary).  The result is simpler and more clear, identifying
> dependencies in source files more tightly.  It's also a bit more
> idiomatic for well-written C code.

Ok, let's try this approach. I have to add many #include, but I'm fine
to be very explicit about includes. One example (internal/mem.h):
https://github.com/python/cpython/pull/10249

I'm writing "try" rather than "do", because something Py_BUILD_CORE
core is mixed with public headers. Like #ifdef Py_BUILD_CORE ... #else
... #endif. See datetime.h for an example. It's not easy to separate
both implementations.
https://github.com/python/cpython/pull/10238


> > Second milestone:
> >
> > * Find a solution for Py_LIMITED_API :-)

My current idea is to:

* Remove all "#ifndef Py_LIMITED_API" and "#ifdef Py_BUILD_CORE" from
Include/*.h
* Move "#ifndef Py_LIMITED_API" code to Include/public/
* Include "Include/public/<name>.h" from "Include/<name>.h"

IMHO here we really want to automatically include
"Include/public/<name>.h" from "Include/<name>.h" to not impact the
public API in any way.

Py_BUILD_CORE is very different: it's only consumed inside Python, so
it's fine to "break the API" and force to add new includes.

Victor

From ericsnowcurrently at gmail.com  Wed Oct 31 15:39:51 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 31 Oct 2018 13:39:51 -0600
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGGJTom5=a0VAe8dHpvUP5KPO219G7zdUKwuDX-VGm4THA@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>
 <CA+3bQGGkUMPOfy_7_O3YC0i4tMx-gTwt7tb89fGhvJ3xuVLhjQ@mail.gmail.com>
 <1540791122.3282066.1557851872.38471E47@webmail.messagingengine.com>
 <CA+3bQGGJTom5=a0VAe8dHpvUP5KPO219G7zdUKwuDX-VGm4THA@mail.gmail.com>
Message-ID: <CALFfu7Cmwijh38n7bHtgVwxFeR1JOAEcbjx+EZAXmY9FHEJaow@mail.gmail.com>

On Mon, Oct 29, 2018 at 6:39 AM Victor Stinner <vstinner at redhat.com> wrote:
> Le lun. 29 oct. 2018 ? 06:32, Benjamin Peterson <benjamin at python.org> a ?crit :
> > How does the current Include/internal/ directory fail at accomplishing your goal?
>
> Hum, let me understand how I came into this issue. I tried to convert
> _PyObject_GC_TRACK() macro to a static inline function. The macro uses
> "_PyGC_generation0" which is defined earlier as: "extern PyGC_Head
> *_PyGC_generation0;". Problem: this symbol has been removed when Eric
> Snow moved it into _PyRuntime which contains "#define
> _PyGC_generation0 _PyRuntime.gc.generation0".

(FTR, _PyGC_generation0 is in Include/internal/mem.h.)

>
> Hum, how is possible that _PyObject_GC_TRACK() of objimpl.h works as expected?
>
> It seems like all C file using this macro explicitly uses #include
> "internal/pystate.h" which uses #include "internal/mem.h". Oh.
>
> To me, it seems wrong that a function or macro defined in
> Include/objimpl.h requires an explicit #include "internal/pystate.h".
> objimpl.h should be self-sufficient.

I agree. :)  FWIW, when I consolidated the runtime state I took the
approach of re-using existing symbols (where possible) to avoid churn.
That's why the code does not reference _PyRuntime directly.

So, the issue is that public header implicitly requires that the
source file include the internal headers in order to get the symbol.
That is definitely too convoluted.  However, it seems like a
consequence of the public header file relying on the internal runtime
state, which is what you're trying to fix.

Shouldn't _PyObject_GC_TRACK() be moved to an internal include file?
IIRC, there were several similar cases where API (under #ifndef
Py_LIMITED_API) relies on internal runtime state and should probably
be moved out of the public headers.  Any such cases should be fixed.

If any public API must rely on the internal runtime state then it
shouldn't rely on it directly like it currently does.  Instead that
state should be hidden behind public API that the public headers can
access.  Right?

> [snip]
>
> > > Py_BUILD_CORE is not only used to select which functions you get.
> > > Py_BUILD_CORE is also commonly used to get a macro instead of a
> > > function call, for best performances.
> >
> > I think the macro and function versions should just have different names then.
>
> I don't want to break the backward compatibility. I would like to make
> small steps towards a better API. Concrete example with pystate.h:
>
> /* Variable and macro for in-line access to current thread state */
>
> /* Assuming the current thread holds the GIL, this is the
>    PyThreadState for the current thread. */
> #ifdef Py_BUILD_CORE
> #  define _PyThreadState_Current _PyRuntime.gilstate.tstate_current
> #  define PyThreadState_GET() \
>              ((PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current))
> #else
> #  define PyThreadState_GET() PyThreadState_Get()
> #endif

This has a similar problem to _PyObject_GC_TRACK(): it is "public" API
that relies on internal runtime state and on the source code including
the internal headers.

> We cannot remove PyThreadState_GET() from the Python C API. Removing
> any function is likely going to break multiple C extensions.

Right.  There should be very few cases of public API that relies on
internal details.  In this case, could we split things up?  For
example:

"Include/pystate.h"

    // Under Py_BUILD_CORE this is replaced.
    #define PyThreadState_GET() PyThreadState_Get()

"Include/internal/pystate.h"

    #undef PyThreadState_GET
    #define _PyThreadState_Current _PyRuntime.gilstate.tstate_current
    #define PyThreadState_GET() \
               ((PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current))

-eric

From vstinner at redhat.com  Wed Oct 31 15:57:07 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 31 Oct 2018 20:57:07 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGFpcUiu_oyZWrU9POKSVhKYh98F-caoFk1xnhQgR6frMA@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>
 <CA+3bQGFpcUiu_oyZWrU9POKSVhKYh98F-caoFk1xnhQgR6frMA@mail.gmail.com>
Message-ID: <CA+3bQGEgMc1T+UTavAcONFxTsq=ERsCjmZYP7W_3W_yJTAhosQ@mail.gmail.com>

Ok, I proposed to rename internal header files, add an "internal_"
prefix, to avoid this issue:
https://github.com/python/cpython/pull/10263

My PR also adds Include/internal/ to search paths of header files.

Extract of the change:

diff --git a/Include/internal/pystate.h b/Include/internal/internal_pystate.h
--- a/Include/internal/pystate.h
+++ b/Include/internal/internal_pystate.h
@@ -7,9 +7,9 @@ extern "C" {

 #include "pystate.h"
 #include "pythread.h"

-#include "internal/mem.h"
-#include "internal/ceval.h"
-#include "internal/warnings.h"
+#include "internal_pymem.h"
+#include "internal_ceval.h"
+#include "internal_warnings.h"

With this PR, internal_pystate.h of Include/internal/ is able to
properly include pystate.h from Include/ (include the correct header
file).


For practical reasons, I added Include/internal/ to all projects of
the Visual Studio solution (I modified the properties common to all
projects).

Again, this is where I would like to replace "internal_" with
"pycore_": "internal" is too generic, there is a risk that a third
party project also the same header filename. "internal_hash.h" name is
quite general. There is a non-zero risk of conflict with a library. In
the past, we also had issues when compiling OpenSSL for example.

Maybe we can keep "Include/internal/" directory name, but add
"pycore_" rather than "internal_" to header filenames?

Victor

> > > Moreover, files of this subdirectory would have
> > > the prefix "pycore_". For example, Include/objimpl.h would have a
> > > twin: Include/pycore/pycore_objimpl.h which extend the public API with
> > > "core" APIs.
> >
> > I'm not sure why this is necessary.  (...)
> > https://github.com/python/cpython/blob/master/Include/internal/pystate.h#L11
> >     #include "internal/ceval.h"
> > Note that a few lines above in that file I include the public header file:
> >     #include "pystate.h"
>
> The last include doesn't work:
> https://bugs.python.org/issue35081#msg328942
>
> """
> Include/internal/pystate.h uses #include "pystate.h" to include
> Include/pystate.h, but it tries to include itself
> (Include/internal/pystate.h) which does nothing because of "#ifndef
> Py_INTERNAL_PYSTATE_H #define Py_INTERNAL_PYSTATE_H ... #endif".
>
> Remove the #ifndef #define to see the bug:
> (...)
>
> Compilation fails with:
>
> In file included from ./Include/internal/pystate.h:5,
>                  from ./Include/internal/pystate.h:5,
>                  from ./Include/internal/pystate.h:5,
>                  from ./Include/internal/pystate.h:5,
>                  from ./Include/internal/pystate.h:5,
>                  from ./Include/internal/pystate.h:5,
>                  from ./Include/internal/pystate.h:5,
>                  ...
> ./Include/internal/pystate.h:5:21: error: #include nested too deeply
>  #include "pystate.h"
> """
>
> Using <pystate.h> has no effect, it stills looks for Include/internal/pystate.h.
>
> IMHO the only solution is to use different names in Include/ and
> Include/internal/, at least for the header files of Include/internal/
> which also exist in Include/.
>
> Rename Include/internal/pystate.h to Include/internal/pyruntime.h,
> Include/internal/internal_pystate.h or
> Include/internal/pycore_pystate.h.
>
> If we add Include/internal/ to the header search path (gcc -I
> Include/internal/), we can avoid redundant
> "internal/internal_<name>.h" and just write "internal_<name>.h". I
> proposed to rename "internal" to "pycore" to get: #include
> "pycore_pystate.h" instead of #include "internal_pystate.h". But I
> have no strong preference for "pycore" or "internal", both work for
> me.

From vstinner at redhat.com  Wed Oct 31 16:05:42 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 31 Oct 2018 21:05:42 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CALFfu7Cmwijh38n7bHtgVwxFeR1JOAEcbjx+EZAXmY9FHEJaow@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>
 <CA+3bQGGkUMPOfy_7_O3YC0i4tMx-gTwt7tb89fGhvJ3xuVLhjQ@mail.gmail.com>
 <1540791122.3282066.1557851872.38471E47@webmail.messagingengine.com>
 <CA+3bQGGJTom5=a0VAe8dHpvUP5KPO219G7zdUKwuDX-VGm4THA@mail.gmail.com>
 <CALFfu7Cmwijh38n7bHtgVwxFeR1JOAEcbjx+EZAXmY9FHEJaow@mail.gmail.com>
Message-ID: <CA+3bQGH7gYToAdGyCAd9B1WTJwSdhU49CofEmAFMn49PC0nhmw@mail.gmail.com>

Le mer. 31 oct. 2018 ? 20:40, Eric Snow <ericsnowcurrently at gmail.com> a ?crit :
> On Mon, Oct 29, 2018 at 6:39 AM Victor Stinner <vstinner at redhat.com> wrote:
> > Le lun. 29 oct. 2018 ? 06:32, Benjamin Peterson <benjamin at python.org> a ?crit :
> > > How does the current Include/internal/ directory fail at accomplishing your goal?
> >
> > Hum, let me understand how I came into this issue. I tried to convert
> > _PyObject_GC_TRACK() macro to a static inline function. The macro uses
> > "_PyGC_generation0" which is defined earlier as: "extern PyGC_Head
> > *_PyGC_generation0;". Problem: this symbol has been removed when Eric
> > Snow moved it into _PyRuntime which contains "#define
> > _PyGC_generation0 _PyRuntime.gc.generation0".
>
> (FTR, _PyGC_generation0 is in Include/internal/mem.h.)

I pushed a first commit to "cleanup and fix" pystate.h:
https://github.com/python/cpython/commit/9204fb8623896cc5f68ae350784ee25e8a7b2184

* Remove _PyThreadState_Current
* Replace _PyThreadState_Current with _PyRuntime.gilstate.tstate_current

I prefer to be explicit about the reference to _PyRuntime. For
example, in my text editor, I'm able to follow the reference to
_PyRuntime. In a debugger, I'm also able to display
"_PyRuntime.gilstate.tstate_current", whereas gdb fails to display
"_PyThreadState_Current" value since it's a preprocessor only thing.


> > Hum, how is possible that _PyObject_GC_TRACK() of objimpl.h works as expected?
> >
> > It seems like all C file using this macro explicitly uses #include
> > "internal/pystate.h" which uses #include "internal/mem.h". Oh.
> >
> > To me, it seems wrong that a function or macro defined in
> > Include/objimpl.h requires an explicit #include "internal/pystate.h".
> > objimpl.h should be self-sufficient.
>
> I agree. :)  FWIW, when I consolidated the runtime state I took the
> approach of re-using existing symbols (where possible) to avoid churn.
> That's why the code does not reference _PyRuntime directly.

Well, we can now fix these issues.


> Shouldn't _PyObject_GC_TRACK() be moved to an internal include file?

I'm trying to do that, but I have the #include bug: if I have
Include/objimpl.h and Include/internal/objimpl.h, the compiler is
confused and picked the wrong file... And I cannot use #include
"objimpl.h" to include Include/objdump.h from
Include/internal/objdump.h.

... Again, that's why I would like to rename header files.

I will be even worse when I will add a second directory for the stable
ABI (second part of my plan).


> IIRC, there were several similar cases where API (under #ifndef
> Py_LIMITED_API) relies on internal runtime state and should probably
> be moved out of the public headers.  Any such cases should be fixed.

Yep. I'm trying to fix issues one by one. I started with the simplest changes.


> If any public API must rely on the internal runtime state then it
> shouldn't rely on it directly like it currently does.  Instead that
> state should be hidden behind public API that the public headers can
> access.  Right?

Right. Internal things must be moved to Include/internal/.


> This has a similar problem to _PyObject_GC_TRACK(): it is "public" API
> that relies on internal runtime state and on the source code including
> the internal headers.

_PyObject_GC_TRACK() is fine: since it's a private API, we can move it
to Include/internal/ and require an explicit include of the internal
header file.


> > We cannot remove PyThreadState_GET() from the Python C API. Removing
> > any function is likely going to break multiple C extensions.
>
> Right.  There should be very few cases of public API that relies on
> internal details.  In this case, could we split things up?  For
> example:
>
> "Include/pystate.h"
>
>     // Under Py_BUILD_CORE this is replaced.
>     #define PyThreadState_GET() PyThreadState_Get()
>
> "Include/internal/pystate.h"
>
>     #undef PyThreadState_GET
>     #define _PyThreadState_Current _PyRuntime.gilstate.tstate_current
>     #define PyThreadState_GET() \
>                ((PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current))

This API is really the worse. If we do that, it's unclear if we get
the optimized flavor or the PyThreadState_Get() function call... It
depends if "Include/internal/pystate.h" is included or not!?

My idea here is to add a new _PyThreadState_GET() macro which would
only be available in the private API. It would make sure that we pick
the right implementation.

Maybe PyThreadState_GET() can always be an alias to
PyThreadState_Get(). Just make sure that internals use
_PyThreadState_GET().

Victor

From ericsnowcurrently at gmail.com  Wed Oct 31 17:18:58 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 31 Oct 2018 15:18:58 -0600
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGEgMc1T+UTavAcONFxTsq=ERsCjmZYP7W_3W_yJTAhosQ@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>
 <CA+3bQGFpcUiu_oyZWrU9POKSVhKYh98F-caoFk1xnhQgR6frMA@mail.gmail.com>
 <CA+3bQGEgMc1T+UTavAcONFxTsq=ERsCjmZYP7W_3W_yJTAhosQ@mail.gmail.com>
Message-ID: <CALFfu7DqTYquUpv1Rk2Q8+Mi+OKoQsuKxWKOaC5Mk+Ffj+i2HQ@mail.gmail.com>

On Wed, Oct 31, 2018 at 1:57 PM Victor Stinner <vstinner at redhat.com> wrote:
>
> Ok, I proposed to rename internal header files, add an "internal_"
> prefix, to avoid this issue:
> https://github.com/python/cpython/pull/10263
>
> My PR also adds Include/internal/ to search paths of header files.
>
> Extract of the change:
>
> diff --git a/Include/internal/pystate.h b/Include/internal/internal_pystate.h
> --- a/Include/internal/pystate.h
> +++ b/Include/internal/internal_pystate.h
> @@ -7,9 +7,9 @@ extern "C" {
>
>  #include "pystate.h"
>  #include "pythread.h"
>
> -#include "internal/mem.h"
> -#include "internal/ceval.h"
> -#include "internal/warnings.h"
> +#include "internal_pymem.h"
> +#include "internal_ceval.h"
> +#include "internal_warnings.h"
>
> With this PR, internal_pystate.h of Include/internal/ is able to
> properly include pystate.h from Include/ (include the correct header
> file).
>
>
> For practical reasons, I added Include/internal/ to all projects of
> the Visual Studio solution (I modified the properties common to all
> projects).
>
> Again, this is where I would like to replace "internal_" with
> "pycore_": "internal" is too generic, there is a risk that a third
> party project also the same header filename. "internal_hash.h" name is
> quite general. There is a non-zero risk of conflict with a library. In
> the past, we also had issues when compiling OpenSSL for example.

I'm not sure of this being a real problem, but I don't see a problem
with mitigating the risk. :)

> Maybe we can keep "Include/internal/" directory name, but add
> "pycore_" rather than "internal_" to header filenames?

this sounds good to me.  thanks for chasing this down.

-eric

From ericsnowcurrently at gmail.com  Wed Oct 31 17:39:28 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 31 Oct 2018 15:39:28 -0600
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGH7gYToAdGyCAd9B1WTJwSdhU49CofEmAFMn49PC0nhmw@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <1540759207.1168671.1557524968.5CFA99F3@webmail.messagingengine.com>
 <CA+3bQGGkUMPOfy_7_O3YC0i4tMx-gTwt7tb89fGhvJ3xuVLhjQ@mail.gmail.com>
 <1540791122.3282066.1557851872.38471E47@webmail.messagingengine.com>
 <CA+3bQGGJTom5=a0VAe8dHpvUP5KPO219G7zdUKwuDX-VGm4THA@mail.gmail.com>
 <CALFfu7Cmwijh38n7bHtgVwxFeR1JOAEcbjx+EZAXmY9FHEJaow@mail.gmail.com>
 <CA+3bQGH7gYToAdGyCAd9B1WTJwSdhU49CofEmAFMn49PC0nhmw@mail.gmail.com>
Message-ID: <CALFfu7AJsoU6HG_43YGFkFu1jkXZZFDCNvJ6iPZR42eZpTDpww@mail.gmail.com>

On Wed, Oct 31, 2018 at 2:05 PM Victor Stinner <vstinner at redhat.com> wrote:
>
> Le mer. 31 oct. 2018 ? 20:40, Eric Snow <ericsnowcurrently at gmail.com> a ?crit :
> > On Mon, Oct 29, 2018 at 6:39 AM Victor Stinner <vstinner at redhat.com> wrote:
> > > Le lun. 29 oct. 2018 ? 06:32, Benjamin Peterson <benjamin at python.org> a ?crit :
> > > > How does the current Include/internal/ directory fail at accomplishing your goal?
> > >
> > > Hum, let me understand how I came into this issue. I tried to convert
> > > _PyObject_GC_TRACK() macro to a static inline function. The macro uses
> > > "_PyGC_generation0" which is defined earlier as: "extern PyGC_Head
> > > *_PyGC_generation0;". Problem: this symbol has been removed when Eric
> > > Snow moved it into _PyRuntime which contains "#define
> > > _PyGC_generation0 _PyRuntime.gc.generation0".
> >
> > (FTR, _PyGC_generation0 is in Include/internal/mem.h.)
>
> I pushed a first commit to "cleanup and fix" pystate.h:
> https://github.com/python/cpython/commit/9204fb8623896cc5f68ae350784ee25e8a7b2184

Thanks for that.  I don't have much time to review (typing w/left
hand, sleeping babe in right), so it may make sense to get feedback
from others (Nick?) before going too far down this road, just for a
sanity check.

>
> * Remove _PyThreadState_Current
> * Replace _PyThreadState_Current with _PyRuntime.gilstate.tstate_current
>
> I prefer to be explicit about the reference to _PyRuntime. For
> example, in my text editor, I'm able to follow the reference to
> _PyRuntime. In a debugger, I'm also able to display
> "_PyRuntime.gilstate.tstate_current", whereas gdb fails to display
> "_PyThreadState_Current" value since it's a preprocessor only thing.

As long as we avoid pulling internal API into public header files,
which is exactly the point of your efforts. :)

>
>
> > > Hum, how is possible that _PyObject_GC_TRACK() of objimpl.h works as expected?
> > >
> > > It seems like all C file using this macro explicitly uses #include
> > > "internal/pystate.h" which uses #include "internal/mem.h". Oh.
> > >
> > > To me, it seems wrong that a function or macro defined in
> > > Include/objimpl.h requires an explicit #include "internal/pystate.h".
> > > objimpl.h should be self-sufficient.
> >
> > I agree. :)  FWIW, when I consolidated the runtime state I took the
> > approach of re-using existing symbols (where possible) to avoid churn.
> > That's why the code does not reference _PyRuntime directly.
>
> Well, we can now fix these issues.

+1

>
>
> > Shouldn't _PyObject_GC_TRACK() be moved to an internal include file?
>
> I'm trying to do that, but I have the #include bug: if I have
> Include/objimpl.h and Include/internal/objimpl.h, the compiler is
> confused and picked the wrong file... And I cannot use #include
> "objimpl.h" to include Include/objdump.h from
> Include/internal/objdump.h.
>
> ... Again, that's why I would like to rename header files.
>
> I will be even worse when I will add a second directory for the stable
> ABI (second part of my plan).

+1

>
>
> > IIRC, there were several similar cases where API (under #ifndef
> > Py_LIMITED_API) relies on internal runtime state and should probably
> > be moved out of the public headers.  Any such cases should be fixed.
>
> Yep. I'm trying to fix issues one by one. I started with the simplest changes.

+1

>
>
> > If any public API must rely on the internal runtime state then it
> > shouldn't rely on it directly like it currently does.  Instead that
> > state should be hidden behind public API that the public headers can
> > access.  Right?
>
> Right. Internal things must be moved to Include/internal/.

OK, good.

>
>
> > This has a similar problem to _PyObject_GC_TRACK(): it is "public" API
> > that relies on internal runtime state and on the source code including
> > the internal headers.
>
> _PyObject_GC_TRACK() is fine: since it's a private API, we can move it
> to Include/internal/ and require an explicit include of the internal
> header file.
>
>
> > > We cannot remove PyThreadState_GET() from the Python C API. Removing
> > > any function is likely going to break multiple C extensions.
> >
> > Right.  There should be very few cases of public API that relies on
> > internal details.  In this case, could we split things up?  For
> > example:
> >
> > "Include/pystate.h"
> >
> >     // Under Py_BUILD_CORE this is replaced.
> >     #define PyThreadState_GET() PyThreadState_Get()
> >
> > "Include/internal/pystate.h"
> >
> >     #undef PyThreadState_GET
> >     #define _PyThreadState_Current _PyRuntime.gilstate.tstate_current
> >     #define PyThreadState_GET() \
> >                ((PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current))
>
> This API is really the worse. If we do that, it's unclear if we get
> the optimized flavor or the PyThreadState_Get() function call... It
> depends if "Include/internal/pystate.h" is included or not!?

Fair enough.  we just need a way to provide the optimized impl without
depending on internal API (e.g. _PyRuntime) in public header files,
right?  There should only be two or three cases where it's an issue
(i.e. involving actual public API).

>
> My idea here is to add a new _PyThreadState_GET() macro which would
> only be available in the private API. It would make sure that we pick
> the right implementation.
>
> Maybe PyThreadState_GET() can always be an alias to
> PyThreadState_Get(). Just make sure that internals use
> _PyThreadState_GET().

+1

FWIW, I think you're on the right track with all this. :)

-eric

>
> Victor

From vstinner at redhat.com  Wed Oct 31 18:27:49 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 31 Oct 2018 23:27:49 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CALFfu7DqTYquUpv1Rk2Q8+Mi+OKoQsuKxWKOaC5Mk+Ffj+i2HQ@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>
 <CA+3bQGFpcUiu_oyZWrU9POKSVhKYh98F-caoFk1xnhQgR6frMA@mail.gmail.com>
 <CA+3bQGEgMc1T+UTavAcONFxTsq=ERsCjmZYP7W_3W_yJTAhosQ@mail.gmail.com>
 <CALFfu7DqTYquUpv1Rk2Q8+Mi+OKoQsuKxWKOaC5Mk+Ffj+i2HQ@mail.gmail.com>
Message-ID: <CA+3bQGEE8Ji+CWNx-LL2ambgyqa2MQz781f+WX=yWbqLkWA=nA@mail.gmail.com>

Le mer. 31 oct. 2018 ? 22:19, Eric Snow <ericsnowcurrently at gmail.com> a ?crit :
> > Maybe we can keep "Include/internal/" directory name, but add
> > "pycore_" rather than "internal_" to header filenames?
>
> this sounds good to me.  thanks for chasing this down.

What do you prefer?

(A Include/internal/internal_pystate.h
(B) Include/internal/pycore_pystate.h
(C) Include/pycore/pycore_pystate.h

Victor

From ericsnowcurrently at gmail.com  Wed Oct 31 18:37:54 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 31 Oct 2018 16:37:54 -0600
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGEE8Ji+CWNx-LL2ambgyqa2MQz781f+WX=yWbqLkWA=nA@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>
 <CA+3bQGFpcUiu_oyZWrU9POKSVhKYh98F-caoFk1xnhQgR6frMA@mail.gmail.com>
 <CA+3bQGEgMc1T+UTavAcONFxTsq=ERsCjmZYP7W_3W_yJTAhosQ@mail.gmail.com>
 <CALFfu7DqTYquUpv1Rk2Q8+Mi+OKoQsuKxWKOaC5Mk+Ffj+i2HQ@mail.gmail.com>
 <CA+3bQGEE8Ji+CWNx-LL2ambgyqa2MQz781f+WX=yWbqLkWA=nA@mail.gmail.com>
Message-ID: <CALFfu7Atg1TZzGjAdaRVzXAdhG5MZ4o81AbKwK5G-N84qn8N5g@mail.gmail.com>

B
On Wed, Oct 31, 2018 at 4:28 PM Victor Stinner <vstinner at redhat.com> wrote:
>
> Le mer. 31 oct. 2018 ? 22:19, Eric Snow <ericsnowcurrently at gmail.com> a ?crit :
> > > Maybe we can keep "Include/internal/" directory name, but add
> > > "pycore_" rather than "internal_" to header filenames?
> >
> > this sounds good to me.  thanks for chasing this down.
>
> What do you prefer?
>
> (A Include/internal/internal_pystate.h
> (B) Include/internal/pycore_pystate.h
> (C) Include/pycore/pycore_pystate.h
>
> Victor

From vstinner at redhat.com  Wed Oct 31 19:36:09 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 1 Nov 2018 00:36:09 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CALFfu7Atg1TZzGjAdaRVzXAdhG5MZ4o81AbKwK5G-N84qn8N5g@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>
 <CA+3bQGFpcUiu_oyZWrU9POKSVhKYh98F-caoFk1xnhQgR6frMA@mail.gmail.com>
 <CA+3bQGEgMc1T+UTavAcONFxTsq=ERsCjmZYP7W_3W_yJTAhosQ@mail.gmail.com>
 <CALFfu7DqTYquUpv1Rk2Q8+Mi+OKoQsuKxWKOaC5Mk+Ffj+i2HQ@mail.gmail.com>
 <CA+3bQGEE8Ji+CWNx-LL2ambgyqa2MQz781f+WX=yWbqLkWA=nA@mail.gmail.com>
 <CALFfu7Atg1TZzGjAdaRVzXAdhG5MZ4o81AbKwK5G-N84qn8N5g@mail.gmail.com>
Message-ID: <CA+3bQGHoe9hoqXX146Uc=7WABLG+8vZ=33Lc63ONiM9jQ54OCw@mail.gmail.com>

Ok, thanks. I decided to remove the redundant "py", so I renamed
"pystate.h" to "pycore_state.h" (single "py", instead of
"pycore_pystate.h", double "py py").

I updated my PR:
https://github.com/python/cpython/pull/10263

* Rename Include/internal/ header files:

  * pyatomic.h -> pycore_atomic.h
  * ceval.h -> pycore_ceval.h
  * condvar.h -> pycore_condvar.h
  * context.h -> pycore_context.h
  * pygetopt.h -> pycore_getopt.h
  * gil.h -> pycore_gil.h
  * hamt.h -> pycore_hamt.h
  * hash.h -> pycore_hash.h
  * mem.h -> pycore_mem.h
  * pystate.h -> pycore_state.h
  * warnings.h -> pycore_warnings.h

* PCbuild project, Makefile.pre.in, Modules/Setup: add the
  Include/internal/ directory to the search paths of header files.
* Update include. For example, replace #include "internal/mem.h"
  with #include "pycore_mem.h".

Victor
Le mer. 31 oct. 2018 ? 23:38, Eric Snow <ericsnowcurrently at gmail.com> a ?crit :
>
> B
> On Wed, Oct 31, 2018 at 4:28 PM Victor Stinner <vstinner at redhat.com> wrote:
> >
> > Le mer. 31 oct. 2018 ? 22:19, Eric Snow <ericsnowcurrently at gmail.com> a ?crit :
> > > > Maybe we can keep "Include/internal/" directory name, but add
> > > > "pycore_" rather than "internal_" to header filenames?
> > >
> > > this sounds good to me.  thanks for chasing this down.
> >
> > What do you prefer?
> >
> > (A Include/internal/internal_pystate.h
> > (B) Include/internal/pycore_pystate.h
> > (C) Include/pycore/pycore_pystate.h
> >
> > Victor

From ericsnowcurrently at gmail.com  Wed Oct 31 21:35:09 2018
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 31 Oct 2018 19:35:09 -0600
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CA+3bQGHoe9hoqXX146Uc=7WABLG+8vZ=33Lc63ONiM9jQ54OCw@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>
 <CA+3bQGFpcUiu_oyZWrU9POKSVhKYh98F-caoFk1xnhQgR6frMA@mail.gmail.com>
 <CA+3bQGEgMc1T+UTavAcONFxTsq=ERsCjmZYP7W_3W_yJTAhosQ@mail.gmail.com>
 <CALFfu7DqTYquUpv1Rk2Q8+Mi+OKoQsuKxWKOaC5Mk+Ffj+i2HQ@mail.gmail.com>
 <CA+3bQGEE8Ji+CWNx-LL2ambgyqa2MQz781f+WX=yWbqLkWA=nA@mail.gmail.com>
 <CALFfu7Atg1TZzGjAdaRVzXAdhG5MZ4o81AbKwK5G-N84qn8N5g@mail.gmail.com>
 <CA+3bQGHoe9hoqXX146Uc=7WABLG+8vZ=33Lc63ONiM9jQ54OCw@mail.gmail.com>
Message-ID: <CALFfu7BrLThnaMXQEd+wSsmzka8o+OkcGhwzktamLzNpF=OkZg@mail.gmail.com>

On the one hand dropping redundancy in the filename is fine.  On the other
having the names mirror the public header files is valuable.  How about
leaving the base names alone and change the directory to "pyinternal"?

-eric

On Wed, Oct 31, 2018, 17:36 Victor Stinner <vstinner at redhat.com wrote:

> Ok, thanks. I decided to remove the redundant "py", so I renamed
> "pystate.h" to "pycore_state.h" (single "py", instead of
> "pycore_pystate.h", double "py py").
>
> I updated my PR:
> https://github.com/python/cpython/pull/10263
>
> * Rename Include/internal/ header files:
>
>   * pyatomic.h -> pycore_atomic.h
>   * ceval.h -> pycore_ceval.h
>   * condvar.h -> pycore_condvar.h
>   * context.h -> pycore_context.h
>   * pygetopt.h -> pycore_getopt.h
>   * gil.h -> pycore_gil.h
>   * hamt.h -> pycore_hamt.h
>   * hash.h -> pycore_hash.h
>   * mem.h -> pycore_mem.h
>   * pystate.h -> pycore_state.h
>   * warnings.h -> pycore_warnings.h
>
> * PCbuild project, Makefile.pre.in, Modules/Setup: add the
>   Include/internal/ directory to the search paths of header files.
> * Update include. For example, replace #include "internal/mem.h"
>   with #include "pycore_mem.h".
>
> Victor
> Le mer. 31 oct. 2018 ? 23:38, Eric Snow <ericsnowcurrently at gmail.com> a
> ?crit :
> >
> > B
> > On Wed, Oct 31, 2018 at 4:28 PM Victor Stinner <vstinner at redhat.com>
> wrote:
> > >
> > > Le mer. 31 oct. 2018 ? 22:19, Eric Snow <ericsnowcurrently at gmail.com>
> a ?crit :
> > > > > Maybe we can keep "Include/internal/" directory name, but add
> > > > > "pycore_" rather than "internal_" to header filenames?
> > > >
> > > > this sounds good to me.  thanks for chasing this down.
> > >
> > > What do you prefer?
> > >
> > > (A Include/internal/internal_pystate.h
> > > (B) Include/internal/pycore_pystate.h
> > > (C) Include/pycore/pycore_pystate.h
> > >
> > > Victor
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181031/eaa38d4a/attachment.html>

From vstinner at redhat.com  Wed Oct 31 22:40:50 2018
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 1 Nov 2018 03:40:50 +0100
Subject: [Python-Dev] Rename Include/internals/ to Include/pycore/
In-Reply-To: <CALFfu7BrLThnaMXQEd+wSsmzka8o+OkcGhwzktamLzNpF=OkZg@mail.gmail.com>
References: <CA+3bQGE7UriagGc_FYuM7ge5oGX+6+_w2c3HY1WDj8JY=6FuzQ@mail.gmail.com>
 <CALFfu7DFq=9QEaTT=sy-XK=B+X1f+qjo7YnMD37HVGRGKu54mg@mail.gmail.com>
 <CA+3bQGFpcUiu_oyZWrU9POKSVhKYh98F-caoFk1xnhQgR6frMA@mail.gmail.com>
 <CA+3bQGEgMc1T+UTavAcONFxTsq=ERsCjmZYP7W_3W_yJTAhosQ@mail.gmail.com>
 <CALFfu7DqTYquUpv1Rk2Q8+Mi+OKoQsuKxWKOaC5Mk+Ffj+i2HQ@mail.gmail.com>
 <CA+3bQGEE8Ji+CWNx-LL2ambgyqa2MQz781f+WX=yWbqLkWA=nA@mail.gmail.com>
 <CALFfu7Atg1TZzGjAdaRVzXAdhG5MZ4o81AbKwK5G-N84qn8N5g@mail.gmail.com>
 <CA+3bQGHoe9hoqXX146Uc=7WABLG+8vZ=33Lc63ONiM9jQ54OCw@mail.gmail.com>
 <CALFfu7BrLThnaMXQEd+wSsmzka8o+OkcGhwzktamLzNpF=OkZg@mail.gmail.com>
Message-ID: <CA+3bQGHXfDDq2hUBnDpBk8DqAA586vunHCJXiYERgNXksKnU1w@mail.gmail.com>

I pushed many changes. I moved most of the Py_BUILD_CORE code out of
Include/ header files.

Le jeu. 1 nov. 2018 ? 02:35, Eric Snow <ericsnowcurrently at gmail.com> a ?crit :
> On the one hand dropping redundancy in the filename is fine.  On the other having the names mirror the public header files is valuable.

Current content of Include/internal/:

pycore_accu.h
pycore_atomic.h
pycore_ceval.h
pycore_condvar.h
pycore_context.h
pycore_getopt.h
pycore_gil.h
pycore_hamt.h
pycore_hash.h
pycore_lifecycle.h
pycore_mem.h
pycore_pathconfig.h
pycore_state.h
pycore_warnings.h

I tried to find a kind of consistency. For example, there was
Include/internal/mem.h vs Internal/pymem.h.

I renamed Include/pyatomic.h to Include/internal/pycore_atomic.h.
Previously, the file had a "py" prefix to avoid the risk of having an
<atomic.h> header file coming from somewhere else, but now with the
"pycore_" prefix, it's very unlikely.

I renamed Include/accu.h to Include/internal/pycore_accu.h.

There are 4 header files which are in Include/ and Include/internal/
and the one in internal has no "py":

* pyhash.h and pycore_hash.h
* pylifecycle.h and pycore_lifecycle.h
* pymem.h and pycore_mem.h
* pystate.h and pycore_state.h

I wrote a PR to rename these 4 header files:
https://github.com/python/cpython/pull/10275


> How about leaving the base names alone and change the directory to "pyinternal"?

The name of the directory doesn't matter to fix the #include bug when
two header files have the same filename in two directories (especially
when Include/ and Include/internal/ have the same filename).

Note: Since I added Include/internal/ to the search paths for header
files, the name of the directory doesn't matter anymore. C code only
uses the filename without "internal/" to include an internal header:
#include "pycore_mem.h". Benjamin and you were opposed to change the
name, so I kept "Include/internal/".

Victor