From ezio.melotti at gmail.com  Fri Mar  1 02:05:55 2019
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Fri, 1 Mar 2019 08:05:55 +0100
Subject: [Python-Dev] Summary of Python tracker Issues
In-Reply-To: <q5ae7e$603g$1@blaine.gmane.org>
References: <20190208180753.1.782F2F2B9AA3561B@roundup.psfhosted.org>
 <CAEfz+TwdqQm829Q_2RWJq9QVDsnRX53JkFYVMqFP0G+OSqJKiA@mail.gmail.com>
 <CAK256p3ruU-RCv-k_XeXSsu+33oKKnE29YK=bK=pQBJb=7tEgg@mail.gmail.com>
 <q59m60$1i0v$1@blaine.gmane.org>
 <CAK256p1kP_D9AYYfCbcgOXp5tb7oMHVbZ8x7JBi9yO9QsO4B0g@mail.gmail.com>
 <bb74cef5-2393-dccb-a6c5-5e491bbf66be@g.nevcal.com>
 <q59oo7$71ib$1@blaine.gmane.org>
 <5d12e862-5398-5701-2503-b745c0875976@g.nevcal.com>
 <q5ae7e$603g$1@blaine.gmane.org>
Message-ID: <CACBhJdGqHTJCrZfGDtEnwBMs5JXTpjyNa04qiR5sOf4yb9OzbQ@mail.gmail.com>

On Fri, Mar 1, 2019 at 5:59 AM Terry Reedy <tjreedy at udel.edu> wrote:
>
> On 2/28/2019 6:54 PM, Glenn Linderman wrote:
>
> > There seems to be enough evidence that something went wrong somewhere,
> > though, and whoever maintains that process should start investigating,
> > but it would still be nice to get confirmation from a non-Google email
> > recipient whether they did or did not get the Summary messages.
> >
> > I wonder if there is a way to manually send them, and if the missing two
> > weeks of activity can be reported... once the sending problem is
> > understood and resolved.
>
> I posted a note to the core-workflow list, but I don't know if anyone
> with power or knowledge still reads it.
>

The tracker got migrated recently, and that's the most likely cause of
the missing reports.
We'll look into it and get them back :)

> To get a listing, go to the tracker search page, put
> 2019-02-09 to 2019-03-01
> in the date box, and change status to don't care.  At the moment, this
> returns 204 issues.
>
> --
> Terry Jan Reedy
>

From encukou at gmail.com  Fri Mar  1 05:41:03 2019
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 1 Mar 2019 11:41:03 +0100
Subject: [Python-Dev] Compile-time resolution of packages [Was: Another
 update for PEP 394...]
In-Reply-To: <CAGE7PNL3L_jrxMk0CKr2b0RQX6R3u9N7XyO+CeC8w2Y0cTuoxg@mail.gmail.com>
References: <9e69c6dc-07cd-2265-b4b8-b9f7e9f81b00@gmail.com>
 <a096f664-3000-272a-ae5f-7713c936aa2c@gmail.com>
 <37ba6931-faa0-0c9c-b9e5-067eb123e313@gmail.com>
 <58F34E40-11B8-4F36-AF7E-C9022D4F48DF@python.org>
 <CAGE7PNK0o1tG84q06VUttGOBNkd2g3ftmsUA1U3NscjcGFGRQg@mail.gmail.com>
 <20190226220418.b36jw33qthdv5i5l@python.ca>
 <CABVPEKqPoOycqrwVjfCZo3J+4bgF-o1i3WB7nM5P+a3zZ0UkbQ@mail.gmail.com>
 <CAGE7PNL3L_jrxMk0CKr2b0RQX6R3u9N7XyO+CeC8w2Y0cTuoxg@mail.gmail.com>
Message-ID: <37ff84c0-7b7d-485f-e061-9f9b8f96fdb4@gmail.com>

On 2/28/19 6:56 PM, Gregory P. Smith wrote:
> 
> On Wed, Feb 27, 2019 at 5:12 PM Toshio Kuratomi <a.badger at gmail.com 
> <mailto:a.badger at gmail.com>> wrote:
> 
> 
>     On Tue, Feb 26, 2019 at 2:07 PM Neil Schemenauer
>     <nas-python at arctrix.com <mailto:nas-python at arctrix.com>> wrote:
> 
>         On 2019-02-26, Gregory P. Smith wrote:
>          > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
>         <barry at python.org <mailto:barry at python.org>> wrote:
>          > For an OS distro provided interpreter, being able to restrict
>         its use to
>          > only OS distro provided software would be ideal (so ideal
>         that people who
>          > haven't learned the hard distro maintenance lessons may hate
>         me for it).
> 
>     This idea has some definite problems.? I think enforcing it via
>     convention is about as much as would be good to do.? Anything more
>     and you make it hard for people who really need to use the vendor
>     provided interpreter from being able to do so.
> 
>     Why might someone need to use the distro provided interpreter?
> 
>     * Vendor provides some python modules in their system packages which
>     are not installable from pip (possibly even a proprietary extension
>     module, so not even buildable from source or copyable from the
>     system location) which the end user needs to use to do something to
>     their system.
>     * End user writes a python module which is a plugin to a system tool
>     which has to be installed into the system python to from which that
>     system tool runs.? The user then wants to write a script which uses
>     the system tool with the plugin in order to do something to their
>     system outside of the system tool (perhaps the system tool is
>     GUI-driven and the user wants to automate a part of it via the
>     python module).? They need their script to use the system python so
>     that they are using the same code as the system tool itself would use.
> 
>     There's probably other scenarios where the benefits of locking the
>     user out of the system python outweigh the benefits but these are
>     the ones that I've run across lately.
> 
> 
> Agreed.? The convention approach as someone said RHEL 8 has apparently 
> done with an os distro reserved interpreter (yay) is likely good enough 
> for most situations.
> 
> I'd go a /little/ further than that and suggest such an os distro 
> reserved interpreter attempt to prevent installation of packages (ie: 
> remove pip/ensurepip/distutils) via any other means than the OS package 
> manager (rpms, debs).? Obviously that can't actually prevent someone 
> from figuring out how to run getpip or manually installing trees of 
> packages within its sys.path, but it acts as a deterrent suggesting that 
> this interpreter is not intended for arbitrary software installation.

Currently, in RHEL/Fedora:
- "sudo pip" installs to /usr/local/, RPM packages install to /usr/
- with "-I", the interpreter does not look into /usr/local/.
AFAIK, Debian/Ubuntu have something very similar, and were first to do it.

What remains to tie this together is making "platform-python" always run 
with -I. This is PEP 432's "system-python" use case / design goal.
(The RHEL/Fedora platform-python was initially called system-python. We 
renamed to it so it doesn't clash with the PEP. It's the same use case, 
but we don't want to assign semantics to the name prematurely. 
Cutrrently, system-python is waiting for the larger-scale restructuring, 
and hopefully wider/longer discussion.)


From aixtools at felt.demon.nl  Fri Mar  1 08:51:40 2019
From: aixtools at felt.demon.nl (Michael Felt)
Date: Fri, 1 Mar 2019 14:51:40 +0100
Subject: [Python-Dev] before I open an issue re: posix.stat and/or
 os.stat
In-Reply-To: <d229a52f-3c40-515b-917f-3c9a89d9f6a2@felt.demon.nl>
References: <d229a52f-3c40-515b-917f-3c9a89d9f6a2@felt.demon.nl>
Message-ID: <ECA5961E-5CED-4C8A-AF7E-5D311137C9DD@felt.demon.nl>

Shortening the original mail to something shorter.

The reason I am starting here, in -dev, rather than as an issue directly, is because I would like some direction/recommendation from concerned individuals before I take a "outsider" approach. Too often I have learned that I guessed wrong how the Python community "sees the world".
> On 2/21/2019 11:26 AM, Michael wrote:
> It seems so - however, Is there something such as PyUnsignedLong and is
> that large enough for a "long long"? and if it exists, would that make
> the value positive (for the first test).
> 
> posix.major and os.major will need to mask away the MSB and
> posix.makedev and os.makedev will need to add it back.
> 
> OR - do I need to make the PyStat values "the same" in both 32-bit and
> 64-bit?
I think I already have the answer to this - follow the platform (as far as size goes).

On 32-bit max(1) == 256 (2^^8) and on 64-bit 65536 (2^^16).

However, I am still puzzeled re: the idiosyncratic MSB addition - that the value for 
st.st_dev

on AIX 64-bit has always (as far back as I could look at least) had the MSB set so that, as a signed number st.st_dev is always negative. And MSB is not part of the major device number. It's just there.

So, it has been a few days since I studied this (waiting for reactions) - but is the return suppossed to always be a value returned by the posixmodule.c (if so, can strip the MSB bit and make everything positive - even for st.st_dev (but that would break under a direct comparision with a C program that does not strip the MSB).

Currently, the macros on AIX for 64-bit are broken, so I cannot guess how they will decide to implement major() and mkdevice() as far as striping (and adding) the MSB when "constructing" the st_dev number from two "regular" major and minor numbers.

Parked for now,

Best wishes,

Michael
> 
> Puzzled on what you think is the correct approach.
> 
> Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190301/36629631/attachment.html>

From willingc at gmail.com  Fri Mar  1 01:48:52 2019
From: willingc at gmail.com (Carol Willing)
Date: Thu, 28 Feb 2019 22:48:52 -0800
Subject: [Python-Dev] Update from the Python Steering Council about CPython
 Development
Message-ID: <2AACF155-36B1-4D80-A2C1-A948561C45E4@getmailspring.com>

The Python Steering Council is pleased to provide an update to the Python community about Steering Council activity and CPython development. We've created a GitHub repo for Steering Council updates and helpful documents: https://github.com/python/steering-council

Here's the latest update written after our meeting on February 26th:
https://github.com/python/steering-council/blob/master/updates/2019-02-26_steering-council-update.md (https://link.getmailspring.com/link/2AACF155-36B1-4D80-A2C1-A948561C45E4 at getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fpython%2Fsteering-council%2Fblob%2Fmaster%2Fupdates%2F2019-02-26_steering-council-update.md&recipient=cHl0aG9uLWRldkBweXRob24ub3Jn)
We'll be posting updates after each steering council meeting which will be roughly twice a month.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190228/db1930b4/attachment.html>

From status at bugs.python.org  Fri Mar  1 13:08:03 2019
From: status at bugs.python.org (Python tracker)
Date: Fri,  1 Mar 2019 18:08:03 +0000 (UTC)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190301180803.B9E9652B1B3@bugs.ams1.psf.io>


ACTIVITY SUMMARY (2019-02-22 - 2019-03-01)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    7012 (+13)
  closed 40898 (+64)
  total  47910 (+77)

Open issues with patches: 2791 


Issues opened (44)
==================

#35459: Use PyDict_GetItemWithError() instead of PyDict_GetItem()
https://bugs.python.org/issue35459  reopened by vstinner

#35931: pdb: "debug print(" crashes with SyntaxError
https://bugs.python.org/issue35931  reopened by blueyed

#36084: Threading: add builtin TID attribute to Thread objects
https://bugs.python.org/issue36084  opened by jaketesler

#36085: Enable better DLL resolution
https://bugs.python.org/issue36085  opened by steve.dower

#36086: Split IDLE into separate feature in Windows installer
https://bugs.python.org/issue36086  opened by kimsey0

#36091: clean up async generator from types module
https://bugs.python.org/issue36091  opened by scotchka

#36092: unittest.mock's patch.object and patch.dict are not supported 
https://bugs.python.org/issue36092  opened by xtreak

#36093: UnicodeEncodeError raise from smtplib.verify() method
https://bugs.python.org/issue36093  opened by Windson Yang

#36094: When using an SMTP SSL connection,, get ValueError.
https://bugs.python.org/issue36094  opened by tyrone-zhao

#36095: Better NaN sorting.
https://bugs.python.org/issue36095  opened by brandtbucher

#36097: Use only public C-API in _xxsubinterpreters module.
https://bugs.python.org/issue36097  opened by eric.snow

#36098: asyncio: ssl client-server with "slow" read
https://bugs.python.org/issue36098  opened by MultiSosnooley

#36099: Clarify the difference between mu and xbar in the statistics d
https://bugs.python.org/issue36099  opened by steven.daprano

#36100: Document the differences between str.isdigit, isdecimal and is
https://bugs.python.org/issue36100  opened by StyXman

#36103: Increase shutil.COPY_BUFSIZE
https://bugs.python.org/issue36103  opened by inada.naoki

#36107: aarch64 python3 buffer overflow with stack protector on rpi3 (
https://bugs.python.org/issue36107  opened by Natanael Copa

#36108: Avoid failing the build on race condition in clean
https://bugs.python.org/issue36108  opened by steve.dower

#36114: test_multiprocessing_spawn changes the execution environment
https://bugs.python.org/issue36114  opened by pablogsal

#36116: test_multiprocessing_spawn fails on AMD64 Windows8 3.x
https://bugs.python.org/issue36116  opened by pablogsal

#36121: csv: Non global alternative to csv.field_size_limit
https://bugs.python.org/issue36121  opened by Carlos Ramos

#36124: Provide convenient C API for storing per-interpreter state
https://bugs.python.org/issue36124  opened by ncoghlan

#36127: Argument Clinic: inline parsing code for functions with keywor
https://bugs.python.org/issue36127  opened by serhiy.storchaka

#36128: ResourceReader for FileLoader inconsistently handles path sepa
https://bugs.python.org/issue36128  opened by indygreg

#36129: io documentation unclear about flush() and close() semantics f
https://bugs.python.org/issue36129  opened by indygreg

#36130: Pdb(skip=[...]) + module without __name__ => TypeError
https://bugs.python.org/issue36130  opened by Anthony Sottile

#36132: Python cannot access hci_channel field in sockaddr_hci
https://bugs.python.org/issue36132  opened by bsder

#36133: ThreadPoolExecutor and ProcessPoolExecutor, dynamic worker cou
https://bugs.python.org/issue36133  opened by Fabian Dill

#36136: Windows: python._pth sets isolated mode late during Python ini
https://bugs.python.org/issue36136  opened by vstinner

#36137: SSL verification fails for some sites inside windows docker co
https://bugs.python.org/issue36137  opened by Mika Fischer

#36138: Improve documentation about converting datetime.timedelta to s
https://bugs.python.org/issue36138  opened by p-ganssle

#36139: release GIL on mmap dealloc
https://bugs.python.org/issue36139  opened by davide.rizzo

#36140: An incorrect check in _msi.c's msidb_getsummaryinformation()
https://bugs.python.org/issue36140  opened by ZackerySpytz

#36141: configure: error: could not find pthreads on your system durin
https://bugs.python.org/issue36141  opened by muhzi

#36142: Add a new _PyPreConfig step to Python initialization to setup 
https://bugs.python.org/issue36142  opened by vstinner

#36143: Auto-generate Lib/keyword.py
https://bugs.python.org/issue36143  opened by gvanrossum

#36144: Dictionary addition.
https://bugs.python.org/issue36144  opened by brandtbucher

#36145: android arm cross compilation fails, config issue
https://bugs.python.org/issue36145  opened by muhzi

#36147: [2.7] Coverity scan: Modules/_ctypes/cfield.c , Variable "resu
https://bugs.python.org/issue36147  opened by cstratak

#36149: use of uninitialised memory in cPickle.
https://bugs.python.org/issue36149  opened by twouters

#36153: Freeze support documentation is misleading.
https://bugs.python.org/issue36153  opened by Sridhar Iyer

#36154: Python quit unexpectedly error
https://bugs.python.org/issue36154  opened by kellena

#36157: Document PyInterpreterState_Main().
https://bugs.python.org/issue36157  opened by eric.snow

#36158: Regex search behaves differently in list comprehension
https://bugs.python.org/issue36158  opened by Matthew Drago

#36159: Modify Formatter Class to handle arbitrary objects
https://bugs.python.org/issue36159  opened by Ross Biro



Most recent 15 issues with no replies (15)
==========================================

#36159: Modify Formatter Class to handle arbitrary objects
https://bugs.python.org/issue36159

#36157: Document PyInterpreterState_Main().
https://bugs.python.org/issue36157

#36149: use of uninitialised memory in cPickle.
https://bugs.python.org/issue36149

#36140: An incorrect check in _msi.c's msidb_getsummaryinformation()
https://bugs.python.org/issue36140

#36138: Improve documentation about converting datetime.timedelta to s
https://bugs.python.org/issue36138

#36136: Windows: python._pth sets isolated mode late during Python ini
https://bugs.python.org/issue36136

#36133: ThreadPoolExecutor and ProcessPoolExecutor, dynamic worker cou
https://bugs.python.org/issue36133

#36130: Pdb(skip=[...]) + module without __name__ => TypeError
https://bugs.python.org/issue36130

#36108: Avoid failing the build on race condition in clean
https://bugs.python.org/issue36108

#36097: Use only public C-API in _xxsubinterpreters module.
https://bugs.python.org/issue36097

#36092: unittest.mock's patch.object and patch.dict are not supported 
https://bugs.python.org/issue36092

#36084: Threading: add builtin TID attribute to Thread objects
https://bugs.python.org/issue36084

#36081: Cannot set LDFLAGS containing $
https://bugs.python.org/issue36081

#36079: pdb on setuptools "ValueError: underlying buffer has been deta
https://bugs.python.org/issue36079

#36076: ssl.get_server_certificate should use SNI
https://bugs.python.org/issue36076



Most recent 15 issues waiting for review (15)
=============================================

#36149: use of uninitialised memory in cPickle.
https://bugs.python.org/issue36149

#36147: [2.7] Coverity scan: Modules/_ctypes/cfield.c , Variable "resu
https://bugs.python.org/issue36147

#36144: Dictionary addition.
https://bugs.python.org/issue36144

#36142: Add a new _PyPreConfig step to Python initialization to setup 
https://bugs.python.org/issue36142

#36140: An incorrect check in _msi.c's msidb_getsummaryinformation()
https://bugs.python.org/issue36140

#36139: release GIL on mmap dealloc
https://bugs.python.org/issue36139

#36130: Pdb(skip=[...]) + module without __name__ => TypeError
https://bugs.python.org/issue36130

#36127: Argument Clinic: inline parsing code for functions with keywor
https://bugs.python.org/issue36127

#36121: csv: Non global alternative to csv.field_size_limit
https://bugs.python.org/issue36121

#36103: Increase shutil.COPY_BUFSIZE
https://bugs.python.org/issue36103

#36097: Use only public C-API in _xxsubinterpreters module.
https://bugs.python.org/issue36097

#36095: Better NaN sorting.
https://bugs.python.org/issue36095

#36094: When using an SMTP SSL connection,, get ValueError.
https://bugs.python.org/issue36094

#36091: clean up async generator from types module
https://bugs.python.org/issue36091

#36086: Split IDLE into separate feature in Windows installer
https://bugs.python.org/issue36086



Top 10 most discussed issues (10)
=================================

#33944: Deprecate and remove pth files
https://bugs.python.org/issue33944  21 msgs

#36030: add internal API function to create tuple without items array 
https://bugs.python.org/issue36030  12 msgs

#36100: Document the differences between str.isdigit, isdecimal and is
https://bugs.python.org/issue36100  11 msgs

#36144: Dictionary addition.
https://bugs.python.org/issue36144  11 msgs

#36085: Enable better DLL resolution
https://bugs.python.org/issue36085  10 msgs

#36095: Better NaN sorting.
https://bugs.python.org/issue36095  10 msgs

#35975: Put back the ability to parse files where async/await aren't k
https://bugs.python.org/issue35975   9 msgs

#33608: [subinterpreters] Add a cross-interpreter-safe mechanism to in
https://bugs.python.org/issue33608   8 msgs

#35813: shared memory construct to avoid need for serialization betwee
https://bugs.python.org/issue35813   8 msgs

#36103: Increase shutil.COPY_BUFSIZE
https://bugs.python.org/issue36103   8 msgs



Issues closed (62)
==================

#8747: Autoconf tests in python not portably correct
https://bugs.python.org/issue8747  closed by ngie

#10308: Modules/getpath.c bugs
https://bugs.python.org/issue10308  closed by vstinner

#13497: Fix for broken nice test on non-broken platforms with pedantic
https://bugs.python.org/issue13497  closed by benjamin.peterson

#22123: Provide a direct function for types.SimpleNamespace()
https://bugs.python.org/issue22123  closed by rhettinger

#24235: ABCs don't fail metaclass instantiation
https://bugs.python.org/issue24235  closed by cheryl.sabella

#24310: Idle documentation -- what to do if you do not see an undersco
https://bugs.python.org/issue24310  closed by terry.reedy

#24643: VS 2015 pyconfig.h #define timezone _timezone conflicts with t
https://bugs.python.org/issue24643  closed by steve.dower

#24740: make patchcheck doesn't detect changes if commit is done first
https://bugs.python.org/issue24740  closed by terry.reedy

#27313: test case failures in test_widgets.ComboboxTest.of test_ttk_gu
https://bugs.python.org/issue27313  closed by ned.deily

#28450: Misleading/inaccurate documentation about unknown escape seque
https://bugs.python.org/issue28450  closed by serhiy.storchaka

#29397: linux/random.h present but cannot be compiled
https://bugs.python.org/issue29397  closed by christian.heimes

#29659: Expose the `length` arg from shutil.copyfileobj for public use
https://bugs.python.org/issue29659  closed by inada.naoki

#31652: make install fails: no module _ctypes
https://bugs.python.org/issue31652  closed by inada.naoki

#31916: ensurepip not honoring value of $(DESTDIR) - pip not installed
https://bugs.python.org/issue31916  closed by cheryl.sabella

#34211: Cygwin build broken due to use of &PyType_Type in static decla
https://bugs.python.org/issue34211  closed by inada.naoki

#34623: _elementtree.c doesn't call XML_SetHashSalt()
https://bugs.python.org/issue34623  closed by larry

#35178: Typo/trivial mistake in warnings.py (may be related to 2.x to 
https://bugs.python.org/issue35178  closed by vstinner

#35512: patch.dict resolves in_dict eagerly (should be late resolved)
https://bugs.python.org/issue35512  closed by xtreak

#35652: Add use_srcentry parameter to shutil.copytree() II
https://bugs.python.org/issue35652  closed by giampaolo.rodola

#35724: Check for main interpreter when checking for "main" thread (fo
https://bugs.python.org/issue35724  closed by eric.snow

#35802: os.stat / os.lstat always present, but code checks hastattr(os
https://bugs.python.org/issue35802  closed by Anthony Sottile

#35886: Move PyInterpreterState into Include/internal/pycore_pystate.h
https://bugs.python.org/issue35886  closed by eric.snow

#36016: Allow gc.getobjects to return the objects tracked by a specifi
https://bugs.python.org/issue36016  closed by inada.naoki

#36018: Add a Normal Distribution class to the statistics module
https://bugs.python.org/issue36018  closed by rhettinger

#36048: Deprecate implicit truncating when convert Python numbers to C
https://bugs.python.org/issue36048  closed by serhiy.storchaka

#36066: Add `empty` block to `for` and `while` loops.
https://bugs.python.org/issue36066  closed by terry.reedy

#36072: str.translate() behaves differently for ASCII-only and other s
https://bugs.python.org/issue36072  closed by terry.reedy

#36083: Misformated manpage: --check-hash-based-pycs ??default??|??alw
https://bugs.python.org/issue36083  closed by benjamin.peterson

#36087: ThreadPoolExecutor max_workers none issue
https://bugs.python.org/issue36087  closed by asvetlov

#36088: zipfile cannot handle zip in zip
https://bugs.python.org/issue36088  closed by serhiy.storchaka

#36089: Formatting/Spelling errors in SimpleHTTPServer docs
https://bugs.python.org/issue36089  closed by serhiy.storchaka

#36090: spelling error in PEP219 introduction
https://bugs.python.org/issue36090  closed by nanjekyejoannah

#36096: IDLE: Refactor class variables to instance variables in colori
https://bugs.python.org/issue36096  closed by terry.reedy

#36101: remove non-ascii characters in docstring
https://bugs.python.org/issue36101  closed by inada.naoki

#36102: TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x
https://bugs.python.org/issue36102  closed by davin

#36104: test_httplib and test_nntplib fail on ARMv7 Ubuntu
https://bugs.python.org/issue36104  closed by vstinner

#36105: Windows: use GetNativeSystemInfo instead of GetSystemInfo
https://bugs.python.org/issue36105  closed by steve.dower

#36106: resolve sinpi() name clash with libm
https://bugs.python.org/issue36106  closed by serhiy.storchaka

#36109: test_descr: test_vicious_descriptor_nonsense() fails randomly
https://bugs.python.org/issue36109  closed by pablogsal

#36110: test_descr: test_vicious_descriptor_nonsense() fails randomly
https://bugs.python.org/issue36110  closed by pablogsal

#36111: Non-zero `offset`s are no longer acceptable with SEEK_END/SEEK
https://bugs.python.org/issue36111  closed by inada.naoki

#36112: os.path.realpath on windows and substed drives
https://bugs.python.org/issue36112  closed by eryksun

#36113: Problem With SciPy Computation of sigma
https://bugs.python.org/issue36113  closed by berker.peksag

#36115: test_ctypes leaks references and memory blocks
https://bugs.python.org/issue36115  closed by miss-islington

#36117: Allow rich comparisons for real-valued complex objects.
https://bugs.python.org/issue36117  closed by serhiy.storchaka

#36118: Cannot correctly concatenate nested list that contains more th
https://bugs.python.org/issue36118  closed by josh.r

#36119: Can't add/append in set/list inside shared dict
https://bugs.python.org/issue36119  closed by josh.r

#36120: Regression - Concurrent Futures
https://bugs.python.org/issue36120  closed by cheryl.sabella

#36122: Second run of 2to3 continues to modify output
https://bugs.python.org/issue36122  closed by bers

#36123: Race condition in test_socket
https://bugs.python.org/issue36123  closed by nanjekyejoannah

#36125: Cannot cross-compile to more featureful but same tune
https://bugs.python.org/issue36125  closed by ned.deily

#36126: Reference count leakage in structseq_repr
https://bugs.python.org/issue36126  closed by serhiy.storchaka

#36131: test.test_urllib2net.TimeoutTest ftp related tests fail due to
https://bugs.python.org/issue36131  closed by vstinner

#36134: test failure : test_re; recipe for target 'test' failed
https://bugs.python.org/issue36134  closed by serhiy.storchaka

#36135: altinstall error Makefile:1140: recipe for target 'altinstall'
https://bugs.python.org/issue36135  closed by matrixise

#36146: Refactor setup.py
https://bugs.python.org/issue36146  closed by vstinner

#36148: smtplib.SMTP.sendmail: mta status codes only accessible by loc
https://bugs.python.org/issue36148  closed by r.david.murray

#36150: Possible assertion failures due to _ctypes.c's PyCData_reduce(
https://bugs.python.org/issue36150  closed by christian.heimes

#36151: Incorrect answer when calculating 11/3
https://bugs.python.org/issue36151  closed by zach.ware

#36152: IDLE: Remove close_when_done from colorizer close()
https://bugs.python.org/issue36152  closed by cheryl.sabella

#36155: ./python -m test -m test_gc fails
https://bugs.python.org/issue36155  closed by pablogsal

#36156: different method, but id function return same value.
https://bugs.python.org/issue36156  closed by inada.naoki

From barry at python.org  Fri Mar  1 18:25:29 2019
From: barry at python.org (Barry Warsaw)
Date: Fri, 1 Mar 2019 15:25:29 -0800
Subject: [Python-Dev] Compile-time resolution of packages [Was: Another
 update for PEP 394...]
In-Reply-To: <37ff84c0-7b7d-485f-e061-9f9b8f96fdb4@gmail.com>
References: <9e69c6dc-07cd-2265-b4b8-b9f7e9f81b00@gmail.com>
 <a096f664-3000-272a-ae5f-7713c936aa2c@gmail.com>
 <37ba6931-faa0-0c9c-b9e5-067eb123e313@gmail.com>
 <58F34E40-11B8-4F36-AF7E-C9022D4F48DF@python.org>
 <CAGE7PNK0o1tG84q06VUttGOBNkd2g3ftmsUA1U3NscjcGFGRQg@mail.gmail.com>
 <20190226220418.b36jw33qthdv5i5l@python.ca>
 <CABVPEKqPoOycqrwVjfCZo3J+4bgF-o1i3WB7nM5P+a3zZ0UkbQ@mail.gmail.com>
 <CAGE7PNL3L_jrxMk0CKr2b0RQX6R3u9N7XyO+CeC8w2Y0cTuoxg@mail.gmail.com>
 <37ff84c0-7b7d-485f-e061-9f9b8f96fdb4@gmail.com>
Message-ID: <CB9C97A6-FD4F-43BA-89F1-137538E9A9E7@python.org>

On Mar 1, 2019, at 02:41, Petr Viktorin <encukou at gmail.com> wrote:

> Currently, in RHEL/Fedora:
> - "sudo pip" installs to /usr/local/, RPM packages install to /usr/
> - with "-I", the interpreter does not look into /usr/local/.
> AFAIK, Debian/Ubuntu have something very similar, and were first to do it.

Debuntu?s pip installs to the user path by default, and you have to do something explicit to install into the system Python.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190301/484e048e/attachment.sig>

From larry at hastings.org  Sat Mar  2 01:05:16 2019
From: larry at hastings.org (Larry Hastings)
Date: Fri, 1 Mar 2019 22:05:16 -0800
Subject: [Python-Dev] Last-minute request: please backport bpo-33329 fix to
 3.4 and 3.5
Message-ID: <eca9d6f2-14a3-0a17-2730-8d74cbb8fa84@hastings.org>


This bug in bpo-33329:

    https://bugs.python.org/issue33329

was fixed for 3.6+, but it also affects 3.4 and 3.5.? The bug is that 
with newer versions of glibc--which I'm pretty sure has shipped on all 
major Linux distros by now--the test suite may send signals that are 
invalid somehow.? As a result the test suite... blocks forever?? I 
think?? Anyway the observed resulting behavior is that there are three 
regression tests in each branch that seemingly never complete.? I 
started the 3.4 regression test suite /nine hours ago/ and it still 
claims to be running--and the 3.5 test suite isn't far behind.? 
Technically, no, it's not a security bug.? But I simply can't ship 3.4 
and 3.5 in this sorry state.

Obviously it'd be best if the folks involved with the original PRs 
(Antoine?) took over.? I'm sending this to a wider audience just because 
I'd hoped to tag the next RCs for 3.4 and 3.5 this weekend, and the 
original participants in this fix may not be available, and I'm hoping I 
won't have to slip the schedule.


Thanks for your time,


//arry/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190301/7c584aca/attachment.html>

From benjamin at python.org  Sun Mar  3 22:30:52 2019
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 03 Mar 2019 22:30:52 -0500
Subject: [Python-Dev] [RELEASE] Python 2.7.16
Message-ID: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>

Hello all,
I'm pleased to announce the immediate availability of Python 2.7.16 for download at https://www.python.org/downloads/release/python-2716/.

The only change since the release candidate was a fix for the IDLE icon on macOS. See https://bugs.python.org/issue32129. Refer to the changelog for a full list of changes: https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16rc1.rst

Please report any bugs to https://bugs.python.org/.

Regards,
Benjamin
2.7 release manager
(on behalf of all Python 2.7's contributors)

From larry at hastings.org  Mon Mar  4 04:23:33 2019
From: larry at hastings.org (Larry Hastings)
Date: Mon, 4 Mar 2019 01:23:33 -0800
Subject: [Python-Dev] [RELEASED] Python 3.4.10rc1 and Python 3.5.7rc1 are
 now available
Message-ID: <a3d8c67e-584c-db8e-134f-66cf922f73ea@hastings.org>


On behalf of the Python development community, I'm chuffed to announce 
the availability of Python 3.4.10rc1 and Python 3.5.7rc1.

Both Python 3.4 and 3.5 are in "security fixes only" mode.? Both 
versions only accept security fixes, not conventional bug fixes, and 
both releases are source-only.

The "final" releases on both these branches should be out in about two 
weeks.? Of particular note: that release of Python 3.4, Python 3.4.10 
final, will be the final release ever in the Python 3.4 series.? After 
3.4.10, the branch will be closed for good and I'll retire as Python 3.4 
Release Manager.? I'll still be the Python 3.5 Release Manager until 3.5 
similarly concludes, approximately eighteen months from now.

You can find Python 3.4.10rc1 here:

    https://www.python.org/downloads/release/python-3410rc1/


And you can find Python 3.5.7rc1 here:

    https://www.python.org/downloads/release/python-357rc1/


Best wishes,



//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190304/491ea9e7/attachment.html>

From j.orponen at 4teamwork.ch  Mon Mar  4 05:29:43 2019
From: j.orponen at 4teamwork.ch (Joni Orponen)
Date: Mon, 4 Mar 2019 11:29:43 +0100
Subject: [Python-Dev] Last-minute request: please backport bpo-33329 fix
 to 3.4 and 3.5
In-Reply-To: <eca9d6f2-14a3-0a17-2730-8d74cbb8fa84@hastings.org>
References: <eca9d6f2-14a3-0a17-2730-8d74cbb8fa84@hastings.org>
Message-ID: <CA+LW3i3jX06H5XbPSnFxcLmL+fQX9Z9yFPMUjhrceR6cKcceSw@mail.gmail.com>

On Sat, Mar 2, 2019 at 7:08 AM Larry Hastings <larry at hastings.org> wrote:

>
> This bug in bpo-33329:
>
> https://bugs.python.org/issue33329
>
> was fixed for 3.6+, but it also affects 3.4 and 3.5.  The bug is that with
> newer versions of glibc--which I'm pretty sure has shipped on all major
> Linux distros by now--the test suite may send signals that are invalid
> somehow.  As a result the test suite... blocks forever?  I think?  Anyway
> the observed resulting behavior is that there are three regression tests in
> each branch that seemingly never complete.  I started the 3.4 regression
> test suite *nine hours ago* and it still claims to be running--and the
> 3.5 test suite isn't far behind.  Technically, no, it's not a security
> bug.  But I simply can't ship 3.4 and 3.5 in this sorry state.
>

This is also potentially affecting PGO builds of 2.7 on Debian Buster with
GCC. Somehow building with Clang is fine.

Does the configure time choice of compiler make a difference here for 3.4
and 3.5?

-- 
Joni Orponen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190304/bb923943/attachment.html>

From aixtools at felt.demon.nl  Mon Mar  4 06:46:35 2019
From: aixtools at felt.demon.nl (Michael)
Date: Mon, 4 Mar 2019 12:46:35 +0100
Subject: [Python-Dev] [RELEASE] Python 2.7.16
In-Reply-To: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>
References: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>
Message-ID: <6010f9bf-5a18-43fd-dd04-1b28cbe39d9e@felt.demon.nl>

On 04/03/2019 04:30, Benjamin Peterson wrote:
> Hello all,
> I'm pleased to announce the immediate availability of Python 2.7.16 for download at https://www.python.org/downloads/release/python-2716/.

Congratulations!


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190304/1d24cba5/attachment.sig>

From vstinner at redhat.com  Mon Mar  4 10:37:00 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 4 Mar 2019 16:37:00 +0100
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.10rc1
 and Python 3.5.7rc1 are now available
In-Reply-To: <a3d8c67e-584c-db8e-134f-66cf922f73ea@hastings.org>
References: <a3d8c67e-584c-db8e-134f-66cf922f73ea@hastings.org>
Message-ID: <CA+3bQGGFsfQxezqrWwwPwXHWooXXWxkzcehTv4td7-5_-tkKfA@mail.gmail.com>

FYI I check and I confirm that all known security vulnerabilities
listed in the link below are fixed in these releases:
https://python-security.readthedocs.io/vulnerabilities.html

Victor

Le lun. 4 mars 2019 ? 10:24, Larry Hastings <larry at hastings.org> a ?crit :
>
>
> On behalf of the Python development community, I'm chuffed to announce the availability of Python 3.4.10rc1 and Python 3.5.7rc1.
>
> Both Python 3.4 and 3.5 are in "security fixes only" mode.  Both versions only accept security fixes, not conventional bug fixes, and both releases are source-only.
>
> The "final" releases on both these branches should be out in about two weeks.  Of particular note: that release of Python 3.4, Python 3.4.10 final, will be the final release ever in the Python 3.4 series.  After 3.4.10, the branch will be closed for good and I'll retire as Python 3.4 Release Manager.  I'll still be the Python 3.5 Release Manager until 3.5 similarly concludes, approximately eighteen months from now.
>
> You can find Python 3.4.10rc1 here:
>
> https://www.python.org/downloads/release/python-3410rc1/
>
>
> And you can find Python 3.5.7rc1 here:
>
> https://www.python.org/downloads/release/python-357rc1/
>
>
> Best wishes,
>
>
>
> /arry
> _______________________________________________
> python-committers mailing list
> python-committers at python.org
> https://mail.python.org/mailman/listinfo/python-committers
> Code of Conduct: https://www.python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From storchaka at gmail.com  Mon Mar  4 12:37:17 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 4 Mar 2019 19:37:17 +0200
Subject: [Python-Dev] Addendum to PEP 530
Message-ID: <q5jnog$3trj$1@blaine.gmane.org>

PEP 530 introduced support for asynchronous comprehensions.

Comprehensions are implemented as local functions. To make a function 
asynchronous you should to add "async" before "def", and after that you 
can use "async for", "async with" and "await" in a function. But you can 
to do this with comprehensions, since these functions are implicit.

PEP 530 allows to make these function asynchronous by just using 
asynchronous syntax inside. "async with" can not be used in 
comprehensions, but using "async for" or "await" makes the comprehension 
asynchronous.

What PEP 530 missed is that asynchronous comprehensions itself are 
asynchronous constructions which can be used only in asynchronous functions.

Issue33346 [1] makes an asynchronous comprehension inside a 
comprehension making the outer comprehension asynchronous as well as 
using explicit "async for" or "await". See details on the issue.

Yury, the author of PEP 530, likes this idea, and there is a ready 
implementation. But making the decision was deferred until a new 
government be stated. Now, after the Steering Council has been elected, 
can it make the decision?

[1] https://bugs.python.org/issue33346


From brett at python.org  Mon Mar  4 15:06:55 2019
From: brett at python.org (Brett Cannon)
Date: Mon, 4 Mar 2019 12:06:55 -0800
Subject: [Python-Dev] Addendum to PEP 530
In-Reply-To: <q5jnog$3trj$1@blaine.gmane.org>
References: <q5jnog$3trj$1@blaine.gmane.org>
Message-ID: <CAP1=2W5CNF4GJ=b=cKc2XO3jqhrAf2Rh47h3FJX56RU3zbvtVg@mail.gmail.com>

Open an issue at https://github.com/python/steering-council/issues if you
want the steering council to consider something.

On Mon, Mar 4, 2019 at 9:38 AM Serhiy Storchaka <storchaka at gmail.com> wrote:

> PEP 530 introduced support for asynchronous comprehensions.
>
> Comprehensions are implemented as local functions. To make a function
> asynchronous you should to add "async" before "def", and after that you
> can use "async for", "async with" and "await" in a function. But you can
> to do this with comprehensions, since these functions are implicit.
>
> PEP 530 allows to make these function asynchronous by just using
> asynchronous syntax inside. "async with" can not be used in
> comprehensions, but using "async for" or "await" makes the comprehension
> asynchronous.
>
> What PEP 530 missed is that asynchronous comprehensions itself are
> asynchronous constructions which can be used only in asynchronous
> functions.
>
> Issue33346 [1] makes an asynchronous comprehension inside a
> comprehension making the outer comprehension asynchronous as well as
> using explicit "async for" or "await". See details on the issue.
>
> Yury, the author of PEP 530, likes this idea, and there is a ready
> implementation. But making the decision was deferred until a new
> government be stated. Now, after the Steering Council has been elected,
> can it make the decision?
>
> [1] https://bugs.python.org/issue33346
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190304/844a49e9/attachment.html>

From tjreedy at udel.edu  Mon Mar  4 19:31:50 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 4 Mar 2019 19:31:50 -0500
Subject: [Python-Dev] [RELEASE] Python 2.7.16
In-Reply-To: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>
References: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>
Message-ID: <q5kg1n$3cnc$1@blaine.gmane.org>

On 3/3/2019 10:30 PM, Benjamin Peterson wrote:

> I'm pleased to announce the immediate availability of Python 2.7.16 for download at https://www.python.org/downloads/release/python-2716/.

On Windows 10, this is an 'unrecognized app' and Windows Defender 
SmartScreen, now default, refuses to run it. "Windows protected your PC" 
until one clicks 'more info' to get 'Run anyway'.  This is new since the 
.rc1 release.  We should either make 2.7 a 'known' app* or say something 
on the download page about clicking 'more info'.  I don't know the 
status of python.org 3.x downloads.

Since Steve Dower put 3.7 on the Windows store, PSF must now be a known 
publisher.  Perhaps he can help make 2.7 'known'.

-- 
Terry Jan Reedy


From brett at snarky.ca  Mon Mar  4 20:43:59 2019
From: brett at snarky.ca (Brett Cannon)
Date: Mon, 4 Mar 2019 17:43:59 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
Message-ID: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>

The steering council has implemented a new idea called sponsors to the PEP
process (added in
https://github.com/python/peps/commit/c58d32c33bd06eb386d3f33963a1434510528f68).
The thinking is that to help make sure PEPs from non-core developers
receive appropriate guidance through the PEP process, a core developer
needs to sign on to be a sponsor of the PEP. Being a sponsor does *not*
preclude the core dev from eventually becoming a co-author or BDFL-delegate
later on (but obviously not both), but the expectation is the sponsor is
supportive of the idea (because if a single core dev won't sign on to help
then what chance does the PEP have of being accepted?).

If this doesn't turn out well we can obviously revert this, but hopefully
this will make things smoother for those who are new to the PEP process.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190304/166b6a50/attachment.html>

From hasan.diwan at gmail.com  Mon Mar  4 23:57:46 2019
From: hasan.diwan at gmail.com (Hasan Diwan)
Date: Mon, 4 Mar 2019 20:57:46 -0800
Subject: [Python-Dev] [RELEASE] Python 2.7.16
In-Reply-To: <q5kg1n$3cnc$1@blaine.gmane.org>
References: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>
 <q5kg1n$3cnc$1@blaine.gmane.org>
Message-ID: <CAP+bYWCnJirLfv-2VUXt5Kwo_O67zBUus50vRqBzj_hNCT0QsQ@mail.gmail.com>

On Mon, 4 Mar 2019 at 16:33, Terry Reedy <tjreedy at udel.edu> wrote:

> On 3/3/2019 10:30 PM, Benjamin Peterson wrote:
>
> > I'm pleased to announce the immediate availability of Python 2.7.16 for
> download at https://www.python.org/downloads/release/python-2716/.
>

Congrats team! -- H
-- 
OpenPGP:
https://sks-keyservers.net/pks/lookup?op=get&search=0xFEBAD7FFD041BBA1
If you wish to request my time, please do so using
*bit.ly/hd1AppointmentRequest
<http://bit.ly/hd1AppointmentRequest>*.
Si vous voudrais faire connnaisance, allez a *bit.ly/hd1AppointmentRequest
<http://bit.ly/hd1AppointmentRequest>*.

<https://sks-keyservers.net/pks/lookup?op=get&search=0xFEBAD7FFD041BBA1>Sent
from my mobile device
Envoye de mon portable
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190304/f004d6d9/attachment.html>

From J.Demeyer at UGent.be  Tue Mar  5 01:11:57 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Tue, 5 Mar 2019 07:11:57 +0100
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
Message-ID: <5C7E132D.20004@UGent.be>

Does this apply to existing draft PEPs or only new ones?

From mhroncok at redhat.com  Tue Mar  5 06:17:48 2019
From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=)
Date: Tue, 5 Mar 2019 12:17:48 +0100
Subject: [Python-Dev] [RELEASE] Python 2.7.16
In-Reply-To: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>
References: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>
Message-ID: <12acd71f-c18b-ea72-1e43-df4c76aa7812@redhat.com>

On 04. 03. 19 4:30, Benjamin Peterson wrote:
> Hello all,
> I'm pleased to announce the immediate availability of Python 2.7.16 for download at https://www.python.org/downloads/release/python-2716/.
> 
> The only change since the release candidate was a fix for the IDLE icon on macOS. See https://bugs.python.org/issue32129. Refer to the changelog for a full list of changes: https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16rc1.rst


https://www.python.org/downloads/release/python-2716/ links changelog to 
https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16.rst 
but that only has 1 change (I suppose against rc1).

Is there a better link, or should I read those two combined?

https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16rc1.rst
https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16.rst

-- 
Miro Hron?ok
--
Phone: +420777974800
IRC: mhroncok

From vstinner at redhat.com  Tue Mar  5 06:35:59 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 5 Mar 2019 12:35:59 +0100
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
Message-ID: <CA+3bQGFhgPymJSrsAqAkkeLv0kDWFQuhiVushuz__kD4jQp5wQ@mail.gmail.com>

Hi,

Le mar. 5 mars 2019 ? 02:53, Brett Cannon <brett at snarky.ca> a ?crit :
> The steering council has implemented a new idea called sponsors to the PEP process (...). The thinking is that to help make sure PEPs from non-core developers receive appropriate guidance through the PEP process (...)

Hum, this isn't fully new, some PEPs already got a "PEP champion" (old
name), no?

A recent example is PEP 572 who has been "championed" by Guido van
Rossum, then by Tim Peters. In this specific case, they became
co-authors :-)

> ... eventually becoming a co-author or BDFL-delegate later

Nitpick: since Python has no more BDFL, maybe the expression should
becocme "PEP-delegate"? ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From cspealma at redhat.com  Tue Mar  5 08:05:31 2019
From: cspealma at redhat.com (Calvin Spealman)
Date: Tue, 5 Mar 2019 08:05:31 -0500
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
Message-ID: <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>

> because if a single core dev won't sign on to help then what chance does
the PEP have of being accepted?

If this is the justification, then I feel like it is a barrier that could
be disassembled, rather than one people need to be granted permission to
climb over.

I am named on one PEP with two others (PEP 3135) and none of us were core
developers, I think? I don't want that barrier added to newcomers, and I
don't disagree there is a pre-existing barrier, but I can't help but feel
this is not the best solution to it.

I'm worried this creates a gatekeeping perception that will scare away
contributors.

On Mon, Mar 4, 2019 at 8:53 PM Brett Cannon <brett at snarky.ca> wrote:

> The steering council has implemented a new idea called sponsors to the PEP
> process (added in
> https://github.com/python/peps/commit/c58d32c33bd06eb386d3f33963a1434510528f68).
> The thinking is that to help make sure PEPs from non-core developers
> receive appropriate guidance through the PEP process, a core developer
> needs to sign on to be a sponsor of the PEP. Being a sponsor does *not*
> preclude the core dev from eventually becoming a co-author or BDFL-delegate
> later on (but obviously not both), but the expectation is the sponsor is
> supportive of the idea (because if a single core dev won't sign on to help
> then what chance does the PEP have of being accepted?).
>
> If this doesn't turn out well we can obviously revert this, but hopefully
> this will make things smoother for those who are new to the PEP process.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/cspealma%40redhat.com
>


-- 

CALVIN SPEALMAN

SENIOR QUALITY ENGINEER

cspealma at redhat.com  M: +1.336.210.5107
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190305/1ba078ba/attachment.html>

From J.Demeyer at UGent.be  Tue Mar  5 08:24:43 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Tue, 5 Mar 2019 14:24:43 +0100
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
Message-ID: <5C7E789B.9010007@UGent.be>

On 2019-03-05 14:05, Calvin Spealman wrote:
> I'm worried this creates a gatekeeping perception that will scare away
> contributors.

+1

I also expressed this worry at https://github.com/python/peps/pull/903

You could keep the positive part of the sponsoring idea (somebody acting 
as mentor) but drop the negative part (make it a hard requirement to 
find a sponsor supporting the proposal before the proposal can even 
become a draft PEP).

From jheiv at jheiv.com  Tue Mar  5 09:20:15 2019
From: jheiv at jheiv.com (James Edwards)
Date: Tue, 5 Mar 2019 09:20:15 -0500
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <5C7E789B.9010007@UGent.be>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
 <5C7E789B.9010007@UGent.be>
Message-ID: <CANbgQ_GNqk19iOpotgGT_mNFUFNBEHoNoEm9tVpV1rZOnLw1kQ@mail.gmail.com>

I have to say, this is sort of surprising for what seems like the first
official action of the steering committee.  Are there really *that many *PEPs
that a team that is now, what, 5x the size of the BFDL model is worried
that they'll be able to keep up?  As a long-time lurker, this hardly seems
to be the case.  Despite the seemingly-well-intentioned rationale, this
seems like an ominous sign.

On Tue, Mar 5, 2019 at 8:33 AM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> On 2019-03-05 14:05, Calvin Spealman wrote:
> > I'm worried this creates a gatekeeping perception that will scare away
> > contributors.
>
> +1
>
> I also expressed this worry at https://github.com/python/peps/pull/903
>
> You could keep the positive part of the sponsoring idea (somebody acting
> as mentor) but drop the negative part (make it a hard requirement to
> find a sponsor supporting the proposal before the proposal can even
> become a draft PEP).
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jheiv%40jheiv.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190305/0eb0f80b/attachment.html>

From status at bugs.python.org  Tue Mar  5 11:25:07 2019
From: status at bugs.python.org (Python tracker)
Date: Tue,  5 Mar 2019 16:25:07 +0000 (UTC)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190305162507.B883A52B1CF@bugs.ams1.psf.io>


ACTIVITY SUMMARY (2019-02-08 - 2019-02-15)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    7009 (+11)
  closed 40747 (+51)
  total  47756 (+62)

Open issues with patches: 2814 


Issues opened (42)
==================

#25737: array is not a Sequence
https://bugs.python.org/issue25737  reopened by josh.r

#35944: Python 3.7 install error
https://bugs.python.org/issue35944  opened by lasonjack

#35945: Cannot distinguish between subtask cancellation and running ta
https://bugs.python.org/issue35945  opened by jnwatson

#35946: Ambiguous documentation for assert_called_with()
https://bugs.python.org/issue35946  opened by chimaerase

#35947: Update libffi_msvc to current version of libffi
https://bugs.python.org/issue35947  opened by Paul Monson

#35949: Move PyThreadState into Include/internal/pycore_pystate.h
https://bugs.python.org/issue35949  opened by eric.snow

#35950: io.BufferedReader.writabe is False, but io.BufferedReader.trun
https://bugs.python.org/issue35950  opened by steverpalmer

#35951: os.renames() creates directories if original name doesn't exis
https://bugs.python.org/issue35951  opened by chris.jerdonek

#35952: test.pythoninfo prints a stack trace and exits with 1 when the
https://bugs.python.org/issue35952  opened by xdegaye

#35953: crosscompilation fails with clang on android
https://bugs.python.org/issue35953  opened by muhzi

#35954: Incoherent type conversion in configparser
https://bugs.python.org/issue35954  opened by Adeokkuw

#35955: difflib reports incorrect location of mismatch
https://bugs.python.org/issue35955  opened by jaraco

#35956: Sort documentation could be improved for complex sorting
https://bugs.python.org/issue35956  opened by FabriceSalvaire

#35957: Indentation explanation is unclear
https://bugs.python.org/issue35957  opened by J??r??me LAURENS

#35959: math.prod(range(10)) caues segfault
https://bugs.python.org/issue35959  opened by xtreak

#35962: Slight error in words in [ 2.4.1. String and Bytes literals ]
https://bugs.python.org/issue35962  opened by Magnien Sebastien

#35964: shutil.make_archive (xxx, tar, root_dir) is adding './' entry 
https://bugs.python.org/issue35964  opened by HFM

#35967: Better platform.processor support
https://bugs.python.org/issue35967  opened by jaraco

#35969: Interpreter crashes with "can't initialize init_sys_streams" w
https://bugs.python.org/issue35969  opened by p-ganssle

#35970: no help flag in base64 util
https://bugs.python.org/issue35970  opened by rkuska

#35971: Documentation should warn about code injection from current wo
https://bugs.python.org/issue35971  opened by Gabriel Corona

#35974: os.DirEntry.inode() returns invalid value within Docker contai
https://bugs.python.org/issue35974  opened by decaz

#35975: Put back the ability to parse files where async/await aren't k
https://bugs.python.org/issue35975  opened by gvanrossum

#35978: test_venv fails in Travis with GCC
https://bugs.python.org/issue35978  opened by xtreak

#35981: shutil make_archive create wrong file when base name contains 
https://bugs.python.org/issue35981  opened by highwind

#35982: Create unit-tests for os.renames()
https://bugs.python.org/issue35982  opened by nanjekyejoannah

#35983: tp_dealloc trashcan shouldn't be called for subclasses
https://bugs.python.org/issue35983  opened by jdemeyer

#35984: test__xxsubinterpreters leaked [3, 4, 3] memory blocks, sum=1
https://bugs.python.org/issue35984  opened by pablogsal

#35989: ipaddress.IPv4Network allows prefix > 32
https://bugs.python.org/issue35989  opened by John Florian

#35990: ipaddress.IPv4Interface won't accept 2-tuple (address, mask)
https://bugs.python.org/issue35990  opened by John Florian

#35992: Metaclasses interfere with __class_getitem__
https://bugs.python.org/issue35992  opened by donovick

#35993: incorrect use of released memory in Python/pystate.c line 284
https://bugs.python.org/issue35993  opened by wjq-security

#35995: logging.handlers.SMTPHandler
https://bugs.python.org/issue35995  opened by lidayan

#35996: Optional modulus argument for new math.prod() function
https://bugs.python.org/issue35996  opened by lschoe

#35997: ImportError: dlopen failed: cannot locate symbol "PyBool_Type"
https://bugs.python.org/issue35997  opened by muhzi

#35998: test_asyncio: test_start_tls_server_1() TimeoutError on Fedora
https://bugs.python.org/issue35998  opened by matrixise

#35999: multpirocessing.Process alive after SIGTERM on parent
https://bugs.python.org/issue35999  opened by lids

#36000: __debug__ is a keyword but not a keyword
https://bugs.python.org/issue36000  opened by bup

#36001: LIBFFI_INCLUDEDIR is not detected when set into a profile nor 
https://bugs.python.org/issue36001  opened by neil pop

#36002: configure --enable-optimizations with clang fails to detect ll
https://bugs.python.org/issue36002  opened by mjpieters

#36003: set better defaults for TCPServer options
https://bugs.python.org/issue36003  opened by giampaolo.rodola

#36004: Add datetime.fromisocalendar
https://bugs.python.org/issue36004  opened by p-ganssle



Most recent 15 issues with no replies (15)
==========================================

#36002: configure --enable-optimizations with clang fails to detect ll
https://bugs.python.org/issue36002

#36001: LIBFFI_INCLUDEDIR is not detected when set into a profile nor 
https://bugs.python.org/issue36001

#35999: multpirocessing.Process alive after SIGTERM on parent
https://bugs.python.org/issue35999

#35990: ipaddress.IPv4Interface won't accept 2-tuple (address, mask)
https://bugs.python.org/issue35990

#35974: os.DirEntry.inode() returns invalid value within Docker contai
https://bugs.python.org/issue35974

#35970: no help flag in base64 util
https://bugs.python.org/issue35970

#35964: shutil.make_archive (xxx, tar, root_dir) is adding './' entry 
https://bugs.python.org/issue35964

#35952: test.pythoninfo prints a stack trace and exits with 1 when the
https://bugs.python.org/issue35952

#35950: io.BufferedReader.writabe is False, but io.BufferedReader.trun
https://bugs.python.org/issue35950

#35947: Update libffi_msvc to current version of libffi
https://bugs.python.org/issue35947

#35939: Remove urllib.parse._splittype from mimetypes.guess_type
https://bugs.python.org/issue35939

#35930: Raising an exception raised in a "future" instance will create
https://bugs.python.org/issue35930

#35926: Need openssl 1.1.1 support on Windows for ARM and ARM64
https://bugs.python.org/issue35926

#35924: curses segfault resizing window
https://bugs.python.org/issue35924

#35919: multiprocessing: shared manager Pool fails with AttributeError
https://bugs.python.org/issue35919



Most recent 15 issues waiting for review (15)
=============================================

#36004: Add datetime.fromisocalendar
https://bugs.python.org/issue36004

#36003: set better defaults for TCPServer options
https://bugs.python.org/issue36003

#35989: ipaddress.IPv4Network allows prefix > 32
https://bugs.python.org/issue35989

#35983: tp_dealloc trashcan shouldn't be called for subclasses
https://bugs.python.org/issue35983

#35982: Create unit-tests for os.renames()
https://bugs.python.org/issue35982

#35975: Put back the ability to parse files where async/await aren't k
https://bugs.python.org/issue35975

#35969: Interpreter crashes with "can't initialize init_sys_streams" w
https://bugs.python.org/issue35969

#35959: math.prod(range(10)) caues segfault
https://bugs.python.org/issue35959

#35954: Incoherent type conversion in configparser
https://bugs.python.org/issue35954

#35951: os.renames() creates directories if original name doesn't exis
https://bugs.python.org/issue35951

#35947: Update libffi_msvc to current version of libffi
https://bugs.python.org/issue35947

#35946: Ambiguous documentation for assert_called_with()
https://bugs.python.org/issue35946

#35941: ssl.enum_certificates() regression
https://bugs.python.org/issue35941

#35936: Give modulefinder some much-needed updates.
https://bugs.python.org/issue35936

#35933: python doc does not say that the state kwarg in Pickler.save_r
https://bugs.python.org/issue35933



Top 10 most discussed issues (10)
=================================

#35969: Interpreter crashes with "can't initialize init_sys_streams" w
https://bugs.python.org/issue35969  14 msgs

#35955: difflib reports incorrect location of mismatch
https://bugs.python.org/issue35955  12 msgs

#35813: shared memory construct to avoid need for serialization betwee
https://bugs.python.org/issue35813  10 msgs

#35959: math.prod(range(10)) caues segfault
https://bugs.python.org/issue35959   9 msgs

#35995: logging.handlers.SMTPHandler
https://bugs.python.org/issue35995   9 msgs

#35155: Clarify Protocol Handlers in urllib.request Docs
https://bugs.python.org/issue35155   8 msgs

#35810: Object Initialization does not incref Heap-allocated Types
https://bugs.python.org/issue35810   8 msgs

#35982: Create unit-tests for os.renames()
https://bugs.python.org/issue35982   8 msgs

#35984: test__xxsubinterpreters leaked [3, 4, 3] memory blocks, sum=1
https://bugs.python.org/issue35984   8 msgs

#1054041: Python doesn't exit with proper resultcode on SIGINT
https://bugs.python.org/issue1054041   8 msgs



Issues closed (51)
==================

#7850: platform.system() should be "macosx" instead of "Darwin" on OS
https://bugs.python.org/issue7850  closed by ronaldoussoren

#12374: Execution model should explain compile vs definition vs execut
https://bugs.python.org/issue12374  closed by ncoghlan

#21107: Add pgen.vcxproj to allow regenerating grammar files on Window
https://bugs.python.org/issue21107  closed by zach.ware

#22062: Fix pathlib.Path.(r)glob doc glitches.
https://bugs.python.org/issue22062  closed by petr.viktorin

#28627: [alpine] shutil.copytree fail to copy a direcotry with broken 
https://bugs.python.org/issue28627  closed by benjamin.peterson

#28673: pyro4 with more than 15 threads often crashes 2.7.12
https://bugs.python.org/issue28673  closed by inada.naoki

#30410: Documentation for sys.stdout encoding does not reflect the new
https://bugs.python.org/issue30410  closed by Mariatta

#31940: copystat on symlinks fails for alpine -- faulty lchmod impleme
https://bugs.python.org/issue31940  closed by benjamin.peterson

#32417: fromutc does not respect datetime subclasses
https://bugs.python.org/issue32417  closed by p-ganssle

#34433: cancel all other pending child futures
https://bugs.python.org/issue34433  closed by yselivanov

#35055: Error when we try to download the epub archive
https://bugs.python.org/issue35055  closed by mdk

#35321: None _frozen_importlib.__spec__.origin attribute
https://bugs.python.org/issue35321  closed by maggyero

#35364: Datetime ???fromtimestamp()??? ignores inheritance if timezone
https://bugs.python.org/issue35364  closed by p-ganssle

#35378: multiprocessing.Pool.imaps iterators do not maintain alive the
https://bugs.python.org/issue35378  closed by pablogsal

#35460: Add PyDict_GetItemStringWithError
https://bugs.python.org/issue35460  closed by ronaldoussoren

#35505: Test test_imaplib fail in test_imap4_host_default_value
https://bugs.python.org/issue35505  closed by vstinner

#35633: test_eintr fails on AIX since fcntl functions were modified
https://bugs.python.org/issue35633  closed by Mariatta

#35688: "pip install --user numpy" fails on Python from the Windows St
https://bugs.python.org/issue35688  closed by mattip

#35706: Make it easier to use a venv with an embedded Python interpret
https://bugs.python.org/issue35706  closed by steve.dower

#35766: Merge typed_ast back into CPython
https://bugs.python.org/issue35766  closed by gvanrossum

#35796: time.localtime returns error for negative values
https://bugs.python.org/issue35796  closed by ammar2

#35833: IDLE: revise doc for control chars sent to Shell
https://bugs.python.org/issue35833  closed by terry.reedy

#35846: Incomplete documentation for re.sub
https://bugs.python.org/issue35846  closed by serhiy.storchaka

#35875: Crash by Pandas - algos.cp36-win_amd64.pyd  join.cp36-win_amd6
https://bugs.python.org/issue35875  closed by eryksun

#35878: ast.c: end_col_offset may be used uninitialized in this functi
https://bugs.python.org/issue35878  closed by levkivskyi

#35887: Doc string for updating the frozen version of importlib in _bo
https://bugs.python.org/issue35887  closed by brett.cannon

#35903: Build of posixshmem.c should probe for required OS functions
https://bugs.python.org/issue35903  closed by nascheme

#35918: multiprocessing's SyncManager.dict.has_key() method is broken
https://bugs.python.org/issue35918  closed by josh.r

#35934: Add socket.create_server() utility function
https://bugs.python.org/issue35934  closed by giampaolo.rodola

#35940: multiprocessing manager tests fail in the Refleaks buildbots
https://bugs.python.org/issue35940  closed by pablogsal

#35948: update version of libffi in cpython-sources-dep
https://bugs.python.org/issue35948  closed by steve.dower

#35958: io.IOBase subclasses don't play nice with abc.abstractmethod
https://bugs.python.org/issue35958  closed by josh.r

#35960: dataclasses.field does not preserve empty metadata object
https://bugs.python.org/issue35960  closed by eric.smith

#35961: test_slice: gc_decref: Assertion "gc_get_refs(g) > 0" failed: 
https://bugs.python.org/issue35961  closed by vstinner

#35963: Python/symtable.c: warning: enumeration value ???FunctionType_
https://bugs.python.org/issue35963  closed by gvanrossum

#35965: Behavior for unittest.assertRaisesRegex differs depending on w
https://bugs.python.org/issue35965  closed by xtreak

#35966: Didn't raise "StopIteration" Error when I use "yield" in the f
https://bugs.python.org/issue35966  closed by SilentGhost

#35968: lib2to3 cannot parse rf''
https://bugs.python.org/issue35968  closed by ned.deily

#35972: _xxsubinterpreters: channel_send() may truncate ints on 32-bit
https://bugs.python.org/issue35972  closed by eric.snow

#35973: `growable_int_array type_ignores` in parsetok.c is not always 
https://bugs.python.org/issue35973  closed by gvanrossum

#35976: Enable Windows projects to build with platform ARM32
https://bugs.python.org/issue35976  closed by steve.dower

#35977: test_slice crashed on s390x Debian 3.x: gc_decref: Assertion "
https://bugs.python.org/issue35977  closed by vstinner

#35979: Incorrect __text_signature__ for the __get__ slot wrapper
https://bugs.python.org/issue35979  closed by SilentGhost

#35980: Py3 BIF random.choices() is O(N**2) but I've written O(N) code
https://bugs.python.org/issue35980  closed by mark.dickinson

#35985: Folding tries to slice from 0 to float("+inf") when maxlength 
https://bugs.python.org/issue35985  closed by Lukas J

#35986: print() documentation typo?
https://bugs.python.org/issue35986  closed by cheryl.sabella

#35987: Mypy and Asyncio import cannot be skipped
https://bugs.python.org/issue35987  closed by levkivskyi

#35988: Python interpreter segfault
https://bugs.python.org/issue35988  closed by josh.r

#35991: potential double free in Modules/_randommodule.c line 295 and 
https://bugs.python.org/issue35991  closed by benjamin.peterson

#35994: In WalkTests of test_os.py, sub2_tree missed the dir "SUB21" i
https://bugs.python.org/issue35994  closed by zach.ware

#36005: [2.7] test_ssl failures on ARMv7 Ubuntu 2.7 with OpenSSL 1.1.1
https://bugs.python.org/issue36005  closed by gregory.p.smith

From status at bugs.python.org  Tue Mar  5 11:30:58 2019
From: status at bugs.python.org (Python tracker)
Date: Tue,  5 Mar 2019 16:30:58 +0000 (UTC)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190305163058.EA55852B1CF@bugs.ams1.psf.io>


ACTIVITY SUMMARY (2019-02-15 - 2019-02-22)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6999 (-10)
  closed 40834 (+87)
  total  47833 (+77)

Open issues with patches: 2795 


Issues opened (45)
==================

#36008: [good first issue] Update documentation for 3.8
https://bugs.python.org/issue36008  opened by cheryl.sabella

#36010: Please provide a .zip Windows release of Python that is not cr
https://bugs.python.org/issue36010  opened by jt

#36011: ssl - tls verify on Windows fails
https://bugs.python.org/issue36011  opened by chris-k

#36015: streamhandler cannot represent streams with an integer as name
https://bugs.python.org/issue36015  opened by Riccardo Magliocchetti

#36016: Allow gc.getobjects to return the objects tracked by a specifi
https://bugs.python.org/issue36016  opened by pablogsal

#36017: test_grp
https://bugs.python.org/issue36017  opened by mjbrands

#36018: Add a Normal Distribution class to the statistics module
https://bugs.python.org/issue36018  opened by rhettinger

#36019: test_urllib fail in s390x buildbots: http://www.example.com/
https://bugs.python.org/issue36019  opened by pablogsal

#36020: HAVE_SNPRINTF and MSVC std::snprintf support
https://bugs.python.org/issue36020  opened by palotasb-conti

#36021: [Security][Windows] webbrowser: WindowsDefault uses os.startfi
https://bugs.python.org/issue36021  opened by vstinner

#36022: [Security] logging.config should not use eval()
https://bugs.python.org/issue36022  opened by vstinner

#36023: Improve configparser.ConfigParser repr
https://bugs.python.org/issue36023  opened by remi.lapeyre

#36025: Breaking change in PyDate_FromTimeStamp API
https://bugs.python.org/issue36025  opened by p-ganssle

#36026: Different error message when sys.settrace is used
https://bugs.python.org/issue36026  opened by SylvainDe

#36027: Support negative exponents in pow() where a modulus is specifi
https://bugs.python.org/issue36027  opened by rhettinger

#36029: Use title-case HTTP header fields
https://bugs.python.org/issue36029  opened by maggyero

#36030: add internal API function to create tuple without items array 
https://bugs.python.org/issue36030  opened by sir-sigurd

#36033: logging.makeLogRecord should update "rv" using a dict defined 
https://bugs.python.org/issue36033  opened by ralonsoh

#36034: Suprise halt caused by -Werror=implicit-function-declaration i
https://bugs.python.org/issue36034  opened by Michael.Felt

#36035: pathlib.Path().rglob() breaks with broken symlinks
https://bugs.python.org/issue36035  opened by J??rg Stucke

#36036: Add method to get user defined command line arguments in unitt
https://bugs.python.org/issue36036  opened by remi.lapeyre

#36041: email: folding of quoted string in display_name violates RFC
https://bugs.python.org/issue36041  opened by aaryn.startmail

#36042: Setting __init_subclass__ and __class_getitem__ methods are in
https://bugs.python.org/issue36042  opened by BTaskaya

#36043: FileCookieJar constructor don't accept PathLike
https://bugs.python.org/issue36043  opened by kapsh

#36044: PROFILE_TASK for PGO build is not a good workload
https://bugs.python.org/issue36044  opened by nascheme

#36045: builtins.help function is not much help with async functions
https://bugs.python.org/issue36045  opened by Dan Rose

#36046: support dropping privileges when running subprocesses
https://bugs.python.org/issue36046  opened by patrick.mclean

#36048: Deprecate implicit truncating when convert Python numbers to C
https://bugs.python.org/issue36048  opened by serhiy.storchaka

#36050: Why does http.client.HTTPResponse._safe_read use MAXAMOUNT
https://bugs.python.org/issue36050  opened by bmerry

#36051: Drop the GIL during large bytes.join operations?
https://bugs.python.org/issue36051  opened by bmerry

#36053: pkgutil.walk_packages jumps out from given path if there is pa
https://bugs.python.org/issue36053  opened by karkucik

#36054: Way to detect CPU count inside docker container
https://bugs.python.org/issue36054  opened by keirlawson

#36058: Improve file decoding before re.search
https://bugs.python.org/issue36058  opened by terry.reedy

#36064: docs: urllib.request.Request not accepting iterables data type
https://bugs.python.org/issue36064  opened by sylye

#36066: Add `empty` block to `for` and `while` loops.
https://bugs.python.org/issue36066  opened by wlohu

#36067: subprocess terminate() "invalid handle" error when process is 
https://bugs.python.org/issue36067  opened by giampaolo.rodola

#36071: Add support for Windows ARM32 in ctypes/libffi
https://bugs.python.org/issue36071  opened by Paul Monson

#36072: str.translate() behaves differently for ASCII-only and other s
https://bugs.python.org/issue36072  opened by sir-sigurd

#36073: sqlite crashes with converters mutating cursor
https://bugs.python.org/issue36073  opened by sir-sigurd

#36075: python 2to3 conversion tool is generating file with extra line
https://bugs.python.org/issue36075  opened by sabakauser

#36076: ssl.get_server_certificate should use SNI
https://bugs.python.org/issue36076  opened by enki

#36077: Inheritance dataclasses fields and default init statement
https://bugs.python.org/issue36077  opened by ???????????? ????????????

#36078: argparse: positional with type=int, default=SUPPRESS raise Val
https://bugs.python.org/issue36078  opened by n8falke

#36079: pdb on setuptools "ValueError: underlying buffer has been deta
https://bugs.python.org/issue36079  opened by Gerrit.Holl

#36081: Cannot set LDFLAGS containing $
https://bugs.python.org/issue36081  opened by Dakon



Most recent 15 issues with no replies (15)
==========================================

#36079: pdb on setuptools "ValueError: underlying buffer has been deta
https://bugs.python.org/issue36079

#36076: ssl.get_server_certificate should use SNI
https://bugs.python.org/issue36076

#36073: sqlite crashes with converters mutating cursor
https://bugs.python.org/issue36073

#36071: Add support for Windows ARM32 in ctypes/libffi
https://bugs.python.org/issue36071

#36058: Improve file decoding before re.search
https://bugs.python.org/issue36058

#36053: pkgutil.walk_packages jumps out from given path if there is pa
https://bugs.python.org/issue36053

#36051: Drop the GIL during large bytes.join operations?
https://bugs.python.org/issue36051

#36036: Add method to get user defined command line arguments in unitt
https://bugs.python.org/issue36036

#36017: test_grp
https://bugs.python.org/issue36017

#36002: configure --enable-optimizations with clang fails to detect ll
https://bugs.python.org/issue36002

#36001: LIBFFI_INCLUDEDIR is not detected when set into a profile nor 
https://bugs.python.org/issue36001

#35999: multpirocessing.Process alive after SIGTERM on parent
https://bugs.python.org/issue35999

#35990: ipaddress.IPv4Interface won't accept 2-tuple (address, mask)
https://bugs.python.org/issue35990

#35974: os.DirEntry.inode() returns invalid value within Docker contai
https://bugs.python.org/issue35974

#35970: no help flag in base64 util
https://bugs.python.org/issue35970



Most recent 15 issues waiting for review (15)
=============================================

#36073: sqlite crashes with converters mutating cursor
https://bugs.python.org/issue36073

#36071: Add support for Windows ARM32 in ctypes/libffi
https://bugs.python.org/issue36071

#36064: docs: urllib.request.Request not accepting iterables data type
https://bugs.python.org/issue36064

#36053: pkgutil.walk_packages jumps out from given path if there is pa
https://bugs.python.org/issue36053

#36046: support dropping privileges when running subprocesses
https://bugs.python.org/issue36046

#36045: builtins.help function is not much help with async functions
https://bugs.python.org/issue36045

#36042: Setting __init_subclass__ and __class_getitem__ methods are in
https://bugs.python.org/issue36042

#36041: email: folding of quoted string in display_name violates RFC
https://bugs.python.org/issue36041

#36036: Add method to get user defined command line arguments in unitt
https://bugs.python.org/issue36036

#36035: pathlib.Path().rglob() breaks with broken symlinks
https://bugs.python.org/issue36035

#36030: add internal API function to create tuple without items array 
https://bugs.python.org/issue36030

#36029: Use title-case HTTP header fields
https://bugs.python.org/issue36029

#36026: Different error message when sys.settrace is used
https://bugs.python.org/issue36026

#36025: Breaking change in PyDate_FromTimeStamp API
https://bugs.python.org/issue36025

#36021: [Security][Windows] webbrowser: WindowsDefault uses os.startfi
https://bugs.python.org/issue36021



Top 10 most discussed issues (10)
=================================

#36021: [Security][Windows] webbrowser: WindowsDefault uses os.startfi
https://bugs.python.org/issue36021  23 msgs

#36010: Please provide a .zip Windows release of Python that is not cr
https://bugs.python.org/issue36010  22 msgs

#35813: shared memory construct to avoid need for serialization betwee
https://bugs.python.org/issue35813  17 msgs

#36027: Support negative exponents in pow() where a modulus is specifi
https://bugs.python.org/issue36027  13 msgs

#36064: docs: urllib.request.Request not accepting iterables data type
https://bugs.python.org/issue36064  13 msgs

#35892: Fix awkwardness of statistics.mode() for multimodal datasets
https://bugs.python.org/issue35892  12 msgs

#36019: test_urllib fail in s390x buildbots: http://www.example.com/
https://bugs.python.org/issue36019  12 msgs

#35925: test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster
https://bugs.python.org/issue35925  10 msgs

#36020: HAVE_SNPRINTF and MSVC std::snprintf support
https://bugs.python.org/issue36020  10 msgs

#36026: Different error message when sys.settrace is used
https://bugs.python.org/issue36026  10 msgs



Issues closed (84)
==================

#1104: msilib.SummaryInfo.GetProperty() truncates the string by one c
https://bugs.python.org/issue1104  closed by steve.dower

#12822: NewGIL should use CLOCK_MONOTONIC if possible.
https://bugs.python.org/issue12822  closed by inada.naoki

#13659: Add a help() viewer for IDLE's Shell.
https://bugs.python.org/issue13659  closed by terry.reedy

#14929: IDLE crashes on *Edit / Find in files ...* command
https://bugs.python.org/issue14929  closed by terry.reedy

#15248: Better explain "TypeError: 'tuple' object is not callable"
https://bugs.python.org/issue15248  closed by serhiy.storchaka

#20687: Change in expectedFailure breaks testtools
https://bugs.python.org/issue20687  closed by cheryl.sabella

#22704: Review extension enable options
https://bugs.python.org/issue22704  closed by terry.reedy

#23077: PEP 1: Allow Provisional status for PEPs
https://bugs.python.org/issue23077  closed by cheryl.sabella

#23428: Use the monotonic clock for thread conditions on POSIX platfor
https://bugs.python.org/issue23428  closed by inada.naoki

#26787: test_distutils fails when configured --with-lto
https://bugs.python.org/issue26787  closed by inada.naoki

#27632: build on AIX fails when builddir != srcdir, more than bad path
https://bugs.python.org/issue27632  closed by Michael.Felt

#28235: In xml.etree.ElementTree docs there is no parser argument in f
https://bugs.python.org/issue28235  closed by cheryl.sabella

#28328: statistics.geometric_mean has no tests. Defer to 3.7?
https://bugs.python.org/issue28328  closed by inada.naoki

#29278: Python 3.6 build fails with parallel make
https://bugs.python.org/issue29278  closed by inada.naoki

#30449: Improve __slots__ datamodel documentation
https://bugs.python.org/issue30449  closed by cheryl.sabella

#31982: 8.3. collections ??? Container datatypes
https://bugs.python.org/issue31982  closed by rhettinger

#32108: configparser bug: section is emptied if you assign a section t
https://bugs.python.org/issue32108  closed by cheryl.sabella

#32657: Mutable Objects in SMTP send_message Signature
https://bugs.python.org/issue32657  closed by inada.naoki

#32808: subprocess.check_output opens an unwanted command line window 
https://bugs.python.org/issue32808  closed by cheryl.sabella

#33961: Inconsistency in exceptions for dataclasses.dataclass document
https://bugs.python.org/issue33961  closed by inada.naoki

#33989: ms.key_compare is not initialized in all paths of list_sort_im
https://bugs.python.org/issue33989  closed by serhiy.storchaka

#34203: documentation: recommend Python 3 over 2 in faq
https://bugs.python.org/issue34203  closed by abcdef

#34294: re module: wrong capturing groups
https://bugs.python.org/issue34294  closed by serhiy.storchaka

#34315: Regex not evalauated correctly
https://bugs.python.org/issue34315  closed by serhiy.storchaka

#34572: C unpickling bypasses import thread safety
https://bugs.python.org/issue34572  closed by pitrou

#34720: Fix test_importlib.test_bad_traverse for AIX
https://bugs.python.org/issue34720  closed by ncoghlan

#34757: Placeholder for discussion on Combined patches for AIX - to re
https://bugs.python.org/issue34757  closed by Michael.Felt

#34785: pty.spawn -- auto-termination after child process is dead (a z
https://bugs.python.org/issue34785  closed by martin.panter

#34817: Ellipsis docs has extra dot in the markdown that makes it look
https://bugs.python.org/issue34817  closed by xtreak

#34981: Unable to install Python from web-based installer and executab
https://bugs.python.org/issue34981  closed by inada.naoki

#35126: Mistake in FAQ about converting number to string.
https://bugs.python.org/issue35126  closed by cheryl.sabella

#35153: Allow to set headers in xmlrpc.client.ServerProxy
https://bugs.python.org/issue35153  closed by vstinner

#35186: distutils.command.upload uses deprecated platform.dist with bd
https://bugs.python.org/issue35186  closed by p-ganssle

#35307: Command line help example  is missing "--prompt" option
https://bugs.python.org/issue35307  closed by cheryl.sabella

#35500: Align expected and actual calls on mock.assert_called_with err
https://bugs.python.org/issue35500  closed by taleinat

#35584: Wrong statement about ^ in howto/regex.rst
https://bugs.python.org/issue35584  closed by rhettinger

#35689: IDLE: Docstrings and test for colorizer
https://bugs.python.org/issue35689  closed by terry.reedy

#35704: On AIX, test_unpack_archive_xztar fails with default MAXDATA s
https://bugs.python.org/issue35704  closed by ncoghlan

#35798: duplicate SyntaxWarning: "is" with a literal
https://bugs.python.org/issue35798  closed by serhiy.storchaka

#35812: Don't log an exception from the main coroutine in asyncio.run(
https://bugs.python.org/issue35812  closed by asvetlov

#35822: _queue _queuemodule.c is missing inside the Setup file
https://bugs.python.org/issue35822  closed by inada.naoki

#35826: Typo in example for async with statement with condition
https://bugs.python.org/issue35826  closed by mhchia

#35852: Fixed tests regenerating using CRLF when running it on Windows
https://bugs.python.org/issue35852  closed by addons_zz

#35867: NameError is not caught at Task execution
https://bugs.python.org/issue35867  closed by pablogsal

#35904: Add statistics.fmean(seq)
https://bugs.python.org/issue35904  closed by rhettinger

#35915: re.search extreme slowness (looks like hang/livelock), searchi
https://bugs.python.org/issue35915  closed by serhiy.storchaka

#35942: posixmodule.c:path_converter() returns an invalid exception me
https://bugs.python.org/issue35942  closed by pablogsal

#35945: Cannot distinguish between subtask cancellation and running ta
https://bugs.python.org/issue35945  closed by jnwatson

#35956: Sort documentation could be improved for complex sorting
https://bugs.python.org/issue35956  closed by rhettinger

#35984: test__xxsubinterpreters leaked [3, 4, 3] memory blocks, sum=1
https://bugs.python.org/issue35984  closed by eric.snow

#35992: Metaclasses interfere with __class_getitem__
https://bugs.python.org/issue35992  closed by levkivskyi

#35993: incorrect use of released memory in Python/pystate.c line 284
https://bugs.python.org/issue35993  closed by vstinner

#36006: [good first issue] Align version changed for truncate in io mo
https://bugs.python.org/issue36006  closed by Mariatta

#36007: python3.8 a1 - docs build requires sphinx 1.7 but uses a 1.8 f
https://bugs.python.org/issue36007  closed by Mariatta

#36009: weakref.ReferenceType is not a valid typing type
https://bugs.python.org/issue36009  closed by levkivskyi

#36012: Investigate slow writes to class variables
https://bugs.python.org/issue36012  closed by rhettinger

#36013: test_signal fails in AMD64 Debian PGO 3.x
https://bugs.python.org/issue36013  closed by gregory.p.smith

#36014: test_help_with_metavar broken
https://bugs.python.org/issue36014  closed by wabba

#36024: ctypes: test_ctypes test_callbacks() crash on AArch64 with SEL
https://bugs.python.org/issue36024  closed by vstinner

#36028: Integer Division discrepancy with float
https://bugs.python.org/issue36028  closed by mark.dickinson

#36031: add internal API function to effectively convert just created 
https://bugs.python.org/issue36031  closed by vstinner

#36032: Wrong output in tutorial (3.1.2. Strings)
https://bugs.python.org/issue36032  closed by serhiy.storchaka

#36037: test_ssl fails on RHEL8 strict OpenSSL configuration
https://bugs.python.org/issue36037  closed by vstinner

#36038: ^ used in inaccurate example in regex-howto
https://bugs.python.org/issue36038  closed by mdk

#36039: Replace append loops with list comprehensions
https://bugs.python.org/issue36039  closed by rhettinger

#36040: Python\ast.c(3875): warning C4244: 'initializing': conversion 
https://bugs.python.org/issue36040  closed by pablogsal

#36047: socket file handle does not support stream write
https://bugs.python.org/issue36047  closed by matrixise

#36049: No __repr__() for queue.PriorityQueue and queue.LifoQueue
https://bugs.python.org/issue36049  closed by rhettinger

#36052: Assignment operator allows to assign to __debug__
https://bugs.python.org/issue36052  closed by pablogsal

#36055: Division using math.pow and math.log approximation fails
https://bugs.python.org/issue36055  closed by mark.dickinson

#36056: importlib does not support pathlib
https://bugs.python.org/issue36056  closed by berker.peksag

#36057: Add docs and tests for ordering in Counter.  [no behavior chan
https://bugs.python.org/issue36057  closed by rhettinger

#36059: Update docs for OrderedDict to reflect that regular dicts are 
https://bugs.python.org/issue36059  closed by rhettinger

#36060: Document how collections.ChainMap() determines iteration order
https://bugs.python.org/issue36060  closed by rhettinger

#36061: zipfile does not handle arcnames with non-ascii characters on 
https://bugs.python.org/issue36061  closed by inada.naoki

#36062: move index normalization from list_slice() to PyList_GetSlice(
https://bugs.python.org/issue36062  closed by serhiy.storchaka

#36063: replace PyTuple_SetItem() with PyTuple_SET_ITEM() in long_divm
https://bugs.python.org/issue36063  closed by serhiy.storchaka

#36065: Add unified C API for accessing bytes and bytearray
https://bugs.python.org/issue36065  closed by serhiy.storchaka

#36068: Make _tuplegetter objects serializable
https://bugs.python.org/issue36068  closed by rhettinger

#36069: asyncio: create_connection cannot handle IPv6 link-local addre
https://bugs.python.org/issue36069  closed by Leonardo M??rlein

#36070: Enclosing scope not visible from within list comprehension
https://bugs.python.org/issue36070  closed by serhiy.storchaka

#36074: Result of `asyncio.Server.sockets` after `Server.close()` is n
https://bugs.python.org/issue36074  closed by mdk

#36080: Ensurepip fails to install pip into a nested virtual environme
https://bugs.python.org/issue36080  closed by Sammy Gillespie

#36082: The built-in round() function giving a wrong output
https://bugs.python.org/issue36082  closed by SilentGhost

From steve.dower at python.org  Tue Mar  5 11:55:41 2019
From: steve.dower at python.org (Steve Dower)
Date: Tue, 5 Mar 2019 08:55:41 -0800
Subject: [Python-Dev] [RELEASE] Python 2.7.16
In-Reply-To: <q5kg1n$3cnc$1@blaine.gmane.org>
References: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>
 <q5kg1n$3cnc$1@blaine.gmane.org>
Message-ID: <b7acd184-fd29-f1f2-642f-5073d51f3436@python.org>

On 04Mar2019 1631, Terry Reedy wrote:
> On 3/3/2019 10:30 PM, Benjamin Peterson wrote:
> 
>> I'm pleased to announce the immediate availability of Python 2.7.16 
>> for download at https://www.python.org/downloads/release/python-2716/.
> 
> On Windows 10, this is an 'unrecognized app' and Windows Defender 
> SmartScreen, now default, refuses to run it. "Windows protected your PC" 
> until one clicks 'more info' to get 'Run anyway'.? This is new since the 
> .rc1 release.? We should either make 2.7 a 'known' app* or say something 
> on the download page about clicking 'more info'.? I don't know the 
> status of python.org 3.x downloads.
> 
> Since Steve Dower put 3.7 on the Windows store, PSF must now be a known 
> publisher.? Perhaps he can help make 2.7 'known'.

SmartScreen should recognize it now that it's been downloaded a few 
times without reporting, but the real problem is that it hasn't been 
signed properly. (I don't have any validation set up for Python 2 on this.)

I'll try and sign the version on my build machine by hand and upload it 
again this week, and maybe add a validation step to interrupt the upload 
if it's not signed, but anything Python 2 is a very low priority for me 
(sorry Benjamin :) ).

Cheers,
Steve

From ezio.melotti at gmail.com  Tue Mar  5 11:57:24 2019
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Tue, 5 Mar 2019 17:57:24 +0100
Subject: [Python-Dev] Summary of Python tracker Issues
In-Reply-To: <CACBhJdGqHTJCrZfGDtEnwBMs5JXTpjyNa04qiR5sOf4yb9OzbQ@mail.gmail.com>
References: <20190208180753.1.782F2F2B9AA3561B@roundup.psfhosted.org>
 <CAEfz+TwdqQm829Q_2RWJq9QVDsnRX53JkFYVMqFP0G+OSqJKiA@mail.gmail.com>
 <CAK256p3ruU-RCv-k_XeXSsu+33oKKnE29YK=bK=pQBJb=7tEgg@mail.gmail.com>
 <q59m60$1i0v$1@blaine.gmane.org>
 <CAK256p1kP_D9AYYfCbcgOXp5tb7oMHVbZ8x7JBi9yO9QsO4B0g@mail.gmail.com>
 <bb74cef5-2393-dccb-a6c5-5e491bbf66be@g.nevcal.com>
 <q59oo7$71ib$1@blaine.gmane.org>
 <5d12e862-5398-5701-2503-b745c0875976@g.nevcal.com>
 <q5ae7e$603g$1@blaine.gmane.org>
 <CACBhJdGqHTJCrZfGDtEnwBMs5JXTpjyNa04qiR5sOf4yb9OzbQ@mail.gmail.com>
Message-ID: <CACBhJdFi0xLrwFuC6Fiqz=V9nM67ku5=2c7WGGSPt-ptKE1dAA@mail.gmail.com>

On Fri, Mar 1, 2019 at 8:05 AM Ezio Melotti <ezio.melotti at gmail.com> wrote:
>
> On Fri, Mar 1, 2019 at 5:59 AM Terry Reedy <tjreedy at udel.edu> wrote:
> >
> > On 2/28/2019 6:54 PM, Glenn Linderman wrote:
> >
> > > There seems to be enough evidence that something went wrong somewhere,
> > > though, and whoever maintains that process should start investigating,
> > > but it would still be nice to get confirmation from a non-Google email
> > > recipient whether they did or did not get the Summary messages.
> > >
> > > I wonder if there is a way to manually send them, and if the missing two
> > > weeks of activity can be reported... once the sending problem is
> > > understood and resolved.
> >
> > I posted a note to the core-workflow list, but I don't know if anyone
> > with power or knowledge still reads it.
> >
>
> The tracker got migrated recently, and that's the most likely cause of
> the missing reports.
> We'll look into it and get them back :)
>

Ernest promptly fixed the issue so last week report was sent out correctly.
I just generated and sent out the two reports that were missing and
updated the tracker stats at
https://bugs.python.org/issue?@template=stats
Note that some values in the report might be a bit off (for example,
the list of issues waiting for review doesn't include issues that were
closed after the selected period, and the patch count includes issues
created before or during the selected period even if a patch was
uploaded after the selected period).
The issues counts and deltas at the top of the summary should be correct.
Let me know if you notice any other problem (and thanks Ned for
bringing this to my attention!).

Best Regards,
Ezio Melotti

> > To get a listing, go to the tracker search page, put
> > 2019-02-09 to 2019-03-01
> > in the date box, and change status to don't care.  At the moment, this
> > returns 204 issues.
> >
> > --
> > Terry Jan Reedy
> >

From steve.dower at python.org  Tue Mar  5 12:14:11 2019
From: steve.dower at python.org (Steve Dower)
Date: Tue, 5 Mar 2019 09:14:11 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CANbgQ_GNqk19iOpotgGT_mNFUFNBEHoNoEm9tVpV1rZOnLw1kQ@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
 <5C7E789B.9010007@UGent.be>
 <CANbgQ_GNqk19iOpotgGT_mNFUFNBEHoNoEm9tVpV1rZOnLw1kQ@mail.gmail.com>
Message-ID: <65a0900d-36f3-32f0-1cd0-e4403368f855@python.org>

On 05Mar2019 0620, James Edwards wrote:
> I have to say, this is sort of surprising for what seems like the first 
> official action of the steering?committee.? Are there really /that many 
> /PEPs that a team that is now, what, 5x the size of the BFDL model is 
> worried that they'll be able to keep up?? As a long-time lurker, this 
> hardly seems to be the case.? Despite the seemingly-well-intentioned 
> rationale, this seems like an ominous sign.

FWIW, I'm 100% on board with this idea (and feel free to continue 
stealing ideas from PEP 8013 ;) ). However, the presentation of it 
certainly didn't emphasize the good rationale for the decision.

In short, the five person steering council is not the equivalent of 5x 
BDFL. If they want to delegate early decision making to the core team as 
a whole, they can, and that's what they've done here.

For non-core developers, and particularly first-time contributors, the 
PEP process typically starts at python-ideas. (Even core devs probably 
ought to start there, though part of being accepted as a core developer 
means we trust your judgement in selecting the correct venue for 
discussion, so if security-sig, datetime-sig, capi-sig, core-workflow, 
or just python-dev is more appropriate, feel free to start there.)

To "get out" of python-ideas, someone needs to suggest where it goes 
next. Most of the time, this is python-dev. However, if you don't have 
*a single* core developer on board from python-ideas, chances are the 
whole team is going to reject the proposal.

In the past, an idea would be shut down by just one negative vote 
(Guido's). In this future, an idea is promoted by just one positive vote 
(any core developer's). It's actually much easier for an "outsider" to 
get their idea in front of the whole core team than before. And 
python-ideas has core developers and regular contributors who have 
self-selected to "triage" ideas and help move them along. If the 
triagers don't like your idea, it's probably not a good idea :)

Asking for proclamation from the council/delegate is literally saying 
"this proposal is ready". As a contributor (first time or 100th time), 
if you think your proposal is ready without *anyone else* agreeing with 
you, then *we* think you have a humility problem and we'd like you to go 
work on that.

It's not a big ask to have one of the lower level mailing lists look at 
your proposal before the council has to make an official decision. You 
should *want* the mailing lists to look at your proposal. I certainly 
do, because every time they do my proposals get better, and I get better 
at writing proposals. This is a situation where everyone wins.

Cheers,
Steve

From brett at python.org  Tue Mar  5 14:30:18 2019
From: brett at python.org (Brett Cannon)
Date: Tue, 5 Mar 2019 11:30:18 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <5C7E132D.20004@UGent.be>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <5C7E132D.20004@UGent.be>
Message-ID: <CAP1=2W7pYv_ujxzvTZ52bRz3qvh0PGrBq28RjvPzfiV8wFfR6A@mail.gmail.com>

On Mon, Mar 4, 2019 at 10:13 PM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> Does this apply to existing draft PEPs or only new ones?
>

Only new ones; this is not retroactive.

-Brett


> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190305/777fd7db/attachment.html>

From brett at python.org  Tue Mar  5 14:41:18 2019
From: brett at python.org (Brett Cannon)
Date: Tue, 5 Mar 2019 11:41:18 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <5C7E789B.9010007@UGent.be>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
 <5C7E789B.9010007@UGent.be>
Message-ID: <CAP1=2W73Tk5C_tHKLjQK3Oi1q80ir0Bd9ZFO3VQJP4xgba9PBA@mail.gmail.com>

On Tue, Mar 5, 2019 at 5:33 AM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> On 2019-03-05 14:05, Calvin Spealman wrote:
> > I'm worried this creates a gatekeeping perception that will scare away
> > contributors.
>

It might, but if people are not prepared properly for the PEP process then
it's best to spare them and everyone else the time sink.


>
> +1
>
> I also expressed this worry at https://github.com/python/peps/pull/903
>
> You could keep the positive part of the sponsoring idea (somebody acting
> as mentor) but drop the negative part (make it a hard requirement to
> find a sponsor supporting the proposal before the proposal can even
> become a draft PEP).
>

The hard requirement is on purpose. This is not a suggestion for a reason
as PEPs are not a cheap, cost-free thing for both authors and those
participating in the discussion. They take up a ton of time and if people
are not properly equipped to be successful then it just leads to a massive
loss of time that could be been better used by everyone involved.

Please also realize that the top 4 PEP authors are on the steering council,
and some of the longest active core devs and PEP editors are on the
council, so this isn't coming from people that lack experience and exposure
to all facets of the PEP process.

As I said in my announcement email, if this turns out to be a bad decision
then we can change the process back/again, but please do realize this is
not coming out of thin air just because we like fiddling with the PEP
process.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190305/1881c105/attachment.html>

From brett at snarky.ca  Tue Mar  5 14:31:08 2019
From: brett at snarky.ca (Brett Cannon)
Date: Tue, 5 Mar 2019 11:31:08 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CA+3bQGFhgPymJSrsAqAkkeLv0kDWFQuhiVushuz__kD4jQp5wQ@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CA+3bQGFhgPymJSrsAqAkkeLv0kDWFQuhiVushuz__kD4jQp5wQ@mail.gmail.com>
Message-ID: <CAP1=2W7bohs32ViRO4jfY8kVSHiFGiEoW8vqnExYbcsLHYYxUg@mail.gmail.com>

On Tue, Mar 5, 2019 at 3:36 AM Victor Stinner <vstinner at redhat.com> wrote:

> Hi,
>
> Le mar. 5 mars 2019 ? 02:53, Brett Cannon <brett at snarky.ca> a ?crit :
> > The steering council has implemented a new idea called sponsors to the
> PEP process (...). The thinking is that to help make sure PEPs from
> non-core developers receive appropriate guidance through the PEP process
> (...)
>
> Hum, this isn't fully new, some PEPs already got a "PEP champion" (old
> name), no?
>

Sure. You can consider this just making it a more formal thing then.

-Brett


>
> A recent example is PEP 572 who has been "championed" by Guido van
> Rossum, then by Tim Peters. In this specific case, they became
> co-authors :-)
>
> > ... eventually becoming a co-author or BDFL-delegate later
>
> Nitpick: since Python has no more BDFL, maybe the expression should
> becocme "PEP-delegate"? ;-)
>

That's covered in PEP 1 as to why the name has been kept so far.

-Brett


>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190305/8488e36c/attachment.html>

From J.Demeyer at UGent.be  Tue Mar  5 14:58:43 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Tue, 5 Mar 2019 20:58:43 +0100
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <65a0900d-36f3-32f0-1cd0-e4403368f855@python.org>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
 <5C7E789B.9010007@UGent.be>
 <CANbgQ_GNqk19iOpotgGT_mNFUFNBEHoNoEm9tVpV1rZOnLw1kQ@mail.gmail.com>
 <65a0900d-36f3-32f0-1cd0-e4403368f855@python.org>
Message-ID: <5C7ED4F3.10302@UGent.be>

On 2019-03-05 18:14, Steve Dower wrote:
> However, if you don't have
> *a single* core developer on board from python-ideas, chances are the
> whole team is going to reject the proposal.

Sure, I couldn't agree more. But this is something that a PEP mentor 
(instead of sponsor) also could deal with. Any potential mentor would 
quickly dismiss the PEP as having no chance and that would work just fine.

The problem with the "sponsor" idea is that the sponsor must come from 
the group of core devs supporting the PEP. What if all core devs 
supporting it don't have time to act as sponsor or just don't care enough?

On the other hand, if there is some support for an idea, then anybody 
should be able to mentor even if the mentor doesn't personally support 
the idea. I guess the mentor shouldn't be opposed either, but there is a 
large gray zone of -0/+0 in between where mentors could come from.


Jeroen.

From brett at python.org  Tue Mar  5 15:39:40 2019
From: brett at python.org (Brett Cannon)
Date: Tue, 5 Mar 2019 12:39:40 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <5C7ED4F3.10302@UGent.be>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
 <5C7E789B.9010007@UGent.be>
 <CANbgQ_GNqk19iOpotgGT_mNFUFNBEHoNoEm9tVpV1rZOnLw1kQ@mail.gmail.com>
 <65a0900d-36f3-32f0-1cd0-e4403368f855@python.org> <5C7ED4F3.10302@UGent.be>
Message-ID: <CAP1=2W4RKiQBUokR9xforwBMQA9=1qhXA_a3_99x6hUjhnS68w@mail.gmail.com>

On Tue, Mar 5, 2019 at 11:59 AM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> On 2019-03-05 18:14, Steve Dower wrote:
> > However, if you don't have
> > *a single* core developer on board from python-ideas, chances are the
> > whole team is going to reject the proposal.
>
> Sure, I couldn't agree more. But this is something that a PEP mentor
> (instead of sponsor) also could deal with. Any potential mentor would
> quickly dismiss the PEP as having no chance and that would work just fine.
>
> The problem with the "sponsor" idea is that the sponsor must come from
> the group of core devs supporting the PEP. What if all core devs
> supporting it don't have time to act as sponsor or just don't care enough?
>

If no one cares enough then there isn't enough support to begin with. I
can't think of a PEP that was accepted with a universally lukewarm
reception; there's always been at least one person who liked the idea to
care.


> On the other hand, if there is some support for an idea, then anybody
> should be able to mentor even if the mentor doesn't personally support
> the idea. I guess the mentor shouldn't be opposed either, but there is a
> large gray zone of -0/+0 in between where mentors could come from.
>

If the hypothetical situation comes up of:

   1. An external person proposes a PEP
   2. No supportive core dev has time to sponsor
   3. Someone who doesn't support the PEP is willing to sponsor

then we can talk about changing the process, but otherwise we are worrying
about a hypothetical situation versus personal experience which suggests
this won't be the case. IOW we need to give this a shot before we consider
changing it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190305/31f8ef4f/attachment.html>

From rosuav at gmail.com  Tue Mar  5 15:45:41 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 6 Mar 2019 07:45:41 +1100
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CAP1=2W4RKiQBUokR9xforwBMQA9=1qhXA_a3_99x6hUjhnS68w@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
 <5C7E789B.9010007@UGent.be>
 <CANbgQ_GNqk19iOpotgGT_mNFUFNBEHoNoEm9tVpV1rZOnLw1kQ@mail.gmail.com>
 <65a0900d-36f3-32f0-1cd0-e4403368f855@python.org> <5C7ED4F3.10302@UGent.be>
 <CAP1=2W4RKiQBUokR9xforwBMQA9=1qhXA_a3_99x6hUjhnS68w@mail.gmail.com>
Message-ID: <CAPTjJmpZKOrWz6DPqtxdDdWUOyvNHH=D7XzRRbwmaEU4ai9_zg@mail.gmail.com>

On Wed, Mar 6, 2019 at 7:41 AM Brett Cannon <brett at python.org> wrote:
>
>
>
> On Tue, Mar 5, 2019 at 11:59 AM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
>>
>> On 2019-03-05 18:14, Steve Dower wrote:
>> > However, if you don't have
>> > *a single* core developer on board from python-ideas, chances are the
>> > whole team is going to reject the proposal.
>>
>> Sure, I couldn't agree more. But this is something that a PEP mentor
>> (instead of sponsor) also could deal with. Any potential mentor would
>> quickly dismiss the PEP as having no chance and that would work just fine.
>>
>> The problem with the "sponsor" idea is that the sponsor must come from
>> the group of core devs supporting the PEP. What if all core devs
>> supporting it don't have time to act as sponsor or just don't care enough?
>
>
> If no one cares enough then there isn't enough support to begin with. I can't think of a PEP that was accepted with a universally lukewarm reception; there's always been at least one person who liked the idea to care.
>
>>
>> On the other hand, if there is some support for an idea, then anybody
>> should be able to mentor even if the mentor doesn't personally support
>> the idea. I guess the mentor shouldn't be opposed either, but there is a
>> large gray zone of -0/+0 in between where mentors could come from.
>
>
> If the hypothetical situation comes up of:
>
> An external person proposes a PEP
> No supportive core dev has time to sponsor
> Someone who doesn't support the PEP is willing to sponsor
>
> then we can talk about changing the process, but otherwise we are worrying about a hypothetical situation versus personal experience which suggests this won't be the case. IOW we need to give this a shot before we consider changing it.
>

How much effort does it take to sponsor a PEP? I'm not a core dev, but
I can help someone with the work of writing and publishing. So if, in
that hypothetical situation, some (very busy) core dev were willing to
say "yeah, go ahead, put my name on it", would that be sufficient? If
so, it shouldn't be a problem to require this - any proposal with
enough support to be worth PEPing should have at least one person
who's willing to have his/her name in the headers.

ChrisA

From steve.dower at python.org  Tue Mar  5 16:02:54 2019
From: steve.dower at python.org (Steve Dower)
Date: Tue, 5 Mar 2019 13:02:54 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CAPTjJmpZKOrWz6DPqtxdDdWUOyvNHH=D7XzRRbwmaEU4ai9_zg@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
 <5C7E789B.9010007@UGent.be>
 <CANbgQ_GNqk19iOpotgGT_mNFUFNBEHoNoEm9tVpV1rZOnLw1kQ@mail.gmail.com>
 <65a0900d-36f3-32f0-1cd0-e4403368f855@python.org> <5C7ED4F3.10302@UGent.be>
 <CAP1=2W4RKiQBUokR9xforwBMQA9=1qhXA_a3_99x6hUjhnS68w@mail.gmail.com>
 <CAPTjJmpZKOrWz6DPqtxdDdWUOyvNHH=D7XzRRbwmaEU4ai9_zg@mail.gmail.com>
Message-ID: <021c02d0-2ba8-db1c-d074-b07e741ed9bd@python.org>

On 05Mar2019 1245, Chris Angelico wrote:
> How much effort does it take to sponsor a PEP? I'm not a core dev, but
> I can help someone with the work of writing and publishing. So if, in
> that hypothetical situation, some (very busy) core dev were willing to
> say "yeah, go ahead, put my name on it", would that be sufficient? If
> so, it shouldn't be a problem to require this - any proposal with
> enough support to be worth PEPing should have at least one person
> who's willing to have his/her name in the headers.

For the record, now that he's joined the conversation, Chris is who I 
had in mind when I invented the term "PEP triager" in my email :)

If we had a way of appointing people who we trust to be non-core dev 
sponsors of PEPs, I'd nominate him. Though I suspect he's well known 
enough to the council that they'd accept his support of a PEP as 
sufficient to consider it from someone who's otherwise completely 
unknown. There are always grey areas in any policy.

Cheers,
Steve

From steve at holdenweb.com  Tue Mar  5 17:53:58 2019
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 5 Mar 2019 22:53:58 +0000
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <021c02d0-2ba8-db1c-d074-b07e741ed9bd@python.org>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CACo5Rz5k=iBypcHFtGXE1mR-3RiME8Z1=4K=fiKTWXH5Zzjn_w@mail.gmail.com>
 <5C7E789B.9010007@UGent.be>
 <CANbgQ_GNqk19iOpotgGT_mNFUFNBEHoNoEm9tVpV1rZOnLw1kQ@mail.gmail.com>
 <65a0900d-36f3-32f0-1cd0-e4403368f855@python.org> <5C7ED4F3.10302@UGent.be>
 <CAP1=2W4RKiQBUokR9xforwBMQA9=1qhXA_a3_99x6hUjhnS68w@mail.gmail.com>
 <CAPTjJmpZKOrWz6DPqtxdDdWUOyvNHH=D7XzRRbwmaEU4ai9_zg@mail.gmail.com>
 <021c02d0-2ba8-db1c-d074-b07e741ed9bd@python.org>
Message-ID: <CAMofdRC+M14VeQBu_rzvf2V4MXhMGaFZ2bR0h_GOObJPtswSGg@mail.gmail.com>

On Tue, Mar 5, 2019 at 9:04 PM Steve Dower <steve.dower at python.org> wrote:

> On 05Mar2019 1245, Chris Angelico wrote:
> > How much effort does it take to sponsor a PEP? I'm not a core dev, but
> > I can help someone with the work of writing and publishing. So if, in
> > that hypothetical situation, some (very busy) core dev were willing to
> > say "yeah, go ahead, put my name on it", would that be sufficient? If
> > so, it shouldn't be a problem to require this - any proposal with
> > enough support to be worth PEPing should have at least one person
> > who's willing to have his/her name in the headers.
>
> For the record, now that he's joined the conversation, Chris is who I
> had in mind when I invented the term "PEP triager" in my email :)
>
> If we had a way of appointing people who we trust to be non-core dev
> sponsors of PEPs, I'd nominate him. Though I suspect he's well known
> enough to the council that they'd accept his support of a PEP as
> sufficient to consider it from someone who's otherwise completely
> unknown. There are always grey areas in any policy.
>

If core devs trust someone enough to let them act as a PEP sponsor, they
probably know at least one core dev well enough to get them to add their
name as sponsor on the condition that they are only responsiible for
ensuring their "proxy" (?) responds adequately and in a sufficiently timely
manner.

I don't have a great deal to add to most conversations here, but I would
urge all concerned to consider Brett's point, from my own now distant
experience as a PSF chair. If I may paraphrase him, it's easier to change
the rules when someone wants or needs to do something outside their current
scope than it is to devise bullet-proof rules. It was only after I learned
this lesson that much of the constitutional lawyering in the PSF was
(gradually) replaced by useful mission-directed volunteer-led activity.

Please don't concern yourselves too much about process, but instead focus
on the desired results. If lawyering is needed, delegate it to the PSF! I'd
rather have you pushing Python forward ;-).

Finally, thanks again to everyone who contributes, particularly for
managing to hide a great deal of Python's modern-day complexity from those
who neither want nor need to know about it. Ultimately I think that is
perhaps the biggest factor fuelling the language's continued growth.

Kind regards
Steve Holden
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190305/e878624a/attachment-0001.html>

From songofacandy at gmail.com  Wed Mar  6 00:21:05 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Wed, 6 Mar 2019 14:21:05 +0900
Subject: [Python-Dev] Summary of Python tracker Issues
In-Reply-To: <CACBhJdFi0xLrwFuC6Fiqz=V9nM67ku5=2c7WGGSPt-ptKE1dAA@mail.gmail.com>
References: <20190208180753.1.782F2F2B9AA3561B@roundup.psfhosted.org>
 <CAEfz+TwdqQm829Q_2RWJq9QVDsnRX53JkFYVMqFP0G+OSqJKiA@mail.gmail.com>
 <CAK256p3ruU-RCv-k_XeXSsu+33oKKnE29YK=bK=pQBJb=7tEgg@mail.gmail.com>
 <q59m60$1i0v$1@blaine.gmane.org>
 <CAK256p1kP_D9AYYfCbcgOXp5tb7oMHVbZ8x7JBi9yO9QsO4B0g@mail.gmail.com>
 <bb74cef5-2393-dccb-a6c5-5e491bbf66be@g.nevcal.com>
 <q59oo7$71ib$1@blaine.gmane.org>
 <5d12e862-5398-5701-2503-b745c0875976@g.nevcal.com>
 <q5ae7e$603g$1@blaine.gmane.org>
 <CACBhJdGqHTJCrZfGDtEnwBMs5JXTpjyNa04qiR5sOf4yb9OzbQ@mail.gmail.com>
 <CACBhJdFi0xLrwFuC6Fiqz=V9nM67ku5=2c7WGGSPt-ptKE1dAA@mail.gmail.com>
Message-ID: <CAEfz+TxQnv2kTKgAE1Mgy827xCEw0C=7eiAv3Cu3un2xVsgULA@mail.gmail.com>

Thank you for fixing it.Weekly status give me motivation to look issues.

-- 
Inada Naoki  <songofacandy at gmail.com>

From benjamin at python.org  Wed Mar  6 00:40:33 2019
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 06 Mar 2019 00:40:33 -0500
Subject: [Python-Dev] [RELEASE] Python 2.7.16
In-Reply-To: <12acd71f-c18b-ea72-1e43-df4c76aa7812@redhat.com>
References: <5cb021d0-bf2f-45aa-98f1-5f4bb77fb4a6@www.fastmail.com>
 <12acd71f-c18b-ea72-1e43-df4c76aa7812@redhat.com>
Message-ID: <3cdb64ff-afc1-43d3-bb69-b866386d3f87@www.fastmail.com>



On Tue, Mar 5, 2019, at 03:18, Miro Hron?ok wrote:
> On 04. 03. 19 4:30, Benjamin Peterson wrote:
> > Hello all,
> > I'm pleased to announce the immediate availability of Python 2.7.16 for download at https://www.python.org/downloads/release/python-2716/.
> > 
> > The only change since the release candidate was a fix for the IDLE icon on macOS. See https://bugs.python.org/issue32129. Refer to the changelog for a full list of changes: https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16rc1.rst
> 
> 
> https://www.python.org/downloads/release/python-2716/ links changelog to 
> https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16.rst 
> but that only has 1 change (I suppose against rc1).
> 
> Is there a better link, or should I read those two combined?
> 
> https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16rc1.rst
> https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16.rst

Correct, those the full delta between 2.7.15 and 2.7.16.

From brett at snarky.ca  Wed Mar  6 12:56:23 2019
From: brett at snarky.ca (Brett Cannon)
Date: Wed, 6 Mar 2019 09:56:23 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
Message-ID: <CAP1=2W4i73qwnZo9BqhQ27M=ZioG=ddOfiHLGKfp87CMyvQyVw@mail.gmail.com>

Just to clarify a key point here which is covered in PEP 1 but based on
offline conversations is being missed, the need for a sponsor only kicks in
for committing it to the peps repo which typically kicks in when
transitioning from python-ideas to python-dev (although if one gets a
sponsor sooner then great as getting mentoring on how to handle the process
is always beneficial) . This should not be a barrier to presenting an idea
or writing up a proto-PEP for python-ideas if people are too shy to ask
upfront for a sponsor until their idea shows merit (and remember that we
already ask people to fork the peps repo and get feedback on their
proto-PEPs in their own fork instead of in the official repo). Nor should
this impact slow-burning ideas which people have to warm up to as that
typically happens before a PEP is written anyway.

I also don't see this preventing PEPs written to be explicitly rejected
either. I personally am happy to sponsor such PEPs as I'm sure several
other core devs are as well.


On Mon, Mar 4, 2019 at 5:43 PM Brett Cannon <brett at snarky.ca> wrote:

> The steering council has implemented a new idea called sponsors to the PEP
> process (added in
> https://github.com/python/peps/commit/c58d32c33bd06eb386d3f33963a1434510528f68).
> The thinking is that to help make sure PEPs from non-core developers
> receive appropriate guidance through the PEP process, a core developer
> needs to sign on to be a sponsor of the PEP. Being a sponsor does *not*
> preclude the core dev from eventually becoming a co-author or BDFL-delegate
> later on (but obviously not both), but the expectation is the sponsor is
> supportive of the idea (because if a single core dev won't sign on to help
> then what chance does the PEP have of being accepted?).
>
> If this doesn't turn out well we can obviously revert this, but hopefully
> this will make things smoother for those who are new to the PEP process.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190306/deac9838/attachment.html>

From arj.python at gmail.com  Wed Mar  6 13:28:12 2019
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Wed, 6 Mar 2019 22:28:12 +0400
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
Message-ID: <CADrxXXkwamz3aR2f7mvyR1=V7vC3MGBXoY7qTAZCH2uYSCsGFw@mail.gmail.com>

As a non core-dev, my enthusiasm for submitting a pep has been diminished
by some 40%.

yours,

Abdur-Rahmaan Janhangeer
http://www.pythonmembers.club | https://github.com/Abdur-rahmaanJ
Mauritius
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190306/815f42f5/attachment.html>

From barry at python.org  Wed Mar  6 14:32:25 2019
From: barry at python.org (Barry Warsaw)
Date: Wed, 6 Mar 2019 11:32:25 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CADrxXXkwamz3aR2f7mvyR1=V7vC3MGBXoY7qTAZCH2uYSCsGFw@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CADrxXXkwamz3aR2f7mvyR1=V7vC3MGBXoY7qTAZCH2uYSCsGFw@mail.gmail.com>
Message-ID: <71E9592A-EC9F-44F2-9999-D89E8D1EE85D@python.org>

On Mar 6, 2019, at 10:28, Abdur-Rahmaan Janhangeer <arj.python at gmail.com> wrote:
> 
> As a non core-dev, my enthusiasm for submitting a pep has been diminished by some 40%.

I?m sorry to hear that.

I think it?s worth keeping in mind that successfully navigating the PEP process can be quite daunting an demoralizing at times.  I wish that weren?t the case, and it was certainly not the intention when the PEP process was originally designed, but Python is a large diverse community now, and a mature, slow moving language, so it?s rare that a great idea makes it through without contention.

I think that the requirement to find a sponsor can help with that, regardless of whether the PEPable idea is ultimately accepted or rejected.  Core devs generally have a pretty good sense of what it takes to get a PEP through the process, can provide encouragement when the millithreads get you down or overwhelmed, and can help short circuit the disappointment, time and emotional investment when the idea has a poor chance of being accepted.  Many of our core devs have gone through it, with both outcomes, so their guidance can be very valuable.  I like to think of the sponsor as a PEP mentor, although sponsors can of course have a range of involvement.  Our core devs want you to succeed (or conversely save you from wasting your time), and they?ve been there, done that.

Core dev sponsors can be a great resource for folks new to the process.

Cheers,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190306/61eb3caa/attachment.sig>

From arj.python at gmail.com  Wed Mar  6 14:57:40 2019
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Wed, 6 Mar 2019 23:57:40 +0400
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <71E9592A-EC9F-44F2-9999-D89E8D1EE85D@python.org>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CADrxXXkwamz3aR2f7mvyR1=V7vC3MGBXoY7qTAZCH2uYSCsGFw@mail.gmail.com>
 <71E9592A-EC9F-44F2-9999-D89E8D1EE85D@python.org>
Message-ID: <CADrxXXmPdn_=Pc3fNNeWm8O9UaWx88Ruked4tEojyT+VVKw2fA@mail.gmail.com>

i think that "should have at least a mentor guiding you" sounds a lot more
better than

a core developer needs to sign on to be a sponsor

that sounds a lot more that without backing, you can't submit a pep, i
guess the core devs wanted to make things easier but the sponsor thing etc
put me off.

for someone using py, ideas sometimes come but since i've not yet submitted
a pep, when i see a change in the flow, i ask: will it be easier or more
difficult to submit peps now? i really got the impression that now chances
are slimmer.

great idea that core mentors now volunteer to guide peps,

sorry for trouble.

yours,

Abdur-Rahmaan Janhangeer
http://www.pythonmembers.club | https://github.com/Abdur-rahmaanJ
Mauritius
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190306/dcb64b73/attachment.html>

From g.rodola at gmail.com  Wed Mar  6 18:58:26 2019
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Thu, 7 Mar 2019 00:58:26 +0100
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CADrxXXmPdn_=Pc3fNNeWm8O9UaWx88Ruked4tEojyT+VVKw2fA@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <CADrxXXkwamz3aR2f7mvyR1=V7vC3MGBXoY7qTAZCH2uYSCsGFw@mail.gmail.com>
 <71E9592A-EC9F-44F2-9999-D89E8D1EE85D@python.org>
 <CADrxXXmPdn_=Pc3fNNeWm8O9UaWx88Ruked4tEojyT+VVKw2fA@mail.gmail.com>
Message-ID: <CAFYqXL_A2C5nm5BNTfNoW7LxY-WiJHsCYiTuMf2AyYrOB9oa_A@mail.gmail.com>

On Wed, Mar 6, 2019 at 8:58 PM Abdur-Rahmaan Janhangeer <
arj.python at gmail.com> wrote:

> i think that "should have at least a mentor guiding you" sounds a lot more
> better than
>
> a core developer needs to sign on to be a sponsor
>
> that sounds a lot more that without backing, you can't submit a pep, i
> guess the core devs wanted to make things easier but the sponsor thing etc
> put me off.
> for someone using py, ideas sometimes come but since i've not yet
> submitted a pep, when i see a change in the flow, i ask: will it be easier
> or more difficult to submit peps now? i really got the impression that now
> chances are slimmer.
>

Before submitting a PEP one (including core-devs) usually starts a
discussion on python-dev/ideas in order to start collecting feedback and,
most importantly, to figure out whether the idea deserves a PEP or not
(often times it doesn't). If the proposal is good it means somebody agreed
with you: those persons are likely gonna be the ones who'll likely sponsor
your PEP. If you can't find such a person immediately and the idea received
positive feedback I imagine you'll just be asked to write a proto-PEP
first, and if that is good enough somebody will eventually sponsor it and
possibly even help you. If you can't find any person then it probably means
it wasn't such a good idea, and that's also good because it will save you
from the trouble of writing the PEP in the first place (meeting the PEP
quality standards is not exactly a piece of cake). I doubt we'll end up in
a situation where a good proposal won't happen just because nobody is
willing to sponsor it.

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/2ecf456d/attachment.html>

From songofacandy at gmail.com  Thu Mar  7 04:18:27 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Thu, 7 Mar 2019 18:18:27 +0900
Subject: [Python-Dev] configparser: should optionxform be idempotent?
Message-ID: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>

Hi, all.

I came from https://bugs.python.org/issue35838
Since there are no "expert" for configparser in
Expert Index, I ask here to make design decision.

The default behavior of CofigParser.optionxform
is str.lowercase().  This is used to canonicalize
option key names.

The document of the optionxform shows example
overrides it to identity function `lambda option: option`.
https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.optionxform

BPO-35838 is issue about optionxform can be called twice
while ConfigParser.read_dict().
If optionxfrom is not idempotent, it creates unexpected option
name.
https://bugs.python.org/issue35838#msg334439

But even if all APIs calls optionxform exactly once, user may
read option name and value, and write updated value with same name.
In this case, user read option name already optionxform-ed
(canonicalized).  So non-idempotent optionxform will break
option name.

So what should we do about optionxform?

a)  Document "optionxform must be idempotent".

b) Ensure all APIs calls optionxform exactly once, and document
   "When you get option name from section objects, it is already
    optionxform-ed.  You can not reuse the option name if
    optionxform is not idempotent, because optionxform will be
    applied to the name again."

I prefer (a) to (b) because it's simple and easy solution.

But for some use cases (e.g. read only, write only, use only
predefined option name and read only it's value), (b) works.
At least issue reporter try this use case and be trapped by
this behavior.

How do you think?

-- 
Inada Naoki  <songofacandy at gmail.com>

From p.f.moore at gmail.com  Thu Mar  7 04:56:49 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 7 Mar 2019 09:56:49 +0000
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
Message-ID: <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>

On Thu, 7 Mar 2019 at 09:21, Inada Naoki <songofacandy at gmail.com> wrote:
> The document of the optionxform shows example
> overrides it to identity function `lambda option: option`.
> https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.optionxform
>
> BPO-35838 is issue about optionxform can be called twice
> while ConfigParser.read_dict().
> If optionxfrom is not idempotent, it creates unexpected option
> name.
> https://bugs.python.org/issue35838#msg334439

I'm not keen on the term "idempotent" here - I wasn't at all clear
what it was intended to convey. But from looking at the bug report, I
see that it basically means "optionxform should be a function which,
when applied more than one time to a value, returns the same result as
if it had been applied once only".

> But even if all APIs calls optionxform exactly once, user may
> read option name and value, and write updated value with same name.
> In this case, user read option name already optionxform-ed
> (canonicalized).  So non-idempotent optionxform will break
> option name.
>
> So what should we do about optionxform?
>
> a)  Document "optionxform must be idempotent".
>
> b) Ensure all APIs calls optionxform exactly once, and document
>    "When you get option name from section objects, it is already
>     optionxform-ed.  You can not reuse the option name if
>     optionxform is not idempotent, because optionxform will be
>     applied to the name again."
>
> I prefer (a) to (b) because it's simple and easy solution.

I strongly prefer (b). I think the example given in the bug report is
a reasonable thing to expect to work. I think that disallowing this
usage is an arbitrary restriction that honestly doesn't have a good
justification *other* than "it's easier for the implementation". It's
obviously not a *common* requirement, otherwise the issue would have
come up more often, but it's a reasonable one (after all, we don't
require similar functions like the key argument to sorted() to conform
to this restriction).

I'd look at the question the other way round. If we *did* insist that
optionxform has to be "idempotent", how would we recommend that the
person who reported the bug achieved the result he's trying to get?
lambda x: x if x.startswith("(") and x.endswith(")") else "(" + x +
")"? That seems a bit fiddly.

If, however, the consensus is that we choose (a), can I ask that we
*don't* use the term "idempotent" when documenting the restriction? I
think it will cause too much confusion - we should explain the
restriction without using obscure terms (and if it's hard to explain
the restriction like that, maybe that demonstrates that it's an
unreasonable restriction to impose? ;-))

Paul

From songofacandy at gmail.com  Thu Mar  7 05:06:31 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Thu, 7 Mar 2019 19:06:31 +0900
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
Message-ID: <CAEfz+Ty2y0+3ZBTB=MekQaBEgcUhtHyEFk=_cTbSzrBintio4A@mail.gmail.com>

On Thu, Mar 7, 2019 at 6:57 PM Paul Moore <p.f.moore at gmail.com> wrote:
>
> I'm not keen on the term "idempotent" here - I wasn't at all clear
> what it was intended to convey. But from looking at the bug report, I
> see that it basically means "optionxform should be a function which,
> when applied more than one time to a value, returns the same result as
> if it had been applied once only".

You're right.  "idempotent" is technical (or mathematical) jargon.
When f(x) satisfies "f(x) == f(f(x)) for all x" restriction, f(x) is idempotent.


>
> I'd look at the question the other way round. If we *did* insist that
> optionxform has to be "idempotent", how would we recommend that the
> person who reported the bug achieved the result he's trying to get?
> lambda x: x if x.startswith("(") and x.endswith(")") else "(" + x +
> ")"? That seems a bit fiddly.

In this case, we recommend not using optionxform to wrap name with
"()" implicitly.  Use wrapped name explicitly instead.

e.g. cfg["section"]["(name)"] = "value"

It's very simple.

-- 
Inada Naoki  <songofacandy at gmail.com>

From tir.karthi at gmail.com  Thu Mar  7 05:30:02 2019
From: tir.karthi at gmail.com (Karthikeyan)
Date: Thu, 7 Mar 2019 16:00:02 +0530
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
Message-ID: <CAAjsFLQcQyyLYa6NA34k3Xjuyk6WmrGmTb5RAQK-=nref3ndsA@mail.gmail.com>

On Thu, Mar 7, 2019 at 2:51 PM Inada Naoki <songofacandy at gmail.com> wrote:

> Hi, all.
>
> I came from https://bugs.python.org/issue35838
> Since there are no "expert" for configparser in
> Expert Index, I ask here to make design decision.
>

There is lukasz.langa in the expert index for configparser at
https://devguide.python.org/experts/#stdlib and that's why I deferred to
them.

The default behavior of CofigParser.optionxform
> is str.lowercase().  This is used to canonicalize
> option key names.
>
> The document of the optionxform shows example
> overrides it to identity function `lambda option: option`.
>
> https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.optionxform
>
> BPO-35838 is issue about optionxform can be called twice
> while ConfigParser.read_dict().
> If optionxfrom is not idempotent, it creates unexpected option
> name.
> https://bugs.python.org/issue35838#msg334439
>
> But even if all APIs calls optionxform exactly once, user may
> read option name and value, and write updated value with same name.
> In this case, user read option name already optionxform-ed
> (canonicalized).  So non-idempotent optionxform will break
> option name.
>
> So what should we do about optionxform?
>
> a)  Document "optionxform must be idempotent".
>

I also feel this is restrictive since wrapping keys with () looks like a
valid use case to me.

b) Ensure all APIs calls optionxform exactly once, and document
>    "When you get option name from section objects, it is already
>     optionxform-ed.  You can not reuse the option name if
>     optionxform is not idempotent, because optionxform will be
>     applied to the name again."
>
> I prefer (a) to (b) because it's simple and easy solution.
>

I initially preferred (b) while read_dict is one case. As you have
mentioned in the tracker there are various scenarios where the transform is
done and stored in the underlying internal dict and then while setting one
section key to another it might apply it again. Also I am afraid there is
less test coverage for optionxform itself so there could be more scenarios
to cover increasing the complexity.

-- 
Regards,
Karthikeyan S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/19cc428d/attachment.html>

From p.f.moore at gmail.com  Thu Mar  7 05:30:10 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 7 Mar 2019 10:30:10 +0000
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CAEfz+Ty2y0+3ZBTB=MekQaBEgcUhtHyEFk=_cTbSzrBintio4A@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <CAEfz+Ty2y0+3ZBTB=MekQaBEgcUhtHyEFk=_cTbSzrBintio4A@mail.gmail.com>
Message-ID: <CACac1F8EC42SCCgLac3F0Ytf-vOzJJ5hTxOu9uh0ZdykVtcq_Q@mail.gmail.com>

On Thu, 7 Mar 2019 at 10:06, Inada Naoki <songofacandy at gmail.com> wrote:
>
> On Thu, Mar 7, 2019 at 6:57 PM Paul Moore <p.f.moore at gmail.com> wrote:
> >
> > I'm not keen on the term "idempotent" here - I wasn't at all clear
> > what it was intended to convey. But from looking at the bug report, I
> > see that it basically means "optionxform should be a function which,
> > when applied more than one time to a value, returns the same result as
> > if it had been applied once only".
>
> You're right.  "idempotent" is technical (or mathematical) jargon.
> When f(x) satisfies "f(x) == f(f(x)) for all x" restriction, f(x) is idempotent.

Thanks. I know what the term means, at least in a mathematical sense -
the computing sense is slightly different (in a subtle way that may
not be relevant here - see
https://stackoverflow.com/questions/1077412/what-is-an-idempotent-operation).

> > I'd look at the question the other way round. If we *did* insist that
> > optionxform has to be "idempotent", how would we recommend that the
> > person who reported the bug achieved the result he's trying to get?
> > lambda x: x if x.startswith("(") and x.endswith(")") else "(" + x +
> > ")"? That seems a bit fiddly.
>
> In this case, we recommend not using optionxform to wrap name with
> "()" implicitly.  Use wrapped name explicitly instead.
>
> e.g. cfg["section"]["(name)"] = "value"
>
> It's very simple.

That argument could be used for any use of optionxform, though -
instead of using the default optionxform, use explicitly-lowercased
values everywhere instead.

I still prefer option (b), allowing general functions for optionxform.
However, I will say (and I should have said in my first mail) that
this is a view based purely on theoretical considerations. I've never
explicitly used optionxform myself, and none of my code would be
impacted in any way regardless of the outcome of this discussion.

Paul

From songofacandy at gmail.com  Thu Mar  7 05:41:21 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Thu, 7 Mar 2019 19:41:21 +0900
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CACac1F8EC42SCCgLac3F0Ytf-vOzJJ5hTxOu9uh0ZdykVtcq_Q@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <CAEfz+Ty2y0+3ZBTB=MekQaBEgcUhtHyEFk=_cTbSzrBintio4A@mail.gmail.com>
 <CACac1F8EC42SCCgLac3F0Ytf-vOzJJ5hTxOu9uh0ZdykVtcq_Q@mail.gmail.com>
Message-ID: <CAEfz+TyzDCrWuDrSZK9_dB4pcxqmuyLxTfC9ECScp-zK34TNgw@mail.gmail.com>

>
> That argument could be used for any use of optionxform, though -
> instead of using the default optionxform, use explicitly-lowercased
> values everywhere instead.
>

It can't be usable if the config format case-insensitive.

value = (cfg.get("section", "name") or cfg.get("section", "Name")
        or cfg.get("section", "nAme") or cfg.get("section", "naMe")...)


> I still prefer option (b), allowing general functions for optionxform.
> However, I will say (and I should have said in my first mail) that
> this is a view based purely on theoretical considerations. I've never
> explicitly used optionxform myself, and none of my code would be
> impacted in any way regardless of the outcome of this discussion.
>
> Paul

If we choose (b), I think core developer must check test coverage for
optionxform before documenting non-idempotent optionxform
is allowed explicitly.

I don't have motivation for that because I never used configparser in such way.

The PR looks good to me for the particular case the issue describe.
So I will merge the PR without updating document when we chose (b).

But let's wait a few days for other comments.

Regards,

-- 
Inada Naoki  <songofacandy at gmail.com>

From steve at pearwood.info  Thu Mar  7 07:39:50 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 7 Mar 2019 23:39:50 +1100
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
Message-ID: <20190307123950.GL4465@ando.pearwood.info>

On Thu, Mar 07, 2019 at 09:56:49AM +0000, Paul Moore wrote:
> On Thu, 7 Mar 2019 at 09:21, Inada Naoki <songofacandy at gmail.com> wrote:
> > The document of the optionxform shows example
> > overrides it to identity function `lambda option: option`.
> > https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.optionxform
> >
> > BPO-35838 is issue about optionxform can be called twice
> > while ConfigParser.read_dict().
> > If optionxfrom is not idempotent, it creates unexpected option
> > name.
> > https://bugs.python.org/issue35838#msg334439
> 
> I'm not keen on the term "idempotent" here - I wasn't at all clear
> what it was intended to convey. But from looking at the bug report, I
> see that it basically means "optionxform should be a function which,
> when applied more than one time to a value, returns the same result as
> if it had been applied once only".

That's what "idempotent" means :-)


[...]
> > So what should we do about optionxform?
> >
> > a)  Document "optionxform must be idempotent".
> >
> > b) Ensure all APIs calls optionxform exactly once, and document
> >    "When you get option name from section objects, it is already
> >     optionxform-ed.  You can not reuse the option name if
> >     optionxform is not idempotent, because optionxform will be
> >     applied to the name again."
> >
> > I prefer (a) to (b) because it's simple and easy solution.
> 
> I strongly prefer (b).

I don't have a strong opinion, but I have a mild preference for taking 
responsibility for idempotency out of the user's hands if practical.


[...]
> I'd look at the question the other way round. If we *did* insist that
> optionxform has to be "idempotent", how would we recommend that the
> person who reported the bug achieved the result he's trying to get?
> lambda x: x if x.startswith("(") and x.endswith(")") else "(" + x +
> ")"? That seems a bit fiddly.

Writing idempotent functions often is.

 
> If, however, the consensus is that we choose (a), can I ask that we
> *don't* use the term "idempotent" when documenting the restriction?

Why use one word when twenty-four will do? *wink*


> I think it will cause too much confusion - we should explain the
> restriction without using obscure terms (and if it's hard to explain
> the restriction like that, maybe that demonstrates that it's an
> unreasonable restriction to impose? ;-))

Please, idempotent is a standard term of art, especially for those 
working with RESTful interfaces.

http://restcookbook.com/HTTP%20Methods/idempotency/

It might be obscure to you, but then nearly every jargon term will be 
obscure to somebody. Nobody is born knowing what terms like

    multiprocessing
    threading
    metaclass
    decorator
    comprehension
    futures

etc mean. They're all "obscure jargon" terms to someone. The first time 
I came across "tuple", I had no idea what it meant (and in fact it took 
me many years to stop misspelling it "turple").

By all means include a definition of idempotent (perhaps a link to the 
glossary). But we shouldn't avoid useful, precise terminology because 
some people haven't come across it yet.



-- 
Steven

From songofacandy at gmail.com  Thu Mar  7 08:26:48 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Thu, 7 Mar 2019 22:26:48 +0900
Subject: [Python-Dev] Compact ordered set
In-Reply-To: <FC1A9CE1-0353-41A2-81A1-3D48E03292E9@gmail.com>
References: <CAEfz+TwBihOdZRBY8_F230fDbhB-3chReMoG8dQCE66=zZmbeQ@mail.gmail.com>
 <A1B3DF9E-9220-44E2-A9CB-9D7014A61767@gmail.com>
 <FC1A9CE1-0353-41A2-81A1-3D48E03292E9@gmail.com>
Message-ID: <CAEfz+TwT2Ft2i_uA_heinkwHoGvb-Y6KyReFDpb_MoUT3HTgvg@mail.gmail.com>

ordered set 2

Hi, Raymond.

Thank you for your detailed response and I'm sorry about the late reply.

All of your points make sense to me.  My implementation has not been
battle-tested yet.

As I wrote in a previous mail, there is only one benchmark in
pyperformance was affected significantly.  (My implementation was 13%
faster.)

After that, I wrote some microbenchmarks.  Some run faster on the
current implementation and some run faster on the ordered
implementation.
https://github.com/methane/sandbox/tree/master/python/ordered-set

For example, current implementation is at most 20% faster on creating
new set by consecutive integers. (e.g. `set(range(N))`)   As you know,
it is because hash(int) is consecutive too.

On the other hand, ordered implementation is at most 17% faster on
creating new set by string values.

But I don't know how this microbenchmark results affects real world
set workloads.  Pyperformance doesn't have enough variations of set
worklods.

Would you please tell me how did you gather vary set workloads?


> * To get the same lookup performance, the density of index table would need to go down to around 25%. Otherwise, there's no way to make up for the extra indirection and the loss of cache locality.

Yes.  Currently, I chose capacity ratio=50%, and I removed 4X resize
on small sets.  So density is about 25~50% for now.  Performance of
simple lookup is 5~8% slower.

More small capacity ratio may improve performance a bit, but it tends
worse memory efficiency when table is 32bit or 64bit.

> * There was a small win on iteration performance because its cheaper to loop over a dense array than a sparse array (fewer memory access and elimination of the unpredictable branch).  This is nice because iteration performance matters in some key use cases.

Yes.  And it can be huge performance benefit on extreme cases.
(e.g. https://bugs.python.org/issue32846)

> * I gave up on ordering right away.  If we care about performance, keys can be stored in the order added; but no effort should be expended to maintain order if subsequent deletions occur.  Likewise, to keep set-to-set operations efficient (i.e. looping over the smaller input), no order guarantee should be given for those operations.  In general, we can let order happen but should not guarantee it and work to maintain it or slow-down essential operations to make them ordered.

I agree.  Maybe, set shouldn't guarantee preserving insertion order unlike dict.
It loses almost half of benefit of new implementation :-(
But order is stable in most cases anyway regardless hash randomization.
Stable pyc can be compiled without PYTHONHASHSEED=0, and sets in
logs are almost stable and readable.


> * Compacting does make sets a little smaller but does cost an indirection and incurs a cost for switching index sizes between 1-byte arrays, 2-byte arrays, 4-byte arrays, and 8-byte arrays.  Those don't seem very expensive; however, set lookups are already very cheap when the hash values are known (when they're not, the cost of computing the hash value tends to dominate anything done by the setobject itself).

Yes.  Ordered implementation makes simple case slower.  But memory
efficiency is very different if application uses tons of small sets.

> * I couldn't find any existing application that would notice the benefit of making sets a bit smaller.  Most applications use dictionaries (directly or indirectly) everywhere, so compaction was an overall win.  Sets tend to be used more sparsely (no pun intended) and tend to be only a small part of overall memory usage. I had to consider this when bumping the load factor down to 60%, prioritizing speed over space.

You're right.  Number of set objects in Python interpreter is very few
than dict.
And ordered set is not memory efficient for large set.

Maybe, we couldn't find clear net win by this set implementation.
I will stop this work at some point.

Thank you,

--
Inada Naoki  <songofacandy at gmail.com>

From p.f.moore at gmail.com  Thu Mar  7 09:39:23 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 7 Mar 2019 14:39:23 +0000
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <20190307123950.GL4465@ando.pearwood.info>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <20190307123950.GL4465@ando.pearwood.info>
Message-ID: <CACac1F9y6W8cbkyVLXf1M8UVxExDU=Z8Csvz50NVcF6xqhYu+g@mail.gmail.com>

On Thu, 7 Mar 2019 at 12:42, Steven D'Aprano <steve at pearwood.info> wrote:

> > I'm not keen on the term "idempotent" here - I wasn't at all clear
> > what it was intended to convey. But from looking at the bug report, I
> > see that it basically means "optionxform should be a function which,
> > when applied more than one time to a value, returns the same result as
> > if it had been applied once only".
>
> That's what "idempotent" means :-)

Sigh. I'm not interested in an extended debate on fiddly details here.

I know that's what "idempotent" means. I said I "wasn't clear", and
the context clarified it for me (as would going and looking it up,
which I *also* did).There's a subtle difference in the mathematical
and computing meanings (around functions with side-effects, which
aren't a thing in maths), which IMO makes the term *very slightly*
ambiguous - but more to the point, it's uncommon jargon in many areas
(searching for "idempotent" in Google shows enough examples of people
asking what it means to bear this out, IMO - feel free to disagree).

> > If, however, the consensus is that we choose (a), can I ask that we
> > *don't* use the term "idempotent" when documenting the restriction?
>
> Why use one word when twenty-four will do? *wink*

Why use possibly-misunderstood jargon when a clear description will
do? Hmm, let me think. I guess it depends on which
carefully-worded-to-make-your-point description of the situation you
choose to use. *wink*

> > I think it will cause too much confusion - we should explain the
> > restriction without using obscure terms (and if it's hard to explain
> > the restriction like that, maybe that demonstrates that it's an
> > unreasonable restriction to impose? ;-))
>
> Please, idempotent is a standard term of art, especially for those
> working with RESTful interfaces.
>
> http://restcookbook.com/HTTP%20Methods/idempotency/
>
> It might be obscure to you, but then nearly every jargon term will be
> obscure to somebody.

I didn't say otherwise. I said "I think it will cause too much
confusion". I stand by that. I value clear, non-technical, terms over
jargon when it's possible to express an idea that way without
sacrificing clarity. I believe that in this case this is possible
(although as I've already said, I think it's better to avoid the whole
question and *not* impose the restriction at all).

I have no idea what proportion of readers of the configparser docs
will be familiar with REST (or with maths, or with any other
context).I doubt you do either, but if you do I'll defer to your
knowledge.

> The first time
> I came across "tuple", I had no idea what it meant (and in fact it took
> me many years to stop misspelling it "turple").

I love that, I may start using it deliberately :-)

> By all means include a definition of idempotent (perhaps a link to the
> glossary). But we shouldn't avoid useful, precise terminology because
> some people haven't come across it yet.

Agreed, but we should also express ideas in a way that is as
accessible to the general reader as possible. Sometimes that means
using (and explaining) precise technical terms, other times it means
using simple language that gets the job done, without using
*unnecessary* jargon. It's a judgement call as to where the dividing
line lies. So I feel that "idempotent" is on one side of the line, and
you think it's on the other. We've expressed our opinions, let's leave
it at that - I don't want to get into an extended debate over "my
experience of what the average user thinks is wider than yours"...

Paul

From storchaka at gmail.com  Thu Mar  7 11:40:12 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 7 Mar 2019 18:40:12 +0200
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
Message-ID: <q5rhhg$63gc$1@blaine.gmane.org>

07.03.19 11:18, Inada Naoki ????:
> So what should we do about optionxform?
> 
> a)  Document "optionxform must be idempotent".
> 
> b) Ensure all APIs calls optionxform exactly once, and document
>     "When you get option name from section objects, it is already
>      optionxform-ed.  You can not reuse the option name if
>      optionxform is not idempotent, because optionxform will be
>      applied to the name again."
> 
> I prefer (a) to (b) because it's simple and easy solution.

I am not expert of configparser, but I prefer (a). The purpose of 
introducing optionxform was to make lowercasing optional.

https://github.com/python/cpython/commit/9e480adf9b3520ea3deb322fd1214f53a2293a0d


From sadhak001 at gmail.com  Thu Mar  7 12:24:48 2019
From: sadhak001 at gmail.com (Mani Sarkar)
Date: Thu, 7 Mar 2019 17:24:48 +0000
Subject: [Python-Dev] Loading modules from a folder
Message-ID: <CAGHtMW=DpdzH_S-AVqwcEqB__2k4cGiXKfrsPkseQ+PZaMLAhg@mail.gmail.com>

Hi,

I have seen multiple ways to load modules from a folder.

Say I have a folder src which contains a number of folders each one of them
is a module.

What are the conventional ways to load modules from such a folder?

I have used

>From src.[module] import xxxxx

But I don't want to use the src prefix, any other ways to indicate that in
python?

Thanks

Cheers,
Mani
-- 

@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  | *@adoptopenjdk
<http://twitter.com/adoptopenjdk> @graalvm <http://twitter.com/graalvm>
@graal <http://twitter.com/graal> @truffleruby
<http://twitter.com/truffleruby> | Dev. communities | *Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * | * *Slideshare
<https://slideshare.net/neomatrix369>* * | **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2019:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/82a35790/attachment.html>

From brett at python.org  Thu Mar  7 13:41:27 2019
From: brett at python.org (Brett Cannon)
Date: Thu, 7 Mar 2019 10:41:27 -0800
Subject: [Python-Dev] Loading modules from a folder
In-Reply-To: <CAGHtMW=DpdzH_S-AVqwcEqB__2k4cGiXKfrsPkseQ+PZaMLAhg@mail.gmail.com>
References: <CAGHtMW=DpdzH_S-AVqwcEqB__2k4cGiXKfrsPkseQ+PZaMLAhg@mail.gmail.com>
Message-ID: <CAP1=2W7JbhY0OyAANBNs7ar7WOm+_5hNnVWZbT6yqKub0VHXpw@mail.gmail.com>

This mailing list is actually for the development *of* Python, not *with*
Python. You can try asking your question on Stack Overflow, python tutor,
or python-list.

On Thu, Mar 7, 2019 at 9:28 AM Mani Sarkar <sadhak001 at gmail.com> wrote:

> Hi,
>
> I have seen multiple ways to load modules from a folder.
>
> Say I have a folder src which contains a number of folders each one of
> them is a module.
>
> What are the conventional ways to load modules from such a folder?
>
> I have used
>
> From src.[module] import xxxxx
>
> But I don't want to use the src prefix, any other ways to indicate that in
> python?
>
> Thanks
>
> Cheers,
> Mani
> --
>
> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
> <http://neomatrix369.wordpress.com/>**  | *@adoptopenjdk
> <http://twitter.com/adoptopenjdk> @graalvm <http://twitter.com/graalvm>
> @graal <http://twitter.com/graal> @truffleruby
> <http://twitter.com/truffleruby> | Dev. communities | *Bitbucket
> <https://bitbucket.org/neomatrix369>* * |  **Github
> <https://github.com/neomatrix369>* * | * *Slideshare
> <https://slideshare.net/neomatrix369>* * | **LinkedIn
> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
> *Come to Devoxx UK 2019:* http://www.devoxx.co.uk/
>
> *Don't chase success, rather aim for "Excellence", and success will come
> chasing after you!*
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/8b8ff747/attachment.html>

From mariatta at python.org  Thu Mar  7 13:49:50 2019
From: mariatta at python.org (Mariatta)
Date: Thu, 7 Mar 2019 10:49:50 -0800
Subject: [Python-Dev] Using CLA assistant for Python contributions
Message-ID: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>

I've posted in Discourse under core-workflow category
<https://discuss.python.org/t/using-cla-assistant-for-python/990>, and this
has been previously discussed on the core-workflow mailing list
<https://mail.python.org/archives/list/core-workflow at python.org/thread/JBV3XJVD2DLDX5DY7TZEA6CO5YPNHJ2C/>,
but I feel this affects the wider contributors to Python, so wanted to
share it here for more visibility.

TL;DR:

We'd like to start using CLA assistant for contributions to Python
(including CPython, devguide, PEPs, all the bots etc). Ernest had set up
our own instance of CLA assistant, and it had been tested by several core
developers. We've also consulted The PSF and Van Lindberg for legal advice.

Unless I hear strong opposition (with reasons) from Python Steering
Council, Python core developers, and active core contributors, I plan to
switch us over to to CLA assistant in the coming week (before my OOOS of
March 18)

How this will affect all contributors to Python old and new:

- you will need to sign the CLA again, even if you've signed it before (in
bpo). It will take you several clicks, but then you'll do this only once,
and it takes effect immediately. (instead of waiting for a PSF staff to
check for it)

- bpo username will no longer be required when signing the CLA

- CLA will be accepted under Apache v2 only (no more Academic Free license)

For even more details, please follow the discourse post and the
core-workflow mailing list linked above, as well as the "CLA" section of my
blog post about Core Python sprint 2018
<https://mariatta.ca/core-sprint-2018-part-2.html#core-sprint-2018-part-2>.

Thanks.
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/e46f7e04/attachment.html>

From mariatta at python.org  Thu Mar  7 14:08:50 2019
From: mariatta at python.org (Mariatta)
Date: Thu, 7 Mar 2019 11:08:50 -0800
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
Message-ID: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>

I'd like to formally present to Python-dev PEP 581: Using GitHub Issues for
CPython

Full text: https://www.python.org/dev/peps/pep-0581/

This is my first PEP, and in my opinion it is ready for wider discussion. I
don't know if it is "ready for pronouncement" so I'm hoping to get more
guidance about it from other experienced core devs and steering council.

I also plan to open discussion about PEP 581 at Python Language Summit, and
since I'm one-half of the language summit chairs, it is quite likely this
discussion will happen.

On that note, you can still sign up for the language summit here:
https://us.pycon.org/2019/events/language-summit/

Note that unlike previous years, you do not need to be invited by a core
developer. ?ukasz and I will be curating the content, and we want to
include more diverse perspectives into language summit discussions.

Thanks.

?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/e2783988/attachment.html>

From sadhak001 at gmail.com  Thu Mar  7 15:27:49 2019
From: sadhak001 at gmail.com (Mani Sarkar)
Date: Thu, 7 Mar 2019 20:27:49 +0000
Subject: [Python-Dev] Loading modules from a folder
In-Reply-To: <CAP1=2W7JbhY0OyAANBNs7ar7WOm+_5hNnVWZbT6yqKub0VHXpw@mail.gmail.com>
References: <CAGHtMW=DpdzH_S-AVqwcEqB__2k4cGiXKfrsPkseQ+PZaMLAhg@mail.gmail.com>
 <CAP1=2W7JbhY0OyAANBNs7ar7WOm+_5hNnVWZbT6yqKub0VHXpw@mail.gmail.com>
Message-ID: <CAGHtMWmpsQvRW4NS8fMind8XdMw+G2oLyJ4QbuV2OUvd0xgTJg@mail.gmail.com>

Cool, thanks!

My apologies - i'll post the question there instead.

On Thu, 7 Mar 2019 at 18:41 Brett Cannon <brett at python.org> wrote:

> This mailing list is actually for the development *of* Python, not *with*
> Python. You can try asking your question on Stack Overflow, python tutor,
> or python-list.
>
> On Thu, Mar 7, 2019 at 9:28 AM Mani Sarkar <sadhak001 at gmail.com> wrote:
>
>> Hi,
>>
>> I have seen multiple ways to load modules from a folder.
>>
>> Say I have a folder src which contains a number of folders each one of
>> them is a module.
>>
>> What are the conventional ways to load modules from such a folder?
>>
>> I have used
>>
>> From src.[module] import xxxxx
>>
>> But I don't want to use the src prefix, any other ways to indicate that
>> in python?
>>
>> Thanks
>>
>> Cheers,
>> Mani
>> --
>>
>> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
>> <http://neomatrix369.wordpress.com/>**  | *@adoptopenjdk
>> <http://twitter.com/adoptopenjdk> @graalvm <http://twitter.com/graalvm>
>> @graal <http://twitter.com/graal> @truffleruby
>> <http://twitter.com/truffleruby> | Dev. communities | *Bitbucket
>> <https://bitbucket.org/neomatrix369>* * |  **Github
>> <https://github.com/neomatrix369>* * | * *Slideshare
>> <https://slideshare.net/neomatrix369>* * | **LinkedIn
>> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
>> *Come to Devoxx UK 2019:* http://www.devoxx.co.uk/
>>
>> *Don't chase success, rather aim for "Excellence", and success will come
>> chasing after you!*
>>
> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> --

@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  | *@adoptopenjdk
<http://twitter.com/adoptopenjdk> @graalvm <http://twitter.com/graalvm>
@graal <http://twitter.com/graal> @truffleruby
<http://twitter.com/truffleruby> | Dev. communities | *Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * | * *Slideshare
<https://slideshare.net/neomatrix369>* * | **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2019:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/b6631977/attachment.html>

From matthew at woodcraft.me.uk  Thu Mar  7 14:56:41 2019
From: matthew at woodcraft.me.uk (Matthew Woodcraft)
Date: Thu, 07 Mar 2019 19:56:41 +0000
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
Message-ID: <1551988232.oqg8xuvpxk.astroid@golux.woodcraft.me.uk>

On 07/03/2019 19.08, Mariatta wrote:
> I'd like to formally present to Python-dev PEP 581: Using GitHub Issues
> for CPython
>
> Full text: https://www.python.org/dev/peps/pep-0581/
>
> This is my first PEP, and in my opinion it is ready for wider
> discussion.

One part of this PEP stands out to me:

| We should not be moving all open issues to GitHub. Issues with little
| or no activity should just be closed. Issues with no decision made for
| years should just be closed.

I strongly advise against closing bug reports just because they're old.

I know that the Python developers value trying to be a welcoming
community. To many people, having a bug report that they put some effort
into closed for no reason other than the passage of time feels like a
slap in the face which stings harder than, for example, intemperate
words on a mailing list.

This is even more true if there won't be an option to re-open the bug,
which seems to be what the PEP is saying will be the case.

If a bug has been around for a long time and hasn't been fixed, the most
useful information for the bug tracker to contain is "this bug has been
around for a long time and it hasn't been fixed". Leaving the bug open
is the simplest way to achieve that.

(I think the above only goes for issues which are actually reporting
bugs. Wishlist items are a different matter.)

-M-



From skip.montanaro at gmail.com  Thu Mar  7 17:01:55 2019
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 7 Mar 2019 16:01:55 -0600
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>
References: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>
Message-ID: <CANc-5UwodQb0uLPA1viNk4wx5hvvmKbHOOGjejY6Peh8a_5syg@mail.gmail.com>

> I'd like to formally present to Python-dev PEP 581: Using GitHub Issues for CPython
>
> Full text: https://www.python.org/dev/peps/pep-0581/

Thanks for doing this. I think there is a pretty strong argument to be
made that mature, widely adopted systems like GitHub (or GitLab or
Bitbucket) should be used where possible. One thing I didn't notice
was any sort of explanation about how CPython wound up on Roundup to
begin with. I think it would be worthwhile to mention a couple
reasons, when the decision was made to use Roundup, etc. Without it, a
casual reader might think the core devs made a horrible mistake way
back when, and are only now getting around to correcting it. I don't
recall when Roundup was adopted, but it was quite awhile ago, and the
issue tracker universe was a much different place. Here are a couple
things I recall (perhaps incorrectly). It's been awhile, but for the
digital spelunkers, I'm sure it's all in an email archive somewhere.
(I didn't find a PEP. Did that decision predate PEPs?)

* Back in the olden days, basically every candidate issue tracker
required modification to make it suitable for a particular project. I
don't rightly recall if Roundup was deemed easier to modify or if
there were just people willing to step up and make the necessary
changes.

* There was a desire to eat your own dog food, and I believe Roundup
is/was written in Python. That would be much less important today.
Plenty of people already eat Python brand Dog Food.?

Skip

From mariatta at python.org  Thu Mar  7 17:11:49 2019
From: mariatta at python.org (Mariatta)
Date: Thu, 7 Mar 2019 14:11:49 -0800
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <1551988232.oqg8xuvpxk.astroid@golux.woodcraft.me.uk>
References: <1551988232.oqg8xuvpxk.astroid@golux.woodcraft.me.uk>
Message-ID: <CAGbohnbjzWaaMa3z_Sj=48HOD0YBSYyZB5eNKiUO8CZ7tV9oxw@mail.gmail.com>

On Thu, Mar 7, 2019 at 12:35 PM Matthew Woodcraft <matthew at woodcraft.me.uk>
wrote:

>
> One part of this PEP stands out to me:
>
> | We should not be moving all open issues to GitHub. Issues with little
> | or no activity should just be closed. Issues with no decision made for
> | years should just be closed.
>
> I strongly advise against closing bug reports just because they're old.
>
> I know that the Python developers value trying to be a welcoming
> community. To many people, having a bug report that they put some effort
> into closed for no reason other than the passage of time feels like a
> slap in the face which stings harder than, for example, intemperate
> words on a mailing list.
>
> This is even more true if there won't be an option to re-open the bug,
> which seems to be what the PEP is saying will be the case.
>
> If a bug has been around for a long time and hasn't been fixed, the most
> useful information for the bug tracker to contain is "this bug has been
> around for a long time and it hasn't been fixed". Leaving the bug open
> is the simplest way to achieve that.
>
> (I think the above only goes for issues which are actually reporting
> bugs. Wishlist items are a different matter.)
>
> -M-
>


Thanks. A similar argument had been made by several other core devs in
person during Language summit 2018 as well as during the drafting of PEP
581, that we shouldn't be closing issues blindly.

I hear you (all) and I see the point. I agree that we shouldn't just be
closing all issues. I will edit the PEP sometime later today (or later this
weekend).


?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/762e2e35/attachment.html>

From mariatta at python.org  Thu Mar  7 17:16:36 2019
From: mariatta at python.org (Mariatta)
Date: Thu, 7 Mar 2019 14:16:36 -0800
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <CA+VECoVi63oLbe7rdF_S1-P0rpUyQpKsObYqRGOBJD7W_mGZEg@mail.gmail.com>
References: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>
 <CA+VECoVi63oLbe7rdF_S1-P0rpUyQpKsObYqRGOBJD7W_mGZEg@mail.gmail.com>
Message-ID: <CAGbohnZQk4Voeb0xW9gUuHQS3j-bhvqqq+JJ_KJ_b=aXC4k=3Q@mail.gmail.com>

On Thu, Mar 7, 2019 at 12:36 PM Manuel Cer?n <ceronman at gmail.com> wrote:

>
> After some frustration with bpo, I decided to file some issues into the
> meta tracker, just to find out that the link [1] provided by the Python
> Developer's Guide [2] is broken, giving a connection timeout.
>
>
Sometime ago we've started experimenting moving the meta tracker to GitHub.
https://github.com/python/bugs.python.org I don't know whether this is now
the "official" place for it, but I've definitely been referring people to
this repo if they need to file issue about bpo.

Again I don't know if this is now official or not, and should we start
updating all documentations accordingly?

?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/4cf1a916/attachment.html>

From mariatta at python.org  Thu Mar  7 17:36:11 2019
From: mariatta at python.org (Mariatta)
Date: Thu, 7 Mar 2019 14:36:11 -0800
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <CANc-5UwodQb0uLPA1viNk4wx5hvvmKbHOOGjejY6Peh8a_5syg@mail.gmail.com>
References: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>
 <CANc-5UwodQb0uLPA1viNk4wx5hvvmKbHOOGjejY6Peh8a_5syg@mail.gmail.com>
Message-ID: <CAGbohnbP0vZ-+RHV+B3gWGg5P-LOWrTtRBG38cRfNkbm8m+G2A@mail.gmail.com>

On Thu, Mar 7, 2019 at 2:02 PM Skip Montanaro <skip.montanaro at gmail.com>
wrote:

>  I think it would be worthwhile to mention a couple
> reasons, when the decision was made to use Roundup, etc. Without it, a
> casual reader might think the core devs made a horrible mistake way
> back when, and are only now getting around to correcting it.
>

I was not involved in core Python development back then, so if it is really
important and if people think such paragraph needs to be part of the PEP,
then perhaps someone else more knowledgeable will need to help with this.

Personally, I don't think it was a horrible mistake. I believe the core
devs back then carefully considered all options and decided that
bpo/roundup was the way to go. And I really don't want to give that
impression to the readers of this PEP that "I" or "core devs" now think it
was a horrible mistake. If there is specific parts of the PEP that gives
people that impression, then I'd definitely want to work and improve that.

?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/9205084b/attachment.html>

From greg at krypto.org  Thu Mar  7 17:51:01 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Thu, 7 Mar 2019 14:51:01 -0800
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <CAGbohnbjzWaaMa3z_Sj=48HOD0YBSYyZB5eNKiUO8CZ7tV9oxw@mail.gmail.com>
References: <1551988232.oqg8xuvpxk.astroid@golux.woodcraft.me.uk>
 <CAGbohnbjzWaaMa3z_Sj=48HOD0YBSYyZB5eNKiUO8CZ7tV9oxw@mail.gmail.com>
Message-ID: <CAGE7PNLMGuJgmYgsZH635YANK71RO6oohh7EgjFGo3UsRrouNw@mail.gmail.com>

On Thu, Mar 7, 2019 at 2:12 PM Mariatta <mariatta at python.org> wrote:

>
> On Thu, Mar 7, 2019 at 12:35 PM Matthew Woodcraft <matthew at woodcraft.me.uk>
> wrote:
>
>>
>> One part of this PEP stands out to me:
>>
>> | We should not be moving all open issues to GitHub. Issues with little
>> | or no activity should just be closed. Issues with no decision made for
>> | years should just be closed.
>>
>> I strongly advise against closing bug reports just because they're old.
>>
>> I know that the Python developers value trying to be a welcoming
>> community. To many people, having a bug report that they put some effort
>> into closed for no reason other than the passage of time feels like a
>> slap in the face which stings harder than, for example, intemperate
>> words on a mailing list.
>>
>> This is even more true if there won't be an option to re-open the bug,
>> which seems to be what the PEP is saying will be the case.
>>
>> If a bug has been around for a long time and hasn't been fixed, the most
>> useful information for the bug tracker to contain is "this bug has been
>> around for a long time and it hasn't been fixed". Leaving the bug open
>> is the simplest way to achieve that.
>>
>> (I think the above only goes for issues which are actually reporting
>> bugs. Wishlist items are a different matter.)
>>
>> -M-
>>
>
>
> Thanks. A similar argument had been made by several other core devs in
> person during Language summit 2018 as well as during the drafting of PEP
> 581, that we shouldn't be closing issues blindly.
>
> I hear you (all) and I see the point. I agree that we shouldn't just be
> closing all issues. I will edit the PEP sometime later today (or later this
> weekend).
>

yay thanks! :)

*(because I fixed this SourceForge issue number era
bug https://bugs.python.org/issue1054041
<https://bugs.python.org/issue1054041> in 3.8 :)*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/683658d1/attachment.html>

From mariatta at python.org  Thu Mar  7 18:34:01 2019
From: mariatta at python.org (Mariatta)
Date: Thu, 7 Mar 2019 15:34:01 -0800
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <CAGE7PNLMGuJgmYgsZH635YANK71RO6oohh7EgjFGo3UsRrouNw@mail.gmail.com>
References: <1551988232.oqg8xuvpxk.astroid@golux.woodcraft.me.uk>
 <CAGbohnbjzWaaMa3z_Sj=48HOD0YBSYyZB5eNKiUO8CZ7tV9oxw@mail.gmail.com>
 <CAGE7PNLMGuJgmYgsZH635YANK71RO6oohh7EgjFGo3UsRrouNw@mail.gmail.com>
Message-ID: <CAGbohnb+gx9_Ye-Ad9Qo68GWQCEiexYiv294nZxcgtxOsmREiQ@mail.gmail.com>

I've made the PR about "not closing all issues":
https://github.com/python/peps/pull/917/files


?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/ba8700e8/attachment.html>

From greg.ewing at canterbury.ac.nz  Thu Mar  7 18:34:46 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 08 Mar 2019 12:34:46 +1300
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
Message-ID: <5C81AA96.7020204@canterbury.ac.nz>

Inada Naoki wrote:

> a)  Document "optionxform must be idempotent".

> b) Ensure all APIs calls optionxform exactly once, and document
 >    [that it must be idempotent in certain special situations].

I think the question that needs to be asked is whether optionxform
is meant to be a canonicalising operation, or a transformation
from an external to an internal form.

Although the docs don't say so explicitly, it seems clear to me
that the author of the module was thinking of it as a transformation
to a canonical form. If that's what is intended, then idempotency
is pretty much implied, from the definition of what canonical
means. If something is already in canonical form, nothing needs
to be done.

The behaviour with regard to initialising from a dict that was
raised in https://bugs.python.org/issue35838#msg334439 is probably
an oversight, but there are at least two other ways that the
module assumes idempotency:

1) The API does not provide any way of accessing an option
*without* applying the transformation. This is a problem if e.g.
you want to iterate over the keys and write values back.

2) The write() method writes out the transformed versions of
option names. If this output is read back in, the transformation
will necessarily be applied a second time. There doesn't seem
to be any workaround for this.

There may also be other ways in which idempotency is assumed
that I haven't thought of.

So, rather than try to have the docs list all the things that
non-idempotency could break, it would be easier to simply
document that idempotency is assumed.

-- 
Greg

From barry at python.org  Thu Mar  7 18:35:42 2019
From: barry at python.org (Barry Warsaw)
Date: Thu, 7 Mar 2019 15:35:42 -0800
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <CAGbohnbP0vZ-+RHV+B3gWGg5P-LOWrTtRBG38cRfNkbm8m+G2A@mail.gmail.com>
References: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>
 <CANc-5UwodQb0uLPA1viNk4wx5hvvmKbHOOGjejY6Peh8a_5syg@mail.gmail.com>
 <CAGbohnbP0vZ-+RHV+B3gWGg5P-LOWrTtRBG38cRfNkbm8m+G2A@mail.gmail.com>
Message-ID: <4CF3BECF-9E75-4FAF-90AB-131034282956@python.org>

On Mar 7, 2019, at 14:36, Mariatta <mariatta at python.org> wrote:

> I was not involved in core Python development back then, so if it is really important and if people think such paragraph needs to be part of the PEP, then perhaps someone else more knowledgeable will need to help with this.
> 
> Personally, I don't think it was a horrible mistake. I believe the core devs back then carefully considered all options and decided that bpo/roundup was the way to go. And I really don't want to give that impression to the readers of this PEP that "I" or "core devs" now think it was a horrible mistake. If there is specific parts of the PEP that gives people that impression, then I'd definitely want to work and improve that.

I did a little bit of archive archeology (always a frightening and humbling black hole spelunking expedition), and here?s a brief history AFAICT.  Dates are approximate.

5/2000 - we move all development (CVS at the time, and bug tracking) to SourceForge.  This roughly coincided with PythonLabs leaving CNRI, so clearly we couldn?t continue running infra off of their systems.

10/2005 - we move to Subversion

9/2006 - we begin to discuss moving off of the SF bug tracker.  I believe that Thomas Wouters, Martin von Loewis, Brett Cannon (big surprise! :), and myself were involved in that effort, with Richard Jones (original author of Roundup) recusing himself.  The candidates were Roundup, Trac, Jira, and Launchpad.  I think Brett did the first round of feature reviews and comparisons.  David Goodger was also involved.  We did want it to be written in Python and we preferred running it on python.org infra, but neither of these were required criteria.

Jira and Roundup made the first cuts, with Launchpad and Trac being discarded as ?having issues? (I don?t have access in memory or emails to any of those details).  Jira was deemed pretty complex, but Atlassian offered hosting.  Roundup was ?not as polished" back then, but that wasn?t necessarily a negative.  It was easy to use and host, and had a complimentary feature set, but we felt like we needed volunteers to help us keep it going.  Richard Jones of course did fantastic work on the software itself, and we did manage to, um, round up enough volunteers to make it a viable choice.

10/2006 - the decision was made to move to Roundup, and we decided to accept Upfront?s offer to host the instance.

3/2007 - new-bugs-announce was created and notifications of new bugs was redirected to that mailing list.

I?ll disappear down that archive rabbit hole now, which in some cases goes back to 1995.  There are so many fun and scary paths to explore.  See you in 6 months.

jeremy-is-salty-ly y?rs,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/40a8bc55/attachment-0001.sig>

From greg.ewing at canterbury.ac.nz  Thu Mar  7 18:56:13 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 08 Mar 2019 12:56:13 +1300
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CACac1F9y6W8cbkyVLXf1M8UVxExDU=Z8Csvz50NVcF6xqhYu+g@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <20190307123950.GL4465@ando.pearwood.info>
 <CACac1F9y6W8cbkyVLXf1M8UVxExDU=Z8Csvz50NVcF6xqhYu+g@mail.gmail.com>
Message-ID: <5C81AF9D.2060002@canterbury.ac.nz>

Paul Moore wrote:
> There's a subtle difference in the mathematical
> and computing meanings [of idempotent] (around functions
 > with side-effects, which aren't a thing in maths)

Not really an issue here, since optionxform shouldn't be having
side effects if it's sane.

In any case, the word is easy enough to avoid in this case.
We could say something like:

    "The optionxform function transforms option names to a
    canonical form. If the name is already in canonical form,
    it should be returned unchanged."

-- 
Greg

From steve at pearwood.info  Thu Mar  7 19:12:37 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 8 Mar 2019 11:12:37 +1100
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <5C81AF9D.2060002@canterbury.ac.nz>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <20190307123950.GL4465@ando.pearwood.info>
 <CACac1F9y6W8cbkyVLXf1M8UVxExDU=Z8Csvz50NVcF6xqhYu+g@mail.gmail.com>
 <5C81AF9D.2060002@canterbury.ac.nz>
Message-ID: <20190308001237.GO4465@ando.pearwood.info>

On Fri, Mar 08, 2019 at 12:56:13PM +1300, Greg Ewing wrote:

> In any case, the word is easy enough to avoid in this case.

I don't think we should avoid using standard terminology even if we can. 
Knowledge of standard terminology is useful, and we don't generally make 
a practice of talking about (e.g.) "simultaneously running subtasks" 
when we can say "threads" instead.

You are happy to use the jargon terms "function" and "canonical form" 
without explanation, which I think proves that one person's jargon is 
another's obvious, clear, precise technical terminology.


> We could say something like:
> 
>    "The optionxform function transforms option names to a
>    canonical form. If the name is already in canonical form,
>    it should be returned unchanged."

How about:

    "The optionxform function transforms option names to a
    canonical form. This should be an idempotent function: 
    if the name is already in canonical form, it should be 
    returned unchanged."


requires six extra words, but it uses the correct technical term which 
will be familiar to some proportion of users, while also explaining the 
term for those who aren't familiar with it. We all win!


-- 
Steven

From brett at python.org  Thu Mar  7 19:36:22 2019
From: brett at python.org (Brett Cannon)
Date: Thu, 7 Mar 2019 16:36:22 -0800
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <4CF3BECF-9E75-4FAF-90AB-131034282956@python.org>
References: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>
 <CANc-5UwodQb0uLPA1viNk4wx5hvvmKbHOOGjejY6Peh8a_5syg@mail.gmail.com>
 <CAGbohnbP0vZ-+RHV+B3gWGg5P-LOWrTtRBG38cRfNkbm8m+G2A@mail.gmail.com>
 <4CF3BECF-9E75-4FAF-90AB-131034282956@python.org>
Message-ID: <CAP1=2W76XdXv4j6Km91ALyr3vSsTtALG_4BNiDUTi2VQOntcLQ@mail.gmail.com>

I'll start by saying I don't think a history lesson is important for this
PEP. This is simply a matter of evaluating whether Roundup or GitHub issues
is better for us and in the future. There's no real mistakes to watch out
for or anything (and if there is it's that self-hosting has a cost ;) .

On Thu, Mar 7, 2019 at 3:38 PM Barry Warsaw <barry at python.org> wrote:

> On Mar 7, 2019, at 14:36, Mariatta <mariatta at python.org> wrote:
>
> > I was not involved in core Python development back then, so if it is
> really important and if people think such paragraph needs to be part of the
> PEP, then perhaps someone else more knowledgeable will need to help with
> this.
> >
> > Personally, I don't think it was a horrible mistake. I believe the core
> devs back then carefully considered all options and decided that
> bpo/roundup was the way to go. And I really don't want to give that
> impression to the readers of this PEP that "I" or "core devs" now think it
> was a horrible mistake. If there is specific parts of the PEP that gives
> people that impression, then I'd definitely want to work and improve that.
>
> I did a little bit of archive archeology (always a frightening and
> humbling black hole spelunking expedition), and here?s a brief history
> AFAICT.  Dates are approximate.
>
> 5/2000 - we move all development (CVS at the time, and bug tracking) to
> SourceForge.  This roughly coincided with PythonLabs leaving CNRI, so
> clearly we couldn?t continue running infra off of their systems.
>
> 10/2005 - we move to Subversion
>
> 9/2006 - we begin to discuss moving off of the SF bug tracker.  I believe
> that Thomas Wouters, Martin von Loewis, Brett Cannon (big surprise! :), and
> myself were involved in that effort, with Richard Jones (original author of
> Roundup) recusing himself.  The candidates were Roundup, Trac, Jira, and
> Launchpad.  I think Brett did the first round of feature reviews and
> comparisons.  David Goodger was also involved.  We did want it to be
> written in Python and we preferred running it on python.org infra, but
> neither of these were required criteria.
>

This was actually my first infrastructure project and how I ended up on the
PSF board and the head of the infrastructure group. :)


>
> Jira and Roundup made the first cuts, with Launchpad and Trac being
> discarded as ?having issues? (I don?t have access in memory or emails to
> any of those details).  Jira was deemed pretty complex, but Atlassian
> offered hosting.  Roundup was ?not as polished" back then, but that wasn?t
> necessarily a negative.  It was easy to use and host, and had a
> complimentary feature set, but we felt like we needed volunteers to help us
> keep it going.  Richard Jones of course did fantastic work on the software
> itself, and we did manage to, um, round up enough volunteers to make it a
> viable choice.
>
> 10/2006 - the decision was made to move to Roundup, and we decided to
> accept Upfront?s offer to host the instance.


You're missing the step of "the decision was made to move to Jira and
people flipped out." :) We actually said Jira was our choice unless enough
people came forward to volunteer to help support us using Roundup. In the
end enough people did step forward and people didn't like us using Java and
a closed-source solution, so we went with Roundup (this is when RMS got
involved and asked us to reconsider; this is also when I learned that
volunteers saying they will help with something doesn't mean they actually
will, especially when they have no established reputation ;) .

The original announcement can be found at
https://mail.python.org/pipermail/python-dev/2006-October/069139.html.

-Brett


> 3/2007 - new-bugs-announce was created and notifications of new bugs was
> redirected to that mailing list.
>
> I?ll disappear down that archive rabbit hole now, which in some cases goes
> back to 1995.  There are so many fun and scary paths to explore.  See you
> in 6 months.
>
> jeremy-is-salty-ly y?rs,
> -Barry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/56c8ede0/attachment.html>

From brett at python.org  Thu Mar  7 19:38:53 2019
From: brett at python.org (Brett Cannon)
Date: Thu, 7 Mar 2019 16:38:53 -0800
Subject: [Python-Dev] PEP 12 updated with templates for header fields and
 sections
Message-ID: <CAP1=2W77qwvoqraaGCRUsA_C_Y8yr1MTLKUvG8t1s+0jRSBuug@mail.gmail.com>

https://github.com/python/peps/blob/master/pep-0012.rst now has a complete
list of header fields along with format clues for easier copy-and-paste use
in creating a new PEP. There is also a section template with one-liner
explanations for what each section is for so people don't accidentally
leave anything out. They are not in a single, unified template to copy to
partially make sure people actually read the PEP before they start writing.
:)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/cd2f20e9/attachment.html>

From steve at pearwood.info  Thu Mar  7 20:42:21 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 8 Mar 2019 12:42:21 +1100
Subject: [Python-Dev] Using CLA assistant for Python contributions
In-Reply-To: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>
References: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>
Message-ID: <20190308014221.GB12502@ando.pearwood.info>

On Thu, Mar 07, 2019 at 10:49:50AM -0800, Mariatta wrote:

> Unless I hear strong opposition (with reasons) from Python Steering
> Council, Python core developers, and active core contributors, I plan to
> switch us over to to CLA assistant in the coming week (before my OOOS of
> March 18)

OOOS? Object Oriented Operating System?


> How this will affect all contributors to Python old and new:
> 
> - you will need to sign the CLA again, even if you've signed it before (in
> bpo). It will take you several clicks, 

If this is a Github thing, it will probably take me a lot more than 
several clicks. It will probably take me a whole OS upgrade.


> but then you'll do this only once,
> and it takes effect immediately. (instead of waiting for a PSF staff to
> check for it)

I don't have an opinion on the specific technology you use for future 
CLAs. If CLA Assistant is the tool you want to use for future 
contributors, then go for it.

But I think it is ... I need a word weaker than rude ... ill-mannered(?) 
to ask people to re-sign an agreement they have already signed, without 
a really good reason.

I suppose it can be forgiven if:

- the signing process is straight-forward and easy;
- and supporting the legacy CLAs is particularly difficult;

but if the existing CLAs haven't expired, haven't been revoked, and 
there's no legal reason why they are no longer in force, then why 
are you asking us to re-sign?

I will if I really must, but I'll feel put out over it.


-- 
Steven

From mariatta at python.org  Thu Mar  7 21:22:11 2019
From: mariatta at python.org (Mariatta)
Date: Thu, 7 Mar 2019 18:22:11 -0800
Subject: [Python-Dev] Using CLA assistant for Python contributions
In-Reply-To: <20190308014221.GB12502@ando.pearwood.info>
References: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>
 <20190308014221.GB12502@ando.pearwood.info>
Message-ID: <CAGbohnbkxefa=LDAbh+yKfxmchQMugqbDTwVnWqFQX2hrAiQGQ@mail.gmail.com>

>
> I plan to
> > switch us over to to CLA assistant in the coming week (before my OOOS of
> > March 18)
> OOOS?


My Out Of Open Source aka I'm not doing any volunteer activities for 6
weeks (details:
https://discuss.python.org/t/mariatta-will-be-ooos-out-of-open-source-starting-march-18-may-9th-2019/973/3
)



> - the signing process is straight-forward and easy;


Yes it really is straight forward and easy. Everything happens on the
browser on GitHub. If you can open GitHub webpage then you can sign the CLA
via CLA assistant, no additional program/downloading/command line is
needed. It can be done with computer or your modern smartphone or
tablet/iPad with internet connection.


?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/45cddafe/attachment.html>

From mariatta at python.org  Thu Mar  7 21:30:00 2019
From: mariatta at python.org (Mariatta)
Date: Thu, 7 Mar 2019 18:30:00 -0800
Subject: [Python-Dev] Using CLA assistant for Python contributions
In-Reply-To: <CAGbohnbkxefa=LDAbh+yKfxmchQMugqbDTwVnWqFQX2hrAiQGQ@mail.gmail.com>
References: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>
 <20190308014221.GB12502@ando.pearwood.info>
 <CAGbohnbkxefa=LDAbh+yKfxmchQMugqbDTwVnWqFQX2hrAiQGQ@mail.gmail.com>
Message-ID: <CAGbohnb68713hzrXXxWPrHtd+su0xRWrUYmEGBF6jJ=po1CbJw@mail.gmail.com>

Thought I'll be a little more clearer: you'll need to re-sign the CLA only
for your future contributions (aka when you make new pull request to
Python).

Your previous CLA record are still available for The PSF, we're just not
going to export over data from existing CLA host to CLA assistant.
?
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190307/99530d8e/attachment.html>

From songofacandy at gmail.com  Thu Mar  7 21:51:25 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Fri, 8 Mar 2019 11:51:25 +0900
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <20190308001237.GO4465@ando.pearwood.info>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <20190307123950.GL4465@ando.pearwood.info>
 <CACac1F9y6W8cbkyVLXf1M8UVxExDU=Z8Csvz50NVcF6xqhYu+g@mail.gmail.com>
 <5C81AF9D.2060002@canterbury.ac.nz> <20190308001237.GO4465@ando.pearwood.info>
Message-ID: <CAEfz+TzEXdrD0-XMJh9MSftjm+tU-9Bqjzo+FRpcAmFsb9dMkw@mail.gmail.com>

>
> > We could say something like:
> >
> >    "The optionxform function transforms option names to a
> >    canonical form. If the name is already in canonical form,
> >    it should be returned unchanged."
>
> How about:
>
>     "The optionxform function transforms option names to a
>     canonical form. This should be an idempotent function:
>     if the name is already in canonical form, it should be
>     returned unchanged."
>
>
> requires six extra words, but it uses the correct technical term which
> will be familiar to some proportion of users, while also explaining the
> term for those who aren't familiar with it. We all win!
>

Thank you for suggestions.

Personally speaking, I think technical jargon is much easier than
normal English idioms or complex English syntax.

I learned "idempotent" long ago while learning HTTP.  On the other hand,
I don't know normal English idioms even 5-year children who speaks
English at home knows.

Technical jargon is good tool to communicate with people uses English
only for programming.  It shows the meaning very clearly with few words.

So I agree with you.  If reader may not know tech jargon widely used,
teach it instead of avoid it.

Regards,
-- 
Inada Naoki  <songofacandy at gmail.com>

From tir.karthi at gmail.com  Fri Mar  8 01:49:00 2019
From: tir.karthi at gmail.com (Karthikeyan)
Date: Fri, 8 Mar 2019 12:19:00 +0530
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>
References: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>
Message-ID: <CAAjsFLRbO9myg_2VhOZJP+_EvTg3cXE3nKf+t6KOxa4sV6jOpg@mail.gmail.com>

On Fri, Mar 8, 2019 at 12:41 AM Mariatta <mariatta at python.org> wrote:

> I'd like to formally present to Python-dev PEP 581: Using GitHub Issues
> for CPython
>
> Full text: https://www.python.org/dev/peps/pep-0581/
>
>
Thanks a lot for doing this!

* The current bug tracker has low contributions and moving to a place like
GitHub would open up to a lot of opportunities like integrations, webhooks
where people can build custom workflow etc. along with reducing the
maintenance burden on the team.
* GitHub also allows markdown and syntax highlighting on code snippets
would be much easier to read compared to current tracker. In some cases
GitHub can even inline the code for a permalink to a single line which adds
more context. Also support for editing comments to fix minor typos links
are great. Emoji support :)
* The current bpo search is surprisingly very effective though it just does
substring search across comments and patches I believe. If not I can google
keywords with site:bugs.python.org to get relevant results. I expect search
to be better on GitHub but worth experimenting since searching to get
related issues/discussion helps a lot in triaging.

Some points worth considering and some of them are addressed in the PEP
already.

* Moving to GitHub also comes at a cost of accessibility where there might
be people posting questions that are more suitable for stackoverflow,
python-list etc. Thus there might be more incoming issues that would
require more effort on triaging.
* There could be explicit guidelines on when a triager should close an
issue and current bpo supports custom fields for resolution, state of the
issue (open/close/pending/needs-test/patch-review)  that are updated.
Besides closing the issue there could be labels or a comment format to
describe the rationale and resolution.
* There could also be guidelines on when to lock the thread since there
could be cases where security issues or issues that can trigger heated
arguments are posted. It could get even more attention when it's posted on
Reddit or hackernews attracting unwanted attention e.g. security issues in
npm posted to reddit. Someone can chime in to lock it but guidelines on
when to do this would be helpful so that decision is not based on personal
opinion to lock it.
* Extended discussions in some cases along with no proper support for
threading could result in lot of duplicated messages that would be hard to
scroll through. It's a problem with current tracker too but something that
can come up where people will use +1 comments instead of using thumbs up
emoji. Rust community had some similar concerns since they do RFC
discussions on GitHub that result in 200+ comments. Though we don't do PEP
discussions some bugs can result in this.
* I wish python-bug-list, weekly summary continues to be integrated with
GitHub too since some of us don't want to watch the repo to get all the
activity along with pull requests and just want to track activity on issues.
* Currently there is some link rot on bpo for svn related links, broken
patch file links etc. Moving to GitHub I guess they are handled like magic
links for PEPs, bpo-NNNN, etc. that are mentioned in the PEP 581.

Personally, I think more people will love it once they get to use it so if
something like 100 issues can be migrated to a sample repo with labels,
content etc. As a shameless plug I tried migrating around 150 issues in
current bpo to GitLab sometime back with components as labels. Though an
apple to oranges comparison (GitLab UI vs GitHub UI) I personally find the
UI (also GitHub UI) very easy to navigate in terms of clicking on labels,
search, sort, filter etc. in https://gitlab.com/tirkarthi/python-bugs/issues
and the issue is more easy to read with markdown support for code where I
added highlight to snippet
https://gitlab.com/tirkarthi/python-bugs/issues/140

-- 
Regards,
Karthikeyan S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190308/88e103cb/attachment-0001.html>

From p.f.moore at gmail.com  Fri Mar  8 02:25:37 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 8 Mar 2019 07:25:37 +0000
Subject: [Python-Dev] Using CLA assistant for Python contributions
In-Reply-To: <CAGbohnb68713hzrXXxWPrHtd+su0xRWrUYmEGBF6jJ=po1CbJw@mail.gmail.com>
References: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>
 <20190308014221.GB12502@ando.pearwood.info>
 <CAGbohnbkxefa=LDAbh+yKfxmchQMugqbDTwVnWqFQX2hrAiQGQ@mail.gmail.com>
 <CAGbohnb68713hzrXXxWPrHtd+su0xRWrUYmEGBF6jJ=po1CbJw@mail.gmail.com>
Message-ID: <CACac1F_fB3uSb=z8zhj6M1PtNgkj3RhJRc7gM8afkpTxsHVfXQ@mail.gmail.com>

On Fri, 8 Mar 2019 at 02:32, Mariatta <mariatta at python.org> wrote:
>
> Thought I'll be a little more clearer: you'll need to re-sign the CLA only for your future contributions (aka when you make new pull request to Python).

My preference would be to just re-sign the CLA *immediately*, and not
wait for when I have a PR - I presume that would be
possible/supported. Instructions on how to do so when the switchover
happens would be useful.

Paul

PS I would also prefer not to have to re-sign, but just have my
existing confirmation carried over. I don't *know* if there are any
implications at my end around re-signing, and my preference is simply
to not ask that question (on the basis of why make things harder for
myself). But if it's needed, then fair enough.

From p.f.moore at gmail.com  Fri Mar  8 02:30:09 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 8 Mar 2019 07:30:09 +0000
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <5C81AF9D.2060002@canterbury.ac.nz>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <20190307123950.GL4465@ando.pearwood.info>
 <CACac1F9y6W8cbkyVLXf1M8UVxExDU=Z8Csvz50NVcF6xqhYu+g@mail.gmail.com>
 <5C81AF9D.2060002@canterbury.ac.nz>
Message-ID: <CACac1F8Ob5JdVzBEK0N9V26_VXJPB6G_D2c+Gv_FNfFZYfCSRQ@mail.gmail.com>

On Thu, 7 Mar 2019 at 23:58, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Paul Moore wrote:
> > There's a subtle difference in the mathematical
> > and computing meanings [of idempotent] (around functions
>  > with side-effects, which aren't a thing in maths)
>
> Not really an issue here, since optionxform shouldn't be having
> side effects if it's sane.
>
> In any case, the word is easy enough to avoid in this case.
> We could say something like:
>
>     "The optionxform function transforms option names to a
>     canonical form. If the name is already in canonical form,
>     it should be returned unchanged."
>

Precisely. +1 on this wording if we choose to go this way.

If someone *really* wants to link the idea to the term "idempotent"
then a simple "(i.e., the function must be idempotent)" would be
sufficient to confirm for people who know the term, avoid making it
unclear for people who don't, and teach people who don't the meaning
of the term.

Paul

From p.f.moore at gmail.com  Fri Mar  8 02:33:26 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 8 Mar 2019 07:33:26 +0000
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <CAEfz+TzEXdrD0-XMJh9MSftjm+tU-9Bqjzo+FRpcAmFsb9dMkw@mail.gmail.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <20190307123950.GL4465@ando.pearwood.info>
 <CACac1F9y6W8cbkyVLXf1M8UVxExDU=Z8Csvz50NVcF6xqhYu+g@mail.gmail.com>
 <5C81AF9D.2060002@canterbury.ac.nz> <20190308001237.GO4465@ando.pearwood.info>
 <CAEfz+TzEXdrD0-XMJh9MSftjm+tU-9Bqjzo+FRpcAmFsb9dMkw@mail.gmail.com>
Message-ID: <CACac1F-FKU_WPMuaqDN1PpoQW4vOr42eRJeBU5TM7YDZ3NmccQ@mail.gmail.com>

On Fri, 8 Mar 2019 at 02:54, Inada Naoki <songofacandy at gmail.com> wrote:
>
> Personally speaking, I think technical jargon is much easier than
> normal English idioms or complex English syntax.
>
> I learned "idempotent" long ago while learning HTTP.  On the other hand,
> I don't know normal English idioms even 5-year children who speaks
> English at home knows.
>
> Technical jargon is good tool to communicate with people uses English
> only for programming.  It shows the meaning very clearly with few words.

Thanks for the reminder that the trade-off is different for
non-English speakers - and my apologies for not taking that into
account.

> So I agree with you.  If reader may not know tech jargon widely used,
> teach it instead of avoid it.

Your point is taken, let's include the term in the explanation, but
let's also spell out the behaviour for people who don't know the term
(or who simply find it less familiar because it's not something
commonly used in areas they work in).

Paul

From eric at trueblade.com  Fri Mar  8 03:26:56 2019
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 8 Mar 2019 08:26:56 +0000
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <20190308001237.GO4465@ando.pearwood.info>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <20190307123950.GL4465@ando.pearwood.info>
 <CACac1F9y6W8cbkyVLXf1M8UVxExDU=Z8Csvz50NVcF6xqhYu+g@mail.gmail.com>
 <5C81AF9D.2060002@canterbury.ac.nz>
 <20190308001237.GO4465@ando.pearwood.info>
Message-ID: <7BF92869-B60F-4AD8-839F-BF514DC0E24E@trueblade.com>


> On Mar 8, 2019, at 12:12 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> 
>> On Fri, Mar 08, 2019 at 12:56:13PM +1300, Greg Ewing wrote:
>> 
>> In any case, the word is easy enough to avoid in this case.
> 
> I don't think we should avoid using standard terminology even if we can. 
> Knowledge of standard terminology is useful, and we don't generally make 
> a practice of talking about (e.g.) "simultaneously running subtasks" 
> when we can say "threads" instead.
> 
> You are happy to use the jargon terms "function" and "canonical form" 
> without explanation, which I think proves that one person's jargon is 
> another's obvious, clear, precise technical terminology.
> 
> 
>> We could say something like:
>> 
>>   "The optionxform function transforms option names to a
>>   canonical form. If the name is already in canonical form,
>>   it should be returned unchanged."
> 
> How about:
> 
>    "The optionxform function transforms option names to a
>    canonical form. This should be an idempotent function: 
>    if the name is already in canonical form, it should be 
>    returned unchanged."

I?d prefer something less passive than ?it should remain unchanged? (as my high school English teacher would say: ?by whom??). Something like ?If optionxform is called on a name that is already in canonical form, then it should return that name unchanged?.  Then add something like ?That is, optionxform should be idempotent?.

Eric

> 
> 
> requires six extra words, but it uses the correct technical term which 
> will be familiar to some proportion of users, while also explaining the 
> term for those who aren't familiar with it. We all win!
> 
> 
> -- 
> Steven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com


From vstinner at redhat.com  Fri Mar  8 03:29:34 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Fri, 8 Mar 2019 09:29:34 +0100
Subject: [Python-Dev] PEP 12 updated with templates for header fields
 and sections
In-Reply-To: <CAP1=2W77qwvoqraaGCRUsA_C_Y8yr1MTLKUvG8t1s+0jRSBuug@mail.gmail.com>
References: <CAP1=2W77qwvoqraaGCRUsA_C_Y8yr1MTLKUvG8t1s+0jRSBuug@mail.gmail.com>
Message-ID: <CA+3bQGE+rgMxo4tYn_K-r8FfRzQ9FPourpO_5FcOFCGY2HB0vg@mail.gmail.com>

Hi Brett,

I like to see discussions where a PEP has been disccussed. There is
the "Post History" just gives dates. The problem is that PEPs are
discussed on 3 mailing lists: python-ideas, python-dev and
python-committers. Maybe some PEP are now also discussed on
discuss.python.org.

Do you have any recommendation to referer to these discussions?

I would like to suggest to add URLs to the first messages of all
threads about a PEP... Well... I know that for PEP 572, this list
would be hard to create and maybe not really useful. Maybe not add
links to *all* threads, but only the most actives or most "useful"
threads? The PEP author would be free to decide which threads are
important or not ;-)

The status quo (Post History header) is fine, if I really want to find
these discussions, I'm able to find them from dates :-)

Victor

Le ven. 8 mars 2019 ? 01:42, Brett Cannon <brett at python.org> a ?crit :
>
> https://github.com/python/peps/blob/master/pep-0012.rst now has a complete list of header fields along with format clues for easier copy-and-paste use in creating a new PEP. There is also a section template with one-liner explanations for what each section is for so people don't accidentally leave anything out. They are not in a single, unified template to copy to partially make sure people actually read the PEP before they start writing. :)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From J.Demeyer at UGent.be  Fri Mar  8 03:37:04 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 8 Mar 2019 09:37:04 +0100
Subject: [Python-Dev] PEP 12 updated with templates for header fields
 and sections
In-Reply-To: <CA+3bQGE+rgMxo4tYn_K-r8FfRzQ9FPourpO_5FcOFCGY2HB0vg@mail.gmail.com>
References: <CAP1=2W77qwvoqraaGCRUsA_C_Y8yr1MTLKUvG8t1s+0jRSBuug@mail.gmail.com>
 <CA+3bQGE+rgMxo4tYn_K-r8FfRzQ9FPourpO_5FcOFCGY2HB0vg@mail.gmail.com>
Message-ID: <5C8229B0.50203@UGent.be>

On 2019-03-08 09:29, Victor Stinner wrote:
> I would like to suggest to add URLs to the first messages of all
> threads about a PEP...

Like this?

https://www.python.org/dev/peps/pep-0580/#discussion

From vstinner at redhat.com  Fri Mar  8 04:04:45 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Fri, 8 Mar 2019 10:04:45 +0100
Subject: [Python-Dev] PEP 12 updated with templates for header fields
 and sections
In-Reply-To: <5C8229B0.50203@UGent.be>
References: <CAP1=2W77qwvoqraaGCRUsA_C_Y8yr1MTLKUvG8t1s+0jRSBuug@mail.gmail.com>
 <CA+3bQGE+rgMxo4tYn_K-r8FfRzQ9FPourpO_5FcOFCGY2HB0vg@mail.gmail.com>
 <5C8229B0.50203@UGent.be>
Message-ID: <CA+3bQGEHpQJumSqhs0yqL+aE2905g98XE3-g4Y9+CQwagAx8-A@mail.gmail.com>

Yeah, such list is more convenient than just "Post-History:
14-Aug-2001, 03-Sept-2001".

Victor

Le ven. 8 mars 2019 ? 09:40, Jeroen Demeyer <J.Demeyer at ugent.be> a ?crit :
>
> On 2019-03-08 09:29, Victor Stinner wrote:
> > I would like to suggest to add URLs to the first messages of all
> > threads about a PEP...
>
> Like this?
>
> https://www.python.org/dev/peps/pep-0580/#discussion
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From steve at pearwood.info  Fri Mar  8 05:36:56 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 8 Mar 2019 21:36:56 +1100
Subject: [Python-Dev] Using CLA assistant for Python contributions
In-Reply-To: <CACac1F_fB3uSb=z8zhj6M1PtNgkj3RhJRc7gM8afkpTxsHVfXQ@mail.gmail.com>
References: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>
 <20190308014221.GB12502@ando.pearwood.info>
 <CAGbohnbkxefa=LDAbh+yKfxmchQMugqbDTwVnWqFQX2hrAiQGQ@mail.gmail.com>
 <CAGbohnb68713hzrXXxWPrHtd+su0xRWrUYmEGBF6jJ=po1CbJw@mail.gmail.com>
 <CACac1F_fB3uSb=z8zhj6M1PtNgkj3RhJRc7gM8afkpTxsHVfXQ@mail.gmail.com>
Message-ID: <20190308103656.GD12502@ando.pearwood.info>

On Fri, Mar 08, 2019 at 07:25:37AM +0000, Paul Moore wrote:

> PS I would also prefer not to have to re-sign, but just have my
> existing confirmation carried over. I don't *know* if there are any
> implications at my end around re-signing, and my preference is simply
> to not ask that question (on the basis of why make things harder for
> myself). But if it's needed, then fair enough.

Because what you don't know can't hurt you?

You're signing a legal document. Shouldn't you understand the 
implications of signing it before you sign?



-- 
Steven

From p.f.moore at gmail.com  Fri Mar  8 05:58:00 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 8 Mar 2019 10:58:00 +0000
Subject: [Python-Dev] Using CLA assistant for Python contributions
In-Reply-To: <20190308103656.GD12502@ando.pearwood.info>
References: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>
 <20190308014221.GB12502@ando.pearwood.info>
 <CAGbohnbkxefa=LDAbh+yKfxmchQMugqbDTwVnWqFQX2hrAiQGQ@mail.gmail.com>
 <CAGbohnb68713hzrXXxWPrHtd+su0xRWrUYmEGBF6jJ=po1CbJw@mail.gmail.com>
 <CACac1F_fB3uSb=z8zhj6M1PtNgkj3RhJRc7gM8afkpTxsHVfXQ@mail.gmail.com>
 <20190308103656.GD12502@ando.pearwood.info>
Message-ID: <CACac1F_cmM53sAmZBsC40+dasM752ZHXWF_m_ABRyScqGP=n4w@mail.gmail.com>

On Fri, 8 Mar 2019 at 10:39, Steven D'Aprano <steve at pearwood.info> wrote:
>
> On Fri, Mar 08, 2019 at 07:25:37AM +0000, Paul Moore wrote:
>
> > PS I would also prefer not to have to re-sign, but just have my
> > existing confirmation carried over. I don't *know* if there are any
> > implications at my end around re-signing, and my preference is simply
> > to not ask that question (on the basis of why make things harder for
> > myself). But if it's needed, then fair enough.
>
> Because what you don't know can't hurt you?
>
> You're signing a legal document. Shouldn't you understand the
> implications of signing it before you sign?

Sigh. Trust me that I know what I'm saying, please. I do understand
the implications. And re-signing a CLA *will* involve different
processes for me than a simple continuance of an existing CLA - even
if that CLA is confirmed by the PSF as legally identical to the new
one. I can, and will, negotiate that difficulty myself if necessary -
but honestly, it's not really the place of anyone on this list to tell
me what I need to do in order to ensure my compliance with the CLA and
with my employer's policies.

I offered the information for awareness, to ensure that the people
proposing this change knew that it had the potential to incur extra
work for people being asked to re-sign, beyond the mere mechanics of
re-signing. I did not expect arguments over my understanding of what I
needed to do (nor did I expect the Spanish Inquisition, but *nobody*
expects that ;-))

Paul

From steve at pearwood.info  Fri Mar  8 06:46:03 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 8 Mar 2019 22:46:03 +1100
Subject: [Python-Dev] Using CLA assistant for Python contributions
In-Reply-To: <CACac1F_cmM53sAmZBsC40+dasM752ZHXWF_m_ABRyScqGP=n4w@mail.gmail.com>
References: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>
 <20190308014221.GB12502@ando.pearwood.info>
 <CAGbohnbkxefa=LDAbh+yKfxmchQMugqbDTwVnWqFQX2hrAiQGQ@mail.gmail.com>
 <CAGbohnb68713hzrXXxWPrHtd+su0xRWrUYmEGBF6jJ=po1CbJw@mail.gmail.com>
 <CACac1F_fB3uSb=z8zhj6M1PtNgkj3RhJRc7gM8afkpTxsHVfXQ@mail.gmail.com>
 <20190308103656.GD12502@ando.pearwood.info>
 <CACac1F_cmM53sAmZBsC40+dasM752ZHXWF_m_ABRyScqGP=n4w@mail.gmail.com>
Message-ID: <20190308114602.GE12502@ando.pearwood.info>

On Fri, Mar 08, 2019 at 10:58:00AM +0000, Paul Moore wrote:
> On Fri, 8 Mar 2019 at 10:39, Steven D'Aprano <steve at pearwood.info> wrote:
> >
> > On Fri, Mar 08, 2019 at 07:25:37AM +0000, Paul Moore wrote:
> >
> > > PS I would also prefer not to have to re-sign, but just have my
> > > existing confirmation carried over. I don't *know* if there are any
> > > implications at my end around re-signing, and my preference is simply
> > > to not ask that question (on the basis of why make things harder for
> > > myself). But if it's needed, then fair enough.
> >
> > Because what you don't know can't hurt you?
> >
> > You're signing a legal document. Shouldn't you understand the
> > implications of signing it before you sign?
> 
> Sigh. Trust me that I know what I'm saying, please. I do understand
> the implications.

I'm sorry Paul, we seem to have difficulty in communicating today.

You explicitly said that you did not know if there are any implications 
for you to re-sign, now you're saying that we ought to trust you that 
you do know the implications. Forgive me if I'm confused.

Oh! I think I get it... when you say you prefer to not to ask the 
question, you mean in the sense of "Don't make me re-sign, and that way 
I won't have to ask about the implications of re-signing".

Is that what you meant? I'm sorry, I truly read it as "I don't know what 
the implications are, and I don't want to know, but I'll sign".

Communication is hard.



> And re-signing a CLA *will* involve different
> processes for me than a simple continuance of an existing CLA - even
> if that CLA is confirmed by the PSF as legally identical to the new
> one.

That's a very good point, and I hope Mariatta is reading :-)

Fortunately, I don't have to worry about re-negotiating a new CLA with 
an employer. But I imagine a lot of people will be in your same 
situation.


> I can, and will, negotiate that difficulty myself if necessary -
> but honestly, it's not really the place of anyone on this list to tell
> me what I need to do in order to ensure my compliance with the CLA and
> with my employer's policies.

Chill out Paul, I'm not your dad and I'm not telling you to clean your 
room :-)

But we are a community, not as close as personal friends, but closer 
than perfect strangers, and if I see people in my community talking 
about doing something which may be problematic for them, the kind thing 
to do is point that out.

Sorry if I stepped on your toes by misunderstanding your words.


-- 
Steven

From tritium-list at sdamon.com  Fri Mar  8 08:09:11 2019
From: tritium-list at sdamon.com (Alex Walters)
Date: Fri, 8 Mar 2019 08:09:11 -0500
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
Message-ID: <081901d4d5b0$1f72e1d0$5e58a570$@sdamon.com>

I'm confused about this.  Didn't you need someone with merge permissions already to merge a pep into the pep repo?  Isn't this just adding a layer of paperwork to something that was already the case for all practical purposes?

> -----Original Message-----
> From: Python-Dev <python-dev-bounces+tritium-
> list=sdamon.com at python.org> On Behalf Of Brett Cannon
> Sent: Monday, March 4, 2019 8:44 PM
> To: python-dev <python-dev at python.org>
> Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
> 
> The steering council has implemented a new idea called sponsors to the PEP
> process (added in
> https://github.com/python/peps/commit/c58d32c33bd06eb386d3f33963a14
> 34510528f68). The thinking is that to help make sure PEPs from non-core
> developers receive appropriate guidance through the PEP process, a core
> developer needs to sign on to be a sponsor of the PEP. Being a sponsor does
> not preclude the core dev from eventually becoming a co-author or BDFL-
> delegate later on (but obviously not both), but the expectation is the
> sponsor is supportive of the idea (because if a single core dev won't sign on
> to help then what chance does the PEP have of being accepted?).
> 
> 
> If this doesn't turn out well we can obviously revert this, but hopefully this
> will make things smoother for those who are new to the PEP process.



From Richard at Damon-Family.org  Fri Mar  8 08:46:02 2019
From: Richard at Damon-Family.org (Richard Damon)
Date: Fri, 8 Mar 2019 08:46:02 -0500
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <081901d4d5b0$1f72e1d0$5e58a570$@sdamon.com>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <081901d4d5b0$1f72e1d0$5e58a570$@sdamon.com>
Message-ID: <46403614-6d88-d4b2-0578-5833db5255ec@Damon-Family.org>

On 3/8/19 8:09 AM, Alex Walters wrote:
> I'm confused about this.  Didn't you need someone with merge permissions already to merge a pep into the pep repo?  Isn't this just adding a layer of paperwork to something that was already the case for all practical purposes?
>
>
I would say the difference that before, the non-committer proposer just
needed to convince some committer that the PEP was 'a good idea' and
worth being discussed. Now, that person needs to persuade some committer
that not only is it a good idea, but that it is worth the committer
committing themselves to supporting the proposer through the process.

-- 
Richard Damon


From steve.dower at python.org  Fri Mar  8 10:58:38 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 8 Mar 2019 07:58:38 -0800
Subject: [Python-Dev] configparser: should optionxform be idempotent?
In-Reply-To: <7BF92869-B60F-4AD8-839F-BF514DC0E24E@trueblade.com>
References: <CAEfz+Twwyt_U=hgm2SUwP_NPefE2n6KSduFBqaOg6-K4c-NNgQ@mail.gmail.com>
 <CACac1F_5MsNeC-pGO7kQRCt7HEAx8yFE2UyVD9i-CFQ3ZDfF8w@mail.gmail.com>
 <20190307123950.GL4465@ando.pearwood.info>
 <CACac1F9y6W8cbkyVLXf1M8UVxExDU=Z8Csvz50NVcF6xqhYu+g@mail.gmail.com>
 <5C81AF9D.2060002@canterbury.ac.nz>
 <20190308001237.GO4465@ando.pearwood.info>
 <7BF92869-B60F-4AD8-839F-BF514DC0E24E@trueblade.com>
Message-ID: <85a1f069-93d8-a16b-1ed3-22965ae4bd4f@python.org>

I just have one thing to add to the discussion, which is this:

https://youtu.be/hAnCiTpxXPg?t=6339

Yes, people actually read and interpret our documentation :)  So 
discussions like this are probably a good thing in terms of getting the 
best descriptions in there, but if we use a specific technical term *not 
quite right* then we will be found out.

(A few of us core devs chatted with Al afterwards and there's no bad 
blood, so don't worry about that.)

Cheers,
Steve

From mariatta at python.org  Fri Mar  8 12:59:46 2019
From: mariatta at python.org (Mariatta)
Date: Fri, 8 Mar 2019 09:59:46 -0800
Subject: [Python-Dev] Using CLA assistant for Python contributions
In-Reply-To: <CACac1F_fB3uSb=z8zhj6M1PtNgkj3RhJRc7gM8afkpTxsHVfXQ@mail.gmail.com>
References: <CAGbohnbpnjLcbXHy9x6CtOY9KRRu5NEasUnkoQ0r32iq7STV7w@mail.gmail.com>
 <20190308014221.GB12502@ando.pearwood.info>
 <CAGbohnbkxefa=LDAbh+yKfxmchQMugqbDTwVnWqFQX2hrAiQGQ@mail.gmail.com>
 <CAGbohnb68713hzrXXxWPrHtd+su0xRWrUYmEGBF6jJ=po1CbJw@mail.gmail.com>
 <CACac1F_fB3uSb=z8zhj6M1PtNgkj3RhJRc7gM8afkpTxsHVfXQ@mail.gmail.com>
Message-ID: <CAGbohnYbp=LYcPwStmiSz3nB7H_xvM8CHC+oLTjZ6w+whCE-uw@mail.gmail.com>

On Thu, Mar 7, 2019 at 11:25 PM Paul Moore <p.f.moore at gmail.com> wrote:

>
> My preference would be to just re-sign the CLA *immediately*, and not
> wait for when I have a PR - I presume that would be
> possible/supported.
>

Yes this is possible, and once the switchover happens, I will post the link
with instructions on how to do that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190308/38794382/attachment.html>

From status at bugs.python.org  Fri Mar  8 13:08:07 2019
From: status at bugs.python.org (Python tracker)
Date: Fri,  8 Mar 2019 18:08:07 +0000 (UTC)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190308180807.56ACE52B1EC@bugs.ams1.psf.io>


ACTIVITY SUMMARY (2019-03-01 - 2019-03-08)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    7031 (+19)
  closed 40963 (+65)
  total  47994 (+84)

Open issues with patches: 2793 


Issues opened (55)
==================

#33329: sigaddset() can fail on some signal numbers
https://bugs.python.org/issue33329  reopened by larry

#36097: Use only public C-API in _xxsubinterpreters module.
https://bugs.python.org/issue36097  reopened by eric.snow

#36125: Cannot cross-compile to more featureful but same tune
https://bugs.python.org/issue36125  reopened by xdegaye

#36160: Multiple errors in test_site.py on sysconfig._CONFIG_VARS.clea
https://bugs.python.org/issue36160  opened by Ivan.Pozdeev

#36161: Use thread-safe functions instead of unsafe ones (crypt, ttyna
https://bugs.python.org/issue36161  opened by twouters

#36164: Updating Py_InspectFlag programmatically
https://bugs.python.org/issue36164  opened by ishimoto

#36165: DOC: ssl.rst is missing formatting on two links
https://bugs.python.org/issue36165  opened by cheryl.sabella

#36166: DOC: Fix markup on function parameter on datamodel.rst
https://bugs.python.org/issue36166  opened by cheryl.sabella

#36167: DOC: Incorrect capitalization in Programming FAQ
https://bugs.python.org/issue36167  opened by cheryl.sabella

#36168: DOC: Fix capitalization in string.rst
https://bugs.python.org/issue36168  opened by cheryl.sabella

#36172: csv module internal consistency
https://bugs.python.org/issue36172  opened by Shane Smith

#36174: Remove licenseUrl field from nuget packages
https://bugs.python.org/issue36174  opened by steve.dower

#36176: Make IDLE Autocomplete / Calltip Window Colors Configurable
https://bugs.python.org/issue36176  opened by greylaw89

#36179: _hashopenssl has reference leaks in OOM case
https://bugs.python.org/issue36179  opened by christian.heimes

#36180: mboxMessage.get_payload throws TypeError on malformed content 
https://bugs.python.org/issue36180  opened by enrico

#36182: Path.write_text() docs do not include the case that a file exi
https://bugs.python.org/issue36182  opened by lys.nikolaou

#36184: [EASY] test_gdb.test_threads() is specific to _POSIX_THREADS, 
https://bugs.python.org/issue36184  opened by vstinner

#36189: DOC: Correct word in tutorial introduction
https://bugs.python.org/issue36189  opened by cheryl.sabella

#36194: Add "make regen-configure"
https://bugs.python.org/issue36194  opened by vstinner

#36195: initializer is not a valid param in ThreadPoolExecutor
https://bugs.python.org/issue36195  opened by harman786

#36201: AssertCountEqual does not work for nested dictionaries.
https://bugs.python.org/issue36201  opened by walterqian

#36202: Calling Py_DecodeLocale() before _PyPreConfig_Write() can prod
https://bugs.python.org/issue36202  opened by vstinner

#36203: PyWeakref_NewRef docs are misleading
https://bugs.python.org/issue36203  opened by Maxwell Bernstein

#36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_Init
https://bugs.python.org/issue36204  opened by vstinner

#36205: Python 3.7 and 3.8 process_time is not reported correctly when
https://bugs.python.org/issue36205  opened by Nitapol

#36207: robotsparser deny all with some rules
https://bugs.python.org/issue36207  opened by quentin-maire

#36208: AsyncIO V4MAPPED addresses with V6ONLY.
https://bugs.python.org/issue36208  opened by jeeger

#36210: ncurses versus cursus integration - platform differences and d
https://bugs.python.org/issue36210  opened by Michael.Felt

#36211: show full url when execute "make -C Doc/ serve"
https://bugs.python.org/issue36211  opened by matrixise

#36212: [2.7] Coverity scan: Modules/_hotshot.c , Variable "s1" going 
https://bugs.python.org/issue36212  opened by cstratak

#36213: subprocess.check_output() fails with OSError: [WinError 87] wh
https://bugs.python.org/issue36213  opened by Geoff.Alexander

#36214: AC_RUN_IFELSE macros not used as arguments of AC_CACHE_VAL et 
https://bugs.python.org/issue36214  opened by xdegaye

#36215: Should AppVeyor run compile Python in debug mode?
https://bugs.python.org/issue36215  opened by vstinner

#36216: urlsplit does not handle NFKC normalization
https://bugs.python.org/issue36216  opened by steve.dower

#36217: recvmsg support on Windows
https://bugs.python.org/issue36217  opened by chrysn

#36218: .sort() segfaults consistently on crafted input
https://bugs.python.org/issue36218  opened by Lyn Levenick

#36219: Add edit option in IDLE to convert smart quotes to ascii quote
https://bugs.python.org/issue36219  opened by rhettinger

#36220: LOAD_NAME and LOAD_GLOBAL handle dict subclasses for globals()
https://bugs.python.org/issue36220  opened by Kevin Shweh

#36221: Setting PYTHONASYNCIODEBUG breaks warnings
https://bugs.python.org/issue36221  opened by ods

#36225: Lingering subinterpreters should be implicitly cleared on shut
https://bugs.python.org/issue36225  opened by ncoghlan

#36226: multipart/related header causes false positive StartBoundaryNo
https://bugs.python.org/issue36226  opened by cschmidbauer

#36227: Add default_namespace argument to xml.etree.ElementTree.tostri
https://bugs.python.org/issue36227  opened by Bernt.R??skar.Brenna

#36229: Linear-time ops for some mutable collections.
https://bugs.python.org/issue36229  opened by brandtbucher

#36230: Please sort assertSetEqual's output
https://bugs.python.org/issue36230  opened by Jess

#36231: no "proper" header files on macOS 10.14 Mojave
https://bugs.python.org/issue36231  opened by dimpase

#36232: Improve error message on dbm.open
https://bugs.python.org/issue36232  opened by rougeth

#36233: xml ElementTree quotation marks of xml version string
https://bugs.python.org/issue36233  opened by Photoniker

#36234: test_os: add tests for invalid uid type
https://bugs.python.org/issue36234  opened by vstinner

#36235: distutils.sysconfig.customize_compiler() overrides CFLAGS var 
https://bugs.python.org/issue36235  opened by vstinner

#36236: Python crash on macOS when CWD is invalid
https://bugs.python.org/issue36236  opened by lkollar

#36238: distutils complains "package init file 'xxx/__init__.py' not f
https://bugs.python.org/issue36238  opened by jt

#36239: gettext: GNUTranslations doesn't parse properly comments in de
https://bugs.python.org/issue36239  opened by vstinner

#36240: Definitions of time
https://bugs.python.org/issue36240  opened by Robert Billing

#36241: MD5 checksum is not valid for v2.7.16 "Windows x86-64 MSI inst
https://bugs.python.org/issue36241  opened by andrejs-sisojevs-accenture

#36243: Python os.listdir fails with FileNotFoundError when directory 
https://bugs.python.org/issue36243  opened by Geoff.Alexander



Most recent 15 issues with no replies (15)
==========================================

#36243: Python os.listdir fails with FileNotFoundError when directory 
https://bugs.python.org/issue36243

#36240: Definitions of time
https://bugs.python.org/issue36240

#36238: distutils complains "package init file 'xxx/__init__.py' not f
https://bugs.python.org/issue36238

#36232: Improve error message on dbm.open
https://bugs.python.org/issue36232

#36231: no "proper" header files on macOS 10.14 Mojave
https://bugs.python.org/issue36231

#36229: Linear-time ops for some mutable collections.
https://bugs.python.org/issue36229

#36225: Lingering subinterpreters should be implicitly cleared on shut
https://bugs.python.org/issue36225

#36221: Setting PYTHONASYNCIODEBUG breaks warnings
https://bugs.python.org/issue36221

#36217: recvmsg support on Windows
https://bugs.python.org/issue36217

#36215: Should AppVeyor run compile Python in debug mode?
https://bugs.python.org/issue36215

#36214: AC_RUN_IFELSE macros not used as arguments of AC_CACHE_VAL et 
https://bugs.python.org/issue36214

#36212: [2.7] Coverity scan: Modules/_hotshot.c , Variable "s1" going 
https://bugs.python.org/issue36212

#36211: show full url when execute "make -C Doc/ serve"
https://bugs.python.org/issue36211

#36210: ncurses versus cursus integration - platform differences and d
https://bugs.python.org/issue36210

#36207: robotsparser deny all with some rules
https://bugs.python.org/issue36207



Most recent 15 issues waiting for review (15)
=============================================

#36236: Python crash on macOS when CWD is invalid
https://bugs.python.org/issue36236

#36235: distutils.sysconfig.customize_compiler() overrides CFLAGS var 
https://bugs.python.org/issue36235

#36234: test_os: add tests for invalid uid type
https://bugs.python.org/issue36234

#36231: no "proper" header files on macOS 10.14 Mojave
https://bugs.python.org/issue36231

#36229: Linear-time ops for some mutable collections.
https://bugs.python.org/issue36229

#36227: Add default_namespace argument to xml.etree.ElementTree.tostri
https://bugs.python.org/issue36227

#36226: multipart/related header causes false positive StartBoundaryNo
https://bugs.python.org/issue36226

#36218: .sort() segfaults consistently on crafted input
https://bugs.python.org/issue36218

#36216: urlsplit does not handle NFKC normalization
https://bugs.python.org/issue36216

#36211: show full url when execute "make -C Doc/ serve"
https://bugs.python.org/issue36211

#36210: ncurses versus cursus integration - platform differences and d
https://bugs.python.org/issue36210

#36195: initializer is not a valid param in ThreadPoolExecutor
https://bugs.python.org/issue36195

#36182: Path.write_text() docs do not include the case that a file exi
https://bugs.python.org/issue36182

#36179: _hashopenssl has reference leaks in OOM case
https://bugs.python.org/issue36179

#36164: Updating Py_InspectFlag programmatically
https://bugs.python.org/issue36164



Top 10 most discussed issues (10)
=================================

#33944: Deprecate and remove pth files
https://bugs.python.org/issue33944  32 msgs

#35967: Better platform.processor support
https://bugs.python.org/issue35967  17 msgs

#33608: Add a cross-interpreter-safe mechanism to indicate that an obj
https://bugs.python.org/issue33608  14 msgs

#36218: .sort() segfaults consistently on crafted input
https://bugs.python.org/issue36218  10 msgs

#36239: gettext: GNUTranslations doesn't parse properly comments in de
https://bugs.python.org/issue36239   9 msgs

#29515: socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
https://bugs.python.org/issue29515   7 msgs

#36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_Init
https://bugs.python.org/issue36204   7 msgs

#36227: Add default_namespace argument to xml.etree.ElementTree.tostri
https://bugs.python.org/issue36227   6 msgs

#36230: Please sort assertSetEqual's output
https://bugs.python.org/issue36230   6 msgs

#33346: Syntax error with async generator inside dictionary comprehens
https://bugs.python.org/issue33346   5 msgs



Issues closed (63)
==================

#8075: Windows (Vista/7) install error when choosing to compile .py f
https://bugs.python.org/issue8075  closed by steve.dower

#9566: Compilation warnings under x64 Windows
https://bugs.python.org/issue9566  closed by vstinner

#10009: Automated MSI installation does not work
https://bugs.python.org/issue10009  closed by steve.dower

#20906: Issues in Unicode HOWTO
https://bugs.python.org/issue20906  closed by vstinner

#21478: mock calls don't propagate to parent (autospec)
https://bugs.python.org/issue21478  closed by xtreak

#24322: Hundreds of linker warnings on Windows
https://bugs.python.org/issue24322  closed by steve.dower

#29205: col_offset for AsyncFunctionDef AST nodes is wrong
https://bugs.python.org/issue29205  closed by serhiy.storchaka

#29440: _dbm requires -lgdbm if gdbm is built as static libraries
https://bugs.python.org/issue29440  closed by yan12125

#30831: Inconsistent or wrong documentation around Asynchronous Contex
https://bugs.python.org/issue30831  closed by yselivanov

#32129: IDLE app icon is blurry on macOS with Aqua Tk 8.6
https://bugs.python.org/issue32129  closed by terry.reedy

#32856: Optimize the `for y in [x]` idiom in comprehensions
https://bugs.python.org/issue32856  closed by serhiy.storchaka

#35198: Build issue while compiling cpp files in AIX
https://bugs.python.org/issue35198  closed by vstinner

#35435: Discourage external links to old docs.
https://bugs.python.org/issue35435  closed by mdk

#35466: Use a linked list for the ceval pending calls.
https://bugs.python.org/issue35466  closed by eric.snow

#35740: openssl version 1.1.1 need to be there in cpython-source-deps 
https://bugs.python.org/issue35740  closed by steve.dower

#35808: Let's retire pgen
https://bugs.python.org/issue35808  closed by pablogsal

#35899: '_is_sunder' function in 'enum' module fails on empty string
https://bugs.python.org/issue35899  closed by ethan.furman

#35975: Put back the ability to parse files where async/await aren't k
https://bugs.python.org/issue35975  closed by gvanrossum

#36019: test_urllib fail in s390x buildbots: http://www.example.com/
https://bugs.python.org/issue36019  closed by vstinner

#36043: FileCookieJar constructor don't accept PathLike
https://bugs.python.org/issue36043  closed by brett.cannon

#36091: clean up async generator from types module
https://bugs.python.org/issue36091  closed by scotchka

#36103: Increase shutil.COPY_BUFSIZE
https://bugs.python.org/issue36103  closed by inada.naoki

#36108: Avoid failing the build on race condition in clean
https://bugs.python.org/issue36108  closed by steve.dower

#36114: test_multiprocessing_spawn dumps core in AMD64 FreeBSD CURRENT
https://bugs.python.org/issue36114  closed by vstinner

#36116: test_multiprocessing_spawn fails on AMD64 Windows8 3.x
https://bugs.python.org/issue36116  closed by vstinner

#36139: release GIL on mmap dealloc
https://bugs.python.org/issue36139  closed by benjamin.peterson

#36140: An incorrect check in _msi.c's msidb_getsummaryinformation()
https://bugs.python.org/issue36140  closed by steve.dower

#36142: Add a new _PyPreConfig step to Python initialization to setup 
https://bugs.python.org/issue36142  closed by vstinner

#36147: [2.7] Coverity scan: Modules/_ctypes/cfield.c , Variable "resu
https://bugs.python.org/issue36147  closed by vstinner

#36149: use of uninitialised memory in cPickle.
https://bugs.python.org/issue36149  closed by twouters

#36154: Python quit unexpectedly error
https://bugs.python.org/issue36154  closed by ned.deily

#36162: error: implicit declaration of function 'sendfile' is invalid 
https://bugs.python.org/issue36162  closed by inada.naoki

#36163: same object, same method, but the is keyword return false.
https://bugs.python.org/issue36163  closed by steven.daprano

#36169: Add overlap() method to statistics.NormalDist()
https://bugs.python.org/issue36169  closed by rhettinger

#36170: posix_spawn doesn't exist in 3.7
https://bugs.python.org/issue36170  closed by ned.deily

#36171: tkinter scrollbar missing 'state' option
https://bugs.python.org/issue36171  closed by cheryl.sabella

#36173: apam
https://bugs.python.org/issue36173  closed by steven.daprano

#36175: Identity of bound methods
https://bugs.python.org/issue36175  closed by inada.naoki

#36177: test_io: test_daemon_threads_shutdown_stdout_deadlock() fails 
https://bugs.python.org/issue36177  closed by vstinner

#36178: type.__init__ called instead of cls.__init__ when inheriting f
https://bugs.python.org/issue36178  closed by steven.daprano

#36181: Add mode parameter to PurePath.write_text to allow for 'a' mod
https://bugs.python.org/issue36181  closed by serhiy.storchaka

#36183: test_gdb failed on AMD64 FreeBSD CURRENT Shared 3.7 and 3.x
https://bugs.python.org/issue36183  closed by vstinner

#36185: [EASY DOC] typo: Doc/c-api/objbuffer.rst:  There is a typo in 
https://bugs.python.org/issue36185  closed by serhiy.storchaka

#36186: [2.7] Coverity scan: Modules/linuxaudiodev.c , fd handle is no
https://bugs.python.org/issue36186  closed by vstinner

#36187: Get rid of NamedStore
https://bugs.python.org/issue36187  closed by serhiy.storchaka

#36188: Remove vestiges of Python 2 unbound methods from Python 3
https://bugs.python.org/issue36188  closed by benjamin.peterson

#36190: file object method .tell() sometimes returns large number when
https://bugs.python.org/issue36190  closed by tim.peters

#36191: pubkeys.txt contains bogus keys
https://bugs.python.org/issue36191  closed by nanjekyejoannah

#36192: Usage of `tmpnam_r` and `tempname` is dangerous, better use `m
https://bugs.python.org/issue36192  closed by matrixise

#36193: Redirected stderr not reset properly when using logging
https://bugs.python.org/issue36193  closed by vinay.sajip

#36196: sys.executable does not return python3 executable when using u
https://bugs.python.org/issue36196  closed by inada.naoki

#36197: Compilation Warning for memoryview.tobytes and _collections._t
https://bugs.python.org/issue36197  closed by vstinner

#36198: Misleading in library/sets document
https://bugs.python.org/issue36198  closed by DeadmanWalking

#36199: libzmq.dll causes uncontrollable screen flickering when access
https://bugs.python.org/issue36199  closed by eryksun

#36200: display index on IndexError
https://bugs.python.org/issue36200  closed by Aditya Shankar

#36206: re.match() not matching escaped hyphens
https://bugs.python.org/issue36206  closed by josh.r

#36209: [EASY Doc] Typo in hashlib error message
https://bugs.python.org/issue36209  closed by vstinner

#36222: ValueError: a coroutine was expected  with asyncio.run
https://bugs.python.org/issue36222  closed by vstinner

#36223: Execution sequence for print function
https://bugs.python.org/issue36223  closed by xtreak

#36224: Python quit unexpectedly error
https://bugs.python.org/issue36224  closed by ned.deily

#36228: Support coercion of complex to float/int
https://bugs.python.org/issue36228  closed by zach.ware

#36237: data_files / Install Additional Files written unclearly such t
https://bugs.python.org/issue36237  closed by jt

#36242: spam
https://bugs.python.org/issue36242  closed by SilentGhost

From brett at python.org  Fri Mar  8 13:20:52 2019
From: brett at python.org (Brett Cannon)
Date: Fri, 8 Mar 2019 10:20:52 -0800
Subject: [Python-Dev] PEP 12 updated with templates for header fields
 and sections
In-Reply-To: <CA+3bQGE+rgMxo4tYn_K-r8FfRzQ9FPourpO_5FcOFCGY2HB0vg@mail.gmail.com>
References: <CAP1=2W77qwvoqraaGCRUsA_C_Y8yr1MTLKUvG8t1s+0jRSBuug@mail.gmail.com>
 <CA+3bQGE+rgMxo4tYn_K-r8FfRzQ9FPourpO_5FcOFCGY2HB0vg@mail.gmail.com>
Message-ID: <CAP1=2W6bryXhM=nwrHoUdrvJGOwFa3gJTz4Me5v8cXfeYE3QcA@mail.gmail.com>

This is all OT for this thread. If you would like to start a new discussion
about changing the PEP process so discussion threads are tracked more
thoroughly then please do.

On Fri, Mar 8, 2019 at 12:29 AM Victor Stinner <vstinner at redhat.com> wrote:

> Hi Brett,
>
> I like to see discussions where a PEP has been discussed. There is
> the "Post History" just gives dates. The problem is that PEPs are
> discussed on 3 mailing lists: python-ideas, python-dev and
> python-committers. Maybe some PEP are now also discussed on
> discuss.python.org.
>
> Do you have any recommendation to refer to these discussions?
>
> I would like to suggest to add URLs to the first messages of all
> threads about a PEP... Well... I know that for PEP 572, this list
> would be hard to create and maybe not really useful. Maybe not add
> links to *all* threads, but only the most actives or most "useful"
> threads? The PEP author would be free to decide which threads are
> important or not ;-)
>
> The status quo (Post History header) is fine, if I really want to find
> these discussions, I'm able to find them from dates :-)
>
> Victor
>
> Le ven. 8 mars 2019 ? 01:42, Brett Cannon <brett at python.org> a ?crit :
> >
> > https://github.com/python/peps/blob/master/pep-0012.rst now has a
> complete list of header fields along with format clues for easier
> copy-and-paste use in creating a new PEP. There is also a section template
> with one-liner explanations for what each section is for so people don't
> accidentally leave anything out. They are not in a single, unified template
> to copy to partially make sure people actually read the PEP before they
> start writing. :)
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190308/8caeaf23/attachment.html>

From brett at python.org  Fri Mar  8 13:26:23 2019
From: brett at python.org (Brett Cannon)
Date: Fri, 8 Mar 2019 10:26:23 -0800
Subject: [Python-Dev] PEPs from non-core devs now need a sponsor
In-Reply-To: <46403614-6d88-d4b2-0578-5833db5255ec@Damon-Family.org>
References: <CAP1=2W7BnsYJga=WbkPLT_xHAeHMAqR4DAP6+iAt+_1fKBXLUQ@mail.gmail.com>
 <081901d4d5b0$1f72e1d0$5e58a570$@sdamon.com>
 <46403614-6d88-d4b2-0578-5833db5255ec@Damon-Family.org>
Message-ID: <CAP1=2W65gd0T82LizyBD2pNXLe6jXyEP-pU_EnxZ3g8uxw42cQ@mail.gmail.com>

On Fri, Mar 8, 2019 at 5:46 AM Richard Damon <Richard at damon-family.org>
wrote:

> On 3/8/19 8:09 AM, Alex Walters wrote:
> > I'm confused about this.  Didn't you need someone with merge permissions
> already to merge a pep into the pep repo?  Isn't this just adding a layer
> of paperwork to something that was already the case for all practical
> purposes?
>

Nope. We don't like paperwork anymore than you or anyone else.


> >
> >
> I would say the difference that before, the non-committer proposer just
> needed to convince some committer that the PEP was 'a good idea' and
> worth being discussed. Now, that person needs to persuade some committer
> that not only is it a good idea, but that it is worth the committer
> committing themselves to supporting the proposer through the process.
>

What Richard said. Before, PEP editors would make sure a PEP was basically
grammatically correct and not totally bonkers, but otherwise didn't make
judgments. Now we're saying there needs to be a core dev to help you
through the process to help make sure you will be successful and at least
*someone* thinks it's a good idea.

-Brett


>
> --
> Richard Damon
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190308/43735081/attachment.html>

From haritha.sharma at rbc.com  Fri Mar  8 15:07:18 2019
From: haritha.sharma at rbc.com (Sharma, Haritha (CWM-NR))
Date: Fri, 8 Mar 2019 20:07:18 +0000
Subject: [Python-Dev] Pythong 3.6.5 Tests failed
Message-ID: <da71cde3235d4c38a9f584873a6adc29@SXGCMB03.fg.rbc.com>

I got error message that test_dtrace and test_normalization failed. Below is the messages when I reran the test. Please let me know whether these errors are okay.

Thanks,
Haritha


[root at mrkdlvaiaas2882 Python-3.6.5]# ./python -m test -v test_normalization
== CPython 3.6.5 (default, Mar 8 2019, 14:29:13) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
== Linux-3.10.0-957.1.3.el7.x86_64-x86_64-with-redhat-7.6-Maipo little-endian
== cwd: /tmp/python/Python-3.6.5/build/test_python_123679
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 0.39 [1/1] test_normalization
test_bug_834676 (test.test_normalization.NormalizationTest) ... ok
test_main (test.test_normalization.NormalizationTest) ... skipped "Use of the 'urlfetch' resource not enabled"

----------------------------------------------------------------------
Ran 2 tests in 0.003s

OK (skipped=1)
1 test OK.

Total duration: 51 ms
Tests result: SUCCESS

[root at mrkdlvaiaas2882 Python-3.6.5]#  ./python -m test -v test_dtrace
== CPython 3.6.5 (default, Mar 8 2019, 14:29:13) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
== Linux-3.10.0-957.1.3.el7.x86_64-x86_64-with-redhat-7.6-Maipo little-endian
== cwd: /tmp/python/Python-3.6.5/build/test_python_124133
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 0.36 [1/1] test_dtrace
skipped 'dtrace(1) failed: /bin/dtrace invalid option -q\nUsage /bin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]'
skipped 'dtrace(1) failed: /bin/dtrace invalid option -q\nUsage /bin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]'
test_function_entry_return (test.test_dtrace.SystemTapNormalTests) ... FAIL
test_gc (test.test_dtrace.SystemTapNormalTests) ... FAIL
test_line (test.test_dtrace.SystemTapNormalTests) ... FAIL
test_verify_call_opcodes (test.test_dtrace.SystemTapNormalTests)
Ensure our call stack test hits all function call opcodes ... ok
test_function_entry_return (test.test_dtrace.SystemTapOptimizedTests) ... FAIL
test_gc (test.test_dtrace.SystemTapOptimizedTests) ... FAIL
test_line (test.test_dtrace.SystemTapOptimizedTests) ... FAIL
test_verify_call_opcodes (test.test_dtrace.SystemTapOptimizedTests)
Ensure our call stack test hits all function call opcodes ... ok

======================================================================
FAIL: test_function_entry_return (test.test_dtrace.SystemTapNormalTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in normalize_trace_output
    result.sort(key=lambda row: int(row[0]))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in <lambda>
    result.sort(key=lambda row: int(row[0]))
ValueError: invalid literal for int() with base 10: "semantic error: while resolving probe point: identifier 'process' at /tmp/python/Python-3.6.5/Lib/test/dtracedata/call_stack.stp:13:7"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 118, in test_function_entry_return
    self.run_case("call_stack")
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 114, in run_case
    name, optimize_python=self.optimize_python)
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 50, in run_case
    optimize_python=optimize_python))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 37, in normalize_trace_output
    "tracer produced unparseable output:\n{}".format(output)
AssertionError: tracer produced unparseable output:
semantic error: while resolving probe point: identifier 'process' at /tmp/python/Python-3.6.5/Lib/test/dtracedata/call_stack.stp:13:7
        source: probe process.mark("function__entry")
                      ^

semantic error: no match

Pass 2: analysis failed.  [man error::pass2]
Number of similar error messages suppressed: 3.
Rerun with -v to see them.


======================================================================
FAIL: test_gc (test.test_dtrace.SystemTapNormalTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in normalize_trace_output
    result.sort(key=lambda row: int(row[0]))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in <lambda>
    result.sort(key=lambda row: int(row[0]))
ValueError: invalid literal for int() with base 10: "semantic error: while resolving probe point: identifier 'process' at /tmp/python/Python-3.6.5/Lib/test/dtracedata/gc.stp:3:7"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 146, in test_gc
    self.run_case("gc")
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 114, in run_case
    name, optimize_python=self.optimize_python)
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 50, in run_case
    optimize_python=optimize_python))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 37, in normalize_trace_output
    "tracer produced unparseable output:\n{}".format(output)
AssertionError: tracer produced unparseable output:
semantic error: while resolving probe point: identifier 'process' at /tmp/python/Python-3.6.5/Lib/test/dtracedata/gc.stp:3:7
        source: probe process.mark("function__entry")
                      ^

semantic error: no match

Pass 2: analysis failed.  [man error::pass2]
Number of similar error messages suppressed: 3.
Rerun with -v to see them.


======================================================================
FAIL: test_line (test.test_dtrace.SystemTapNormalTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in normalize_trace_output
    result.sort(key=lambda row: int(row[0]))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in <lambda>
    result.sort(key=lambda row: int(row[0]))
ValueError: invalid literal for int() with base 10: "Input file '/tmp/python/Python-3.6.5/build/test_python_124133/dtracedata/line.stp' is missing."

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 149, in test_line
    self.run_case("line")
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 114, in run_case
    name, optimize_python=self.optimize_python)
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 50, in run_case
    optimize_python=optimize_python))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 37, in normalize_trace_output
    "tracer produced unparseable output:\n{}".format(output)
AssertionError: tracer produced unparseable output:
Input file '/tmp/python/Python-3.6.5/build/test_python_124133/dtracedata/line.stp' is missing.
Pass 1: parse failed.  [man error::pass1]


======================================================================
FAIL: test_function_entry_return (test.test_dtrace.SystemTapOptimizedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in normalize_trace_output
    result.sort(key=lambda row: int(row[0]))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in <lambda>
    result.sort(key=lambda row: int(row[0]))
ValueError: invalid literal for int() with base 10: "semantic error: while resolving probe point: identifier 'process' at /tmp/python/Python-3.6.5/Lib/test/dtracedata/call_stack.stp:13:7"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 118, in test_function_entry_return
    self.run_case("call_stack")
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 114, in run_case
    name, optimize_python=self.optimize_python)
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 50, in run_case
    optimize_python=optimize_python))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 37, in normalize_trace_output
    "tracer produced unparseable output:\n{}".format(output)
AssertionError: tracer produced unparseable output:
semantic error: while resolving probe point: identifier 'process' at /tmp/python/Python-3.6.5/Lib/test/dtracedata/call_stack.stp:13:7
        source: probe process.mark("function__entry")
                      ^

semantic error: no match

Pass 2: analysis failed.  [man error::pass2]
Number of similar error messages suppressed: 3.
Rerun with -v to see them.


======================================================================
FAIL: test_gc (test.test_dtrace.SystemTapOptimizedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in normalize_trace_output
    result.sort(key=lambda row: int(row[0]))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in <lambda>
    result.sort(key=lambda row: int(row[0]))
ValueError: invalid literal for int() with base 10: "semantic error: while resolving probe point: identifier 'process' at /tmp/python/Python-3.6.5/Lib/test/dtracedata/gc.stp:3:7"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 146, in test_gc
    self.run_case("gc")
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 114, in run_case
    name, optimize_python=self.optimize_python)
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 50, in run_case
    optimize_python=optimize_python))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 37, in normalize_trace_output
    "tracer produced unparseable output:\n{}".format(output)
AssertionError: tracer produced unparseable output:
semantic error: while resolving probe point: identifier 'process' at /tmp/python/Python-3.6.5/Lib/test/dtracedata/gc.stp:3:7
        source: probe process.mark("function__entry")
                      ^

semantic error: no match

Pass 2: analysis failed.  [man error::pass2]
Number of similar error messages suppressed: 3.
Rerun with -v to see them.


======================================================================
FAIL: test_line (test.test_dtrace.SystemTapOptimizedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in normalize_trace_output
    result.sort(key=lambda row: int(row[0]))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 32, in <lambda>
    result.sort(key=lambda row: int(row[0]))
ValueError: invalid literal for int() with base 10: "Input file '/tmp/python/Python-3.6.5/build/test_python_124133/dtracedata/line.stp' is missing."

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 149, in test_line
    self.run_case("line")
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 114, in run_case
    name, optimize_python=self.optimize_python)
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 50, in run_case
    optimize_python=optimize_python))
  File "/tmp/python/Python-3.6.5/Lib/test/test_dtrace.py", line 37, in normalize_trace_output
    "tracer produced unparseable output:\n{}".format(output)
AssertionError: tracer produced unparseable output:
Input file '/tmp/python/Python-3.6.5/build/test_python_124133/dtracedata/line.stp' is missing.
Pass 1: parse failed.  [man error::pass1]


----------------------------------------------------------------------
Ran 8 tests in 4.836s

FAILED (failures=6, skipped=2)
test test_dtrace failed
test_dtrace failed

1 test failed:
    test_dtrace

Total duration: 5 sec
Tests result: FAILURE
_______________________________________________________________________
If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference.  

Si vous recevez ce courriel par erreur, veuillez en aviser l'exp?diteur imm?diatement, par retour de courriel ou par un autre moyen. Vous avez accept? de recevoir le(s) document(s) ci-joint(s) par voie ?lectronique ? l'adresse courriel indiqu?e ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190308/4c575738/attachment-0001.html>

From larry at hastings.org  Sun Mar 10 01:50:43 2019
From: larry at hastings.org (Larry Hastings)
Date: Sat, 9 Mar 2019 22:50:43 -0800
Subject: [Python-Dev] Last-minute request: please backport bpo-33329 fix
 to 3.4 and 3.5
In-Reply-To: <CA+LW3i3jX06H5XbPSnFxcLmL+fQX9Z9yFPMUjhrceR6cKcceSw@mail.gmail.com>
References: <eca9d6f2-14a3-0a17-2730-8d74cbb8fa84@hastings.org>
 <CA+LW3i3jX06H5XbPSnFxcLmL+fQX9Z9yFPMUjhrceR6cKcceSw@mail.gmail.com>
Message-ID: <3d30f424-b3e2-346d-8f15-c77282c9d01a@hastings.org>

On 3/4/19 2:29 AM, Joni Orponen wrote:
> On Sat, Mar 2, 2019 at 7:08 AM Larry Hastings <larry at hastings.org 
> <mailto:larry at hastings.org>> wrote:
>
>     This bug in bpo-33329:
>
>         https://bugs.python.org/issue33329
>
>
> This is also potentially affecting PGO builds of 2.7 on Debian Buster 
> with GCC. Somehow building with Clang is fine.
>
> Does the configure time choice of compiler make a difference here for 
> 3.4 and 3.5?


I don't know.? I only build with the default compiler on my machine, 
gcc.? (My machine is Linux, 64-bit.)? It'd be swell if you tried the RCs 
with clang!


//arry/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190309/bb566611/attachment.html>

From j.orponen at 4teamwork.ch  Mon Mar 11 08:05:13 2019
From: j.orponen at 4teamwork.ch (Joni Orponen)
Date: Mon, 11 Mar 2019 13:05:13 +0100
Subject: [Python-Dev] Last-minute request: please backport bpo-33329 fix
 to 3.4 and 3.5
In-Reply-To: <3d30f424-b3e2-346d-8f15-c77282c9d01a@hastings.org>
References: <eca9d6f2-14a3-0a17-2730-8d74cbb8fa84@hastings.org>
 <CA+LW3i3jX06H5XbPSnFxcLmL+fQX9Z9yFPMUjhrceR6cKcceSw@mail.gmail.com>
 <3d30f424-b3e2-346d-8f15-c77282c9d01a@hastings.org>
Message-ID: <CA+LW3i0XyNiZJDOh9a7VRcHb499k6+FsRra0iGZUxW2A5GQupA@mail.gmail.com>

On Sun, Mar 10, 2019 at 7:50 AM Larry Hastings <larry at hastings.org> wrote:

> On 3/4/19 2:29 AM, Joni Orponen wrote:
>
> On Sat, Mar 2, 2019 at 7:08 AM Larry Hastings <larry at hastings.org> wrote:
>
>> This bug in bpo-33329:
>>
>> https://bugs.python.org/issue33329
>>
>>
>> This is also potentially affecting PGO builds of 2.7 on Debian Buster
> with GCC. Somehow building with Clang is fine.
>
> Does the configure time choice of compiler make a difference here for 3.4
> and 3.5?
>
> I don't know.  I only build with the default compiler on my machine, gcc.
> (My machine is Linux, 64-bit.)  It'd be swell if you tried the RCs with
> clang!
>
> */arry*
>
I've only risen this as a newer glibc was implicated as a potential source
of trouble, I have a lab / 'canary in a coal mine' server on a prerelease
OS version (Debian Buster is currently in feature freeze ~ release
candidate stage) and I've not noticed anything to have happened with a
newer glibc in relation to building Python, but I've not had success
building a PGO build of 2.7.15 with the GCC shipped at that time, but
building with Clang working fine.

I'd thus be inclined to point at digging up relevant changes in GCC than in
glibc. Currently 2.7.16 compiles just fine for me on both GCC and Clang.
2.7.15 still does not compile fine for me on GCC.

FWIW - both Python 3 RCs from above successfully complete a PGO build for
me (and thus implicitly also pass the regression tests) with Clang. They
both get stuck with GCC.

$ cat /etc/debian_version
buster/sid

$ aptitude show base-files
Package: base-files
Version: 10.1
Essential: yes
State: installed

$ aptitude show libc6
Package: libc6
Version: 2.28-8
State: installed

$ aptitude show clang
Package: clang
Version: 1:7.0-47
State: installed

$ clang --version
clang version 7.0.1-8 (tags/RELEASE_701/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

$ aptitude show gcc
Package: gcc
Version: 4:8.3.0-1
State: installed

$ gcc --version
gcc (Debian 8.3.0-2) 8.3.0

And for anyone wishing to try such things out for themselves, one can
simply set the CC environment variable for the autotools provided
./configure script.

See ./configure --help for any further details and instructions.

-- 
Joni Orponen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190311/b1d072ed/attachment.html>

From ncoghlan at gmail.com  Mon Mar 11 08:23:13 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Mar 2019 22:23:13 +1000
Subject: [Python-Dev] Initial updates to PEP 1 for Steering Council based
 governance
Message-ID: <CADiSq7e4TFv2XquDGasDZ72n2NFGs4zjfRnaFEE8qO0mrVhKyw@mail.gmail.com>

Hi folks,

Brett recently posted about the update to PEP 1 that added the PEP Sponsor
role, but folks may not have noticed that we've also made the amendments
needed to allow the PEP decision making process to restart:
https://github.com/python/peps/pull/896/files

This is the smallest change to PEP 1 that we consider potentially viable:
handling all PEPs through the BDFL-Delegate model, with the Steering
Council's primary involvement being to appoint the delegates (or accept
their volunteering), once a PEP has reached the point of being actively
reviewed.

We'll also act as an ongoing consulting resource for anyone that takes on
the BD role for a PEP.

We're currently working through the follow-up activity, which is to review
the list of Draft PEPs, and work out which ones are at a point where
assigning a BDFL-Delegate makes sense.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190311/d5cd5e33/attachment.html>

From liu.denton at gmail.com  Mon Mar 11 05:56:58 2019
From: liu.denton at gmail.com (Denton Liu)
Date: Mon, 11 Mar 2019 02:56:58 -0700
Subject: [Python-Dev] [bpo-35155] Requesting a review
In-Reply-To: <20190227064056.GA2611@archbookpro.localdomain>
References: <20190212101455.GA29427@archbookpro.localdomain>
 <20190227064056.GA2611@archbookpro.localdomain>
Message-ID: <20190311095658.GA785@archbookpro.localdomain>

On Tue, Feb 26, 2019 at 10:40:56PM -0800, Denton Liu wrote:
> On Tue, Feb 12, 2019 at 02:14:55AM -0800, Denton Liu wrote:
> > Hello all,
> > 
> > A couple months back, I reported bpo-35155[1] and I submitted a PR for
> > consideration[2]. After a couple of reviews, it seems like progress has
> > stalled. Would it be possible for someone to review this?
> > 
> > Thanks,
> > 
> > Denton
> > 
> > [1]: https://bugs.python.org/issue35155
> > [2]: https://github.com/python/cpython/pull/10313
> 
> Thanks for the comments and help on the PR!
> 
> It seems like progress on this change has stalled again. If there aren't
> anymore comments, I believe that this PR is ready to be merged.
> 
> Thanks,
> 
> Denton

Hello all,

It seems like the reviews on the PR are still stalled. Would it be
possible for someone to take a look? If there aren't anymore comments, I
believe that the PR is ready to be merged.

Thanks,

Denton

From vstinner at redhat.com  Mon Mar 11 17:02:32 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Mon, 11 Mar 2019 22:02:32 +0100
Subject: [Python-Dev] [python-committers] Last-minute request: please
 backport bpo-33329 fix to 3.4 and 3.5
In-Reply-To: <CA+LW3i0XyNiZJDOh9a7VRcHb499k6+FsRra0iGZUxW2A5GQupA@mail.gmail.com>
References: <eca9d6f2-14a3-0a17-2730-8d74cbb8fa84@hastings.org>
 <CA+LW3i3jX06H5XbPSnFxcLmL+fQX9Z9yFPMUjhrceR6cKcceSw@mail.gmail.com>
 <3d30f424-b3e2-346d-8f15-c77282c9d01a@hastings.org>
 <CA+LW3i0XyNiZJDOh9a7VRcHb499k6+FsRra0iGZUxW2A5GQupA@mail.gmail.com>
Message-ID: <CA+3bQGHn0kQC4MTkuHV1-s84XWri2_DxHNDzKTSrGj2QNXqq3g@mail.gmail.com>

I don't think that a thread about a release is the right place to discuss a
bug. Please open an issue at bugs.python.org ?

Victor

Le lundi 11 mars 2019, Joni Orponen <j.orponen at 4teamwork.ch> a ?crit :
> On Sun, Mar 10, 2019 at 7:50 AM Larry Hastings <larry at hastings.org> wrote:
>>
>> On 3/4/19 2:29 AM, Joni Orponen wrote:
>>
>> On Sat, Mar 2, 2019 at 7:08 AM Larry Hastings <larry at hastings.org> wrote:
>>>
>>> This bug in bpo-33329:
>>>
>>> https://bugs.python.org/issue33329
>>>
>> This is also potentially affecting PGO builds of 2.7 on Debian Buster
with GCC. Somehow building with Clang is fine.
>> Does the configure time choice of compiler make a difference here for
3.4 and 3.5?
>>
>> I don't know.  I only build with the default compiler on my machine,
gcc.  (My machine is Linux, 64-bit.)  It'd be swell if you tried the RCs
with clang!
>>
>> /arry
>
> I've only risen this as a newer glibc was implicated as a potential
source of trouble, I have a lab / 'canary in a coal mine' server on a
prerelease OS version (Debian Buster is currently in feature freeze ~
release candidate stage) and I've not noticed anything to have happened
with a newer glibc in relation to building Python, but I've not had success
building a PGO build of 2.7.15 with the GCC shipped at that time, but
building with Clang working fine.
> I'd thus be inclined to point at digging up relevant changes in GCC than
in glibc. Currently 2.7.16 compiles just fine for me on both GCC and Clang.
2.7.15 still does not compile fine for me on GCC.
> FWIW - both Python 3 RCs from above successfully complete a PGO build for
me (and thus implicitly also pass the regression tests) with Clang. They
both get stuck with GCC.
> $ cat /etc/debian_version
> buster/sid
> $ aptitude show base-files
> Package: base-files
> Version: 10.1
> Essential: yes
> State: installed
> $ aptitude show libc6
> Package: libc6
> Version: 2.28-8
> State: installed
> $ aptitude show clang
> Package: clang
> Version: 1:7.0-47
> State: installed
> $ clang --version
> clang version 7.0.1-8 (tags/RELEASE_701/final)
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /usr/bin
> $ aptitude show gcc
> Package: gcc
> Version: 4:8.3.0-1
> State: installed
> $ gcc --version
> gcc (Debian 8.3.0-2) 8.3.0
> And for anyone wishing to try such things out for themselves, one can
simply set the CC environment variable for the autotools provided
./configure script.
> See ./configure --help for any further details and instructions.
> --
> Joni Orponen

-- 
Night gathers, and now my watch begins. It shall not end until my death.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190311/3b102a40/attachment.html>

From nas-python at arctrix.com  Mon Mar 11 17:56:29 2019
From: nas-python at arctrix.com (Neil Schemenauer)
Date: Mon, 11 Mar 2019 15:56:29 -0600
Subject: [Python-Dev] Register-based VM [Was: Possible performance
 regression]
In-Reply-To: <CA+3bQGEOc53H3VwnAj6Dy1o+txoc244AFnZJKTcWvQDJCg4G3w@mail.gmail.com>
References: <20190225115452.30ed11bb@fsol>
 <9D28FA05-1FDA-43E0-9CD1-DCDDBCE05A03@gmail.com>
 <CALFfu7C=xpvRwbhnnB4fDEFxY7eezV0QHrxUEoX9Tkq3SgGexg@mail.gmail.com>
 <CALFfu7CT9KpV-6bDgi9pPeX0vESu8fOcRT=fkB83jQCYLXKD3g@mail.gmail.com>
 <20190226193512.m5wzrecaxt3yfm7l@python.ca>
 <CA+3bQGGXjMAJ6FQb=65OAjRW+p783ePzGFTcaHWLONZNFs7KJA@mail.gmail.com>
 <20190226205841.wo6a6bc65igmxpk3@python.ca>
 <CA+3bQGGfZ6nRkKS==ArkK8FcZe_8xdrV_bmcaoHOWPpWKVzq5w@mail.gmail.com>
 <5C75BE81.4060209@canterbury.ac.nz>
 <CA+3bQGEOc53H3VwnAj6Dy1o+txoc244AFnZJKTcWvQDJCg4G3w@mail.gmail.com>
Message-ID: <20190311215629.y2j6szbqzzi7jcds@python.ca>

On 2019-02-27, Victor Stinner wrote:
> The compiler begins with using static single assignment form (SSA) but
> then uses a register allocator to reduce the number of used registers.
> Usually, at the end you have less than 5 registers for a whole
> function.

In case anyone is interested on working on this, I dug up some
discussion from years ago.  Advice from Tim Peters:

    [Python-Dev] Rattlesnake progress
    https://mail.python.org/pipermail/python-dev/2002-February/020172.html
    https://mail.python.org/pipermail/python-dev/2002-February/020182.html
    https://mail.python.org/pipermail/python-dev/2002-February/020146.html

Doing a prototype register-based compiler in Python seems like a
good idea.  Using the 'compiler' package would give you a good
start.  I think this is the most recent version of that package:

    https://github.com/pfalcon/python-compiler

Based on a little poking around, I think it has not been updated for
the 16-bit word code.  Shouldn't be too hard to make it work though.

I was thinking about the code format on the weekend.  Using
three-register opcodes seems a good idea.   We could could retain
the 16-bit word code format.  For opcodes that use three registers,
use a second word for the last two registers.  I.e.

    <8 bit opcode><8 bit register #>
    <8 bit register #><8 bit register #>

Limit the number of registers to 256.  If you run out, just push and
pop from stack.  You want to keep the instruction decode path in the
evaluation loop simple and not confuse the CPU branch predictor.

Regards,

  Neil

From mertz at gnosis.cx  Mon Mar 11 18:03:26 2019
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 11 Mar 2019 18:03:26 -0400
Subject: [Python-Dev] Register-based VM [Was: Possible performance
 regression]
In-Reply-To: <20190311215629.y2j6szbqzzi7jcds@python.ca>
References: <20190225115452.30ed11bb@fsol>
 <9D28FA05-1FDA-43E0-9CD1-DCDDBCE05A03@gmail.com>
 <CALFfu7C=xpvRwbhnnB4fDEFxY7eezV0QHrxUEoX9Tkq3SgGexg@mail.gmail.com>
 <CALFfu7CT9KpV-6bDgi9pPeX0vESu8fOcRT=fkB83jQCYLXKD3g@mail.gmail.com>
 <20190226193512.m5wzrecaxt3yfm7l@python.ca>
 <CA+3bQGGXjMAJ6FQb=65OAjRW+p783ePzGFTcaHWLONZNFs7KJA@mail.gmail.com>
 <20190226205841.wo6a6bc65igmxpk3@python.ca>
 <CA+3bQGGfZ6nRkKS==ArkK8FcZe_8xdrV_bmcaoHOWPpWKVzq5w@mail.gmail.com>
 <5C75BE81.4060209@canterbury.ac.nz>
 <CA+3bQGEOc53H3VwnAj6Dy1o+txoc244AFnZJKTcWvQDJCg4G3w@mail.gmail.com>
 <20190311215629.y2j6szbqzzi7jcds@python.ca>
Message-ID: <CAEbHw4ZsBtA5WukU1-VmcdM+-W4BbCrUsz6Op0ON39LoXXWDXg@mail.gmail.com>

Parrot got rather further along than rattlesnake as a register based VM. I
don't think it every really beat CPython in speed though.

http://parrot.org/

On Mon, Mar 11, 2019, 5:57 PM Neil Schemenauer <nas-python at arctrix.com>
wrote:

> On 2019-02-27, Victor Stinner wrote:
> > The compiler begins with using static single assignment form (SSA) but
> > then uses a register allocator to reduce the number of used registers.
> > Usually, at the end you have less than 5 registers for a whole
> > function.
>
> In case anyone is interested on working on this, I dug up some
> discussion from years ago.  Advice from Tim Peters:
>
>     [Python-Dev] Rattlesnake progress
>     https://mail.python.org/pipermail/python-dev/2002-February/020172.html
>     https://mail.python.org/pipermail/python-dev/2002-February/020182.html
>     https://mail.python.org/pipermail/python-dev/2002-February/020146.html
>
> Doing a prototype register-based compiler in Python seems like a
> good idea.  Using the 'compiler' package would give you a good
> start.  I think this is the most recent version of that package:
>
>     https://github.com/pfalcon/python-compiler
>
> Based on a little poking around, I think it has not been updated for
> the 16-bit word code.  Shouldn't be too hard to make it work though.
>
> I was thinking about the code format on the weekend.  Using
> three-register opcodes seems a good idea.   We could could retain
> the 16-bit word code format.  For opcodes that use three registers,
> use a second word for the last two registers.  I.e.
>
>     <8 bit opcode><8 bit register #>
>     <8 bit register #><8 bit register #>
>
> Limit the number of registers to 256.  If you run out, just push and
> pop from stack.  You want to keep the instruction decode path in the
> evaluation loop simple and not confuse the CPU branch predictor.
>
> Regards,
>
>   Neil
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190311/55e4f538/attachment.html>

From g.rodola at gmail.com  Mon Mar 11 19:35:01 2019
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 12 Mar 2019 00:35:01 +0100
Subject: [Python-Dev] (Licensing question) backport of shutil.copyfile()
 functionality
Message-ID: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>

Hello,
some time ago I contributed a couple of patches to speedup shutil.copy*()
functions:
https://bugs.python.org/issue33671
https://bugs.python.org/issue33695
I would like to backport both functionalities so that they can be used on
Python 2.7 and <3.8 and put it on PYPI. In order to do so I will basically
have to copy some parts of shutil module (copytree() function + the
unit-tests I added in BPO-33671 and a couple of other things). Are there
constraints regarding this in terms of license? Am I supposed to use GPL?
(I was thinking about using MIT)

Note: in this package called "zerocopy" I will probably want to expose
other functionalities such as tee(), splice() and CopyFileEx and
TransmitFile on Windows, so it's probably gonna be half a backport and half
a brand new project.

Thanks.

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190312/a40db889/attachment.html>

From v+python at g.nevcal.com  Mon Mar 11 22:00:49 2019
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Mon, 11 Mar 2019 19:00:49 -0700
Subject: [Python-Dev] (Licensing question) backport of shutil.copyfile()
 functionality
In-Reply-To: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
References: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
Message-ID: <0c8dd528-1224-38d7-db20-c5d1bb4ee343@g.nevcal.com>

On 3/11/2019 4:35 PM, Giampaolo Rodola' wrote:
> Hello,
> some time ago I contributed a couple of patches to speedup 
> shutil.copy*() functions:
> https://bugs.python.org/issue33671
> https://bugs.python.org/issue33695
> I would like to backport both functionalities so that they can be used 
> on Python 2.7 and <3.8 and put it on PYPI. In order to do so I will 
> basically have to copy some parts of shutil module (copytree() 
> function + the unit-tests I added in BPO-33671 and a couple of other 
> things). Are there constraints regarding this in terms of license? Am 
> I supposed to use GPL? (I was thinking about using MIT)
>
> Note: in this package called "zerocopy" I will probably want to expose 
> other functionalities such as tee(), splice() and CopyFileEx and 
> TransmitFile on Windows, so it's probably gonna be half a backport and 
> half a brand new project.
>
> Thanks.

Thanks for the contributions. I don't know about the licensing.

I wonder if you should make two packages, though... one just exactly a 
backport of the shutil speedups, and the second containing the new 
functionalities.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190311/dd7b7ffa/attachment.html>

From greg at krypto.org  Mon Mar 11 22:45:03 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 11 Mar 2019 19:45:03 -0700
Subject: [Python-Dev] (Licensing question) backport of shutil.copyfile()
 functionality
In-Reply-To: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
References: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
Message-ID: <CAGE7PNKzSDAyKBN1oNJkX3gOPAfX5jD89b5oKiRPEhUon97oLQ@mail.gmail.com>

Things in the standard library are already covered by the PSF license so
that is what should be kept on backports from the stdlib to earlier
versions.

I do recommend keeping your backported stuff and new functionality separate
(separate packages ideally, but that'll depend on how intertwined things
are) as Glenn also suggested so that your additional functionality can work
on modern Python interpreters without requiring your backported package.

Keep things labelled backports strictly API compatible with a fixed version
of the stdlib.  It lets people using them migrate _to_ the stdlib easier in
the future.

my 2 cents as the subprocess32 maintainer,
-gps

On Mon, Mar 11, 2019 at 4:35 PM Giampaolo Rodola' <g.rodola at gmail.com>
wrote:

> Hello,
> some time ago I contributed a couple of patches to speedup shutil.copy*()
> functions:
> https://bugs.python.org/issue33671
> https://bugs.python.org/issue33695
> I would like to backport both functionalities so that they can be used on
> Python 2.7 and <3.8 and put it on PYPI. In order to do so I will basically
> have to copy some parts of shutil module (copytree() function + the
> unit-tests I added in BPO-33671 and a couple of other things). Are there
> constraints regarding this in terms of license? Am I supposed to use GPL?
> (I was thinking about using MIT)
>
> Note: in this package called "zerocopy" I will probably want to expose
> other functionalities such as tee(), splice() and CopyFileEx and
> TransmitFile on Windows, so it's probably gonna be half a backport and half
> a brand new project.
>
> Thanks.
>
> --
> Giampaolo - http://grodola.blogspot.com
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190311/b05e4370/attachment.html>

From songofacandy at gmail.com  Mon Mar 11 22:54:31 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Tue, 12 Mar 2019 11:54:31 +0900
Subject: [Python-Dev] (Licensing question) backport of shutil.copyfile()
 functionality
In-Reply-To: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
References: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
Message-ID: <CAEfz+TyGaHv3DaAbxMfz5dyT2HsrmSp4Cx2S5Cv3h4cyMiOhKA@mail.gmail.com>

On Tue, Mar 12, 2019 at 8:38 AM Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>
> Hello,
> some time ago I contributed a couple of patches to speedup shutil.copy*() functions:
> https://bugs.python.org/issue33671
> https://bugs.python.org/issue33695
> I would like to backport both functionalities so that they can be used on Python 2.7 and <3.8 and put it on PYPI. In order to do so I will basically have to copy some parts of shutil module (copytree() function + the unit-tests I added in BPO-33671 and a couple of other things). Are there constraints regarding this in terms of license? Am I supposed to use GPL? (I was thinking about using MIT)
>

First of all, I'm sorry if I'm wrong.  I'm not lawyer.

You can use both of GPL and MIT.  Users can use your package under it.

On the other hand, when you publish your package, *you* should follow
PSF license.
Read this.  https://docs.python.org/3/license.html

"""
3. In the event Licensee prepares a derivative work that is based on or
   incorporates Python 3.7.2 or any part thereof, and wants to make the
   derivative work available to others as provided herein, then Licensee hereby
   agrees to include in any such work a brief summary of the changes
made to Python
   3.7.2.
"""

As you can see, PSF license doesn't force you to use PSF license. (not
"copyleft")
All you need is "brief summary of the changes made to Python".

I think it can be:

# Copyright 2001- Python Software Foundation.
# Copyright 2019- Your Name
#
# Following code is from Python standard library (Lib/shutil.py)
#
# Changelog:
#   * ...

Regards,
-- 
Inada Naoki  <songofacandy at gmail.com>

From tjreedy at udel.edu  Tue Mar 12 01:27:14 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 12 Mar 2019 01:27:14 -0400
Subject: [Python-Dev] (Licensing question) backport of shutil.copyfile()
 functionality
In-Reply-To: <CAEfz+TyGaHv3DaAbxMfz5dyT2HsrmSp4Cx2S5Cv3h4cyMiOhKA@mail.gmail.com>
References: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
 <CAEfz+TyGaHv3DaAbxMfz5dyT2HsrmSp4Cx2S5Cv3h4cyMiOhKA@mail.gmail.com>
Message-ID: <q67fvk$1154$1@blaine.gmane.org>

On 3/11/2019 10:54 PM, Inada Naoki wrote:

>> Hello,
>> some time ago I contributed a couple of patches to speedup shutil.copy*() functions:
>> https://bugs.python.org/issue33671
>> https://bugs.python.org/issue33695

You retain copyright on the code you contributed.

>> I would like to backport both functionalities so that they can be used on Python 2.7 and <3.8 and put it on PYPI. In order to do so I will basically have to copy some parts of shutil module (copytree() function + the unit-tests I added in BPO-33671 and a couple of other things). Are there constraints regarding this in terms of license? Am I supposed to use GPL? (I was thinking about using MIT)
>>
> 
> First of all, I'm sorry if I'm wrong.  I'm not lawyer.
> 
> You can use both of GPL and MIT.  Users can use your package under it.
> 
> On the other hand, when you publish your package, *you* should follow
> PSF license.
> Read this.  https://docs.python.org/3/license.html
> 
> """
> 3. In the event Licensee prepares a derivative work that is based on or
>     incorporates Python 3.7.2 or any part thereof, and wants to make the
>     derivative work available to others as provided herein, then Licensee hereby
>     agrees to include in any such work a brief summary of the changes
> made to Python
>     3.7.2.
> """
> 
> As you can see, PSF license doesn't force you to use PSF license. (not
> "copyleft")

In fact, the PSF lawyer says one should not use the 'PSF license' as it 
is specilized for the PSF and Python.

> All you need is "brief summary of the changes made to Python".
> 
> I think it can be:
> 
> # Copyright 2001- Python Software Foundation.
> # Copyright 2019- Your Name
> #
> # Following code is from Python standard library (Lib/shutil.py)
> #
> # Changelog:
> #   * ...


-- 
Terry Jan Reedy


From mariatta at python.org  Tue Mar 12 12:16:15 2019
From: mariatta at python.org (Mariatta)
Date: Tue, 12 Mar 2019 09:16:15 -0700
Subject: [Python-Dev] Reminder: Python Language Summit 2019 Signup
Message-ID: <CAGbohnbq9ss+bsegfOYL3qMgYeevSF2=6z2TiX06jd5zdS9zeQ@mail.gmail.com>

Just another reminder that sign up is still open for Python Language Summit
(until March 21st, 2019)

TL;DR:

When: Wednesday, May 1, 2019, 10am?4pm
Where: Huntington Convention Center, Cleveland, Ohio
Apply: Attendance sign up form <https://goo.gl/forms/pexfOGDjpV0BWMer2>
(before March 21, 2019)
Speak: Topic submission form <https://goo.gl/forms/8l7ITEKOjlWkBjcl1>
(before March 21, 2019)
You will be notified by April 15, 2019
Co-chairs: Mariatta and ?ukasz
Blogger: A. Jesse Jiryu Davis

All language summit attendees are also invited for dinner with PyCon staff
and volunteers right after the summit. Details will be forwarded once we
confirm your attendance. Since I don't want to miss the dinner, we will be
more strict with timing and not go overtime.

So far, we have a number of interesting topics proposed by core developers
as well as by members of the wider Python community. Some of the proposed
topic are: PEP 581, async REPL, Python 3.9 release cadence, core Python
mentorship, manylinux, etc.

If you have any questions, the Users section of our Discourse
instance (https://discuss.python.org/c/users) is the best place to ask.
For private communication, write to mariatta at python.org and/or
lukasz at python.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190312/2e333e82/attachment.html>

From g.rodola at gmail.com  Tue Mar 12 17:55:09 2019
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 12 Mar 2019 22:55:09 +0100
Subject: [Python-Dev] (Licensing question) backport of shutil.copyfile()
 functionality
In-Reply-To: <0c8dd528-1224-38d7-db20-c5d1bb4ee343@g.nevcal.com>
References: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
 <0c8dd528-1224-38d7-db20-c5d1bb4ee343@g.nevcal.com>
Message-ID: <CAFYqXL_YXT=-hbQ3XQKuAVq8aZeyeRXRxLp2PWvew6EymtAD4A@mail.gmail.com>

On Tue, Mar 12, 2019 at 3:01 AM Glenn Linderman <v+python at g.nevcal.com>
wrote:

> On 3/11/2019 4:35 PM, Giampaolo Rodola' wrote:
>
> Hello,
> some time ago I contributed a couple of patches to speedup shutil.copy*()
> functions:
> https://bugs.python.org/issue33671
> https://bugs.python.org/issue33695
> I would like to backport both functionalities so that they can be used on
> Python 2.7 and <3.8 and put it on PYPI. In order to do so I will basically
> have to copy some parts of shutil module (copytree() function + the
> unit-tests I added in BPO-33671 and a couple of other things). Are there
> constraints regarding this in terms of license? Am I supposed to use GPL?
> (I was thinking about using MIT)
>
> Note: in this package called "zerocopy" I will probably want to expose
> other functionalities such as tee(), splice() and CopyFileEx and
> TransmitFile on Windows, so it's probably gonna be half a backport and half
> a brand new project.
>
> Thanks.
>
>
> Thanks for the contributions. I don't know about the licensing.
>
> I wonder if you should make two packages, though... one just exactly a
> backport of the shutil speedups, and the second containing the new
> functionalities.
>

That was my initial thought as well (a "backports.shutil_copy" module
targeting copy* functions only), but (especially after playing with this
today) I think I have something a bit more ambitious in mind. I'm currently
experimenting with different things which could be baked in a third-party
lib and possibly contributed back to Python later on:

1) on OSX we could use f/copyfile() syscall to copy file attrs/metadata;
that may be useful to speedup shutil.copystat() and shutil.copymode()
2) copytree() on OSX could take advantage of f/copyfile() +
COPYFILE_RECURSIVE (which is probably too platform-specific for inclusion)
3) on Linux we could use copy_file_range() as a replacement for
os.sendfile() in shutil.copyfile() (it's supposed to be faster)
4) on Linux ioctl() + FICLONE could be used to implement CoW (copy on
write) instantaneous copies, and could be added as shutil.cow_copyfile()
(haven't look into Windows yet)
5) I was thinking about backporting socket.socket.sendfile() as well, which
uses os.sendfile() on POSIX but not TransmitFile on Windows (asyncio does
though)
6)  another idea (but I'm not sure if it's possible, as I still have to dig
into that) is a new socket recvfile() function boosted up by tee() /
splice() on Linux, which maybe could be contributed back as
socket.socket.recvfile()

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190312/abd71102/attachment.html>

From nad at python.org  Tue Mar 12 19:28:52 2019
From: nad at python.org (Ned Deily)
Date: Tue, 12 Mar 2019 19:28:52 -0400
Subject: [Python-Dev] [RELEASE] Python 3.7.3rc1 is now available for testing.
Message-ID: <DC43F0EE-3377-45B4-B84B-056807BDA332@python.org>

Python 3.7.3rc1 is now available for testing. 3.7.3rc1 is the release
preview of the next maintenance release of Python 3.7, the latest
feature release of Python. Assuming no critical problems are found
prior to 2019-03-25, no code changes are planned between now and the
final release. This release candidate is intended to give you the
opportunity to test the new security and bug fixes in 3.7.3. We
strongly encourage you to test your projects and report issues found
to bugs.python.org as soon as possible. Please keep in mind that this
is a preview release and, thus, its use is not recommended for
production environments.

You can find the release files, a link to the changelog, and more
information here:
    https://www.python.org/downloads/release/python-373rc1/

--
  Ned Deily
  nad at python.org -- []


From greg at krypto.org  Tue Mar 12 20:32:35 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 12 Mar 2019 17:32:35 -0700
Subject: [Python-Dev] (Licensing question) backport of shutil.copyfile()
 functionality
In-Reply-To: <CAFYqXL_YXT=-hbQ3XQKuAVq8aZeyeRXRxLp2PWvew6EymtAD4A@mail.gmail.com>
References: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
 <0c8dd528-1224-38d7-db20-c5d1bb4ee343@g.nevcal.com>
 <CAFYqXL_YXT=-hbQ3XQKuAVq8aZeyeRXRxLp2PWvew6EymtAD4A@mail.gmail.com>
Message-ID: <CAGE7PN+37ccbSRfX9ODUQsQQMvJjqTiJUWh7pnyzNuSTH-ctTQ@mail.gmail.com>

On Tue, Mar 12, 2019 at 2:55 PM Giampaolo Rodola' <g.rodola at gmail.com>
wrote:

>
>
> On Tue, Mar 12, 2019 at 3:01 AM Glenn Linderman <v+python at g.nevcal.com>
> wrote:
>
>> On 3/11/2019 4:35 PM, Giampaolo Rodola' wrote:
>>
>> Hello,
>> some time ago I contributed a couple of patches to speedup shutil.copy*()
>> functions:
>> https://bugs.python.org/issue33671
>> https://bugs.python.org/issue33695
>> I would like to backport both functionalities so that they can be used on
>> Python 2.7 and <3.8 and put it on PYPI. In order to do so I will basically
>> have to copy some parts of shutil module (copytree() function + the
>> unit-tests I added in BPO-33671 and a couple of other things). Are there
>> constraints regarding this in terms of license? Am I supposed to use GPL?
>> (I was thinking about using MIT)
>>
>> Note: in this package called "zerocopy" I will probably want to expose
>> other functionalities such as tee(), splice() and CopyFileEx and
>> TransmitFile on Windows, so it's probably gonna be half a backport and half
>> a brand new project.
>>
>> Thanks.
>>
>>
>> Thanks for the contributions. I don't know about the licensing.
>>
>> I wonder if you should make two packages, though... one just exactly a
>> backport of the shutil speedups, and the second containing the new
>> functionalities.
>>
>
> That was my initial thought as well (a "backports.shutil_copy" module
> targeting copy* functions only), but (especially after playing with this
> today) I think I have something a bit more ambitious in mind. I'm currently
> experimenting with different things which could be baked in a third-party
> lib and possibly contributed back to Python later on:
>

If you might want some of this contributed back to Python later on, you
should not use the GPL.

1) on OSX we could use f/copyfile() syscall to copy file attrs/metadata;
> that may be useful to speedup shutil.copystat() and shutil.copymode()
> 2) copytree() on OSX could take advantage of f/copyfile() +
> COPYFILE_RECURSIVE (which is probably too platform-specific for inclusion)
> 3) on Linux we could use copy_file_range() as a replacement for
> os.sendfile() in shutil.copyfile() (it's supposed to be faster)
> 4) on Linux ioctl() + FICLONE could be used to implement CoW (copy on
> write) instantaneous copies, and could be added as shutil.cow_copyfile()
> (haven't look into Windows yet)
> 5) I was thinking about backporting socket.socket.sendfile() as well,
> which uses os.sendfile() on POSIX but not TransmitFile on Windows (asyncio
> does though)
> 6)  another idea (but I'm not sure if it's possible, as I still have to
> dig into that) is a new socket recvfile() function boosted up by tee() /
> splice() on Linux, which maybe could be contributed back as
> socket.socket.recvfile()
>
> --
> Giampaolo - http://grodola.blogspot.com
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190312/1e2721bd/attachment.html>

From steve at pearwood.info  Tue Mar 12 20:53:19 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 13 Mar 2019 11:53:19 +1100
Subject: [Python-Dev] (Licensing question) backport of shutil.copyfile()
 functionality
In-Reply-To: <CAGE7PN+37ccbSRfX9ODUQsQQMvJjqTiJUWh7pnyzNuSTH-ctTQ@mail.gmail.com>
References: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
 <0c8dd528-1224-38d7-db20-c5d1bb4ee343@g.nevcal.com>
 <CAFYqXL_YXT=-hbQ3XQKuAVq8aZeyeRXRxLp2PWvew6EymtAD4A@mail.gmail.com>
 <CAGE7PN+37ccbSRfX9ODUQsQQMvJjqTiJUWh7pnyzNuSTH-ctTQ@mail.gmail.com>
Message-ID: <20190313005319.GS12502@ando.pearwood.info>

On Tue, Mar 12, 2019 at 05:32:35PM -0700, Gregory P. Smith wrote:

> If you might want some of this contributed back to Python later on, you
> should not use the GPL.

Giampaolo can always change the licence of his work later. You can't 
take away the GPL from work you've already released, but you can 
relicense the same work under any other terms.

The problem may be if others have contributed to Giampaolo's library 
under the GPL, and they refuse to re-licence their contributions. I am 
not a lawyer, but it may be that he can work around that by requiring 
that contributors assign their copyright in the contribution to him. But 
that may discourage contributors.

But since Giampaolo seems to be thinking of using the MIT licence, the 
point may be moot :-)


-- 
Steven

From g.rodola at gmail.com  Tue Mar 12 21:32:11 2019
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Wed, 13 Mar 2019 02:32:11 +0100
Subject: [Python-Dev] (Licensing question) backport of shutil.copyfile()
 functionality
In-Reply-To: <q67fvk$1154$1@blaine.gmane.org>
References: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
 <CAEfz+TyGaHv3DaAbxMfz5dyT2HsrmSp4Cx2S5Cv3h4cyMiOhKA@mail.gmail.com>
 <q67fvk$1154$1@blaine.gmane.org>
Message-ID: <CAFYqXL_stYoV20kZ9x6PgQXd6sdxnDxqvr0v0a7Mc=yws915rg@mail.gmail.com>

On Tue, Mar 12, 2019 at 6:29 AM Terry Reedy <tjreedy at udel.edu> wrote:

> On 3/11/2019 10:54 PM, Inada Naoki wrote:
>
> >> Hello,
> >> some time ago I contributed a couple of patches to speedup
> shutil.copy*() functions:
> >> https://bugs.python.org/issue33671
> >> https://bugs.python.org/issue33695
>
> You retain copyright on the code you contributed.
> https://mail.python.org/mailman/listinfo/python-dev
>

I didn't write shutil.copytree()'s code though, and I'd need to copy it.

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190313/14b18f75/attachment.html>

From remi.lapeyre at henki.fr  Wed Mar 13 06:21:31 2019
From: remi.lapeyre at henki.fr (=?UTF-8?Q?R=C3=A9mi_Lapeyre?=)
Date: Wed, 13 Mar 2019 03:21:31 -0700
Subject: [Python-Dev] int() and math.trunc don't accept objects that
 only define __index__
In-Reply-To: <CADiSq7ezAk=vbKfi37wRjTAOE+iUB2Ay7eLFSusruaO_stLQXg@mail.gmail.com>
References: <CA+jiYFCY-Fboaz=4+hvN_5LG4o=eHyOxUafGaJV6Yc51mOgtVw@mail.gmail.com>
 <q4obm4$1d65$1@blaine.gmane.org>
 <CADiSq7ezAk=vbKfi37wRjTAOE+iUB2Ay7eLFSusruaO_stLQXg@mail.gmail.com>
Message-ID: <CA+jiYFAhZH562xwqP_j3JZR9AcA1ceEZEoYSfgNvbsPkw+1FPw@mail.gmail.com>

Le 22 f?vrier 2019 ? 18:14:01, Nick Coghlan
(ncoghlan at gmail.com(mailto:ncoghlan at gmail.com)) a ?crit:

> On Fri, 22 Feb 2019 at 18:29, Serhiy Storchaka wrote:
> > Should we add default implementations of __float__ and __complex__ when
> > either __index__ or __int__ is defined? Currently:
> >
> > >>> class A:
> > ... def __int__(self): return 42
> > ...
> > >>> int(A())
> > 42
> > >>> float(A())
> > Traceback (most recent call last):
> > File "", line 1, in
> > TypeError: float() argument must be a string or a number, not 'A'
> > >>> complex(A())
> > Traceback (most recent call last):
> > File "", line 1, in
> > TypeError: complex() first argument must be a string or a number, not 'A'
> >
> > Or just document that in order to have a coherent integer type class,
> > when __index__() or __int__() are defined __float__() and __complex__()
> > should also be defined, and all should return equal values.
>
> I think when __index__ is defined, it would be reasonable to have that
> imply the same floating point conversion rules as are applied for
> builtin ints, since the conversion is supposed to be lossless in that
> case (and if it isn't lossless, that's what `__int__` is for).
> However, I don't think the decision is quite as clearcut as it is for
> `__index__` implying `__int__`.

When __index__ is defined it means that there is a lossless conversion
to int possible. In this case, this means a lossless conversion to
float and complex is also possible (with the exception of overflows
but anyone doing float(var) should expect them). In this case, I think
defining __index__ could set a default for __int__, __float__ and
__complex__.

> Lossy conversions to int shouldn't imply anything about conversions to
> real numbers or floating point values.

I think the right behavior is the other way around, it should go the
other way around from a superset to a subset, when __int__ is defined
__float__ should not have a default set because there might be a
better default than float(int(self)), but when __float__ is defined,
it should be safe to make __int__ default to int(float(self)) if not
defined.

This would simplify this behavior:

Python 3.7.2 (default, Jan 13 2019, 12:50:01)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class Test:
... ? ? def __float__(self):
... ? ? ? ? ? ? return 3.14
...
>>> float(Test())
3.14
>>> int(float(Test()))
3
>>> int(Test())
Traceback (most recent call last):
? File "<stdin>", line 1, in <module>
TypeError: int() argument must be a string, a bytes-like object or a
number, not ?Test'


Defining __complex__ shouldn?t do anything since they have one more
dimension there is no clear way to convert to ints and floats which is
already the behavior of the the builtin complex type:

>>> int(complex())
Traceback (most recent call last):
? File "<stdin>", line 1, in <module>
TypeError: can't convert complex to int
>>> float(complex())
Traceback (most recent call last):
? File "<stdin>", line 1, in <module>
TypeError: can't convert complex to float


Here?s an implementation of what I?m thinking of:


>>> def finish_class(cls):
... ? ? def set(cls, name, target):
... ? ? ? ? if not hasattr(cls, name):
... ? ? ? ? ? ? setattr(cls, name, target)
...
... ? ? if hasattr(cls, '__index__'):
... ? ? ? ? set(cls, '__int__', cls.__index__)
... ? ? ? ? set(cls, '__float__', lambda self: float(self.__index__()))
... ? ? ? ? set(cls, '__complex__', lambda self: complex(self.__index__(), 0))
... ? ? if hasattr(cls, '__float__'):
... ? ? ? ? set(cls, '__int__', lambda self: int(float(self)))
...
>>>
>>> class Test:
... ? ? def __index__(self):
... ? ? ? ? return 4
...
>>> finish_class(Test)
>>>
>>> int(Test())
4
>>> float(Test())
4.0
>>> complex(Test())
(4+0j)
>>>
>>> class Test2:
... ? ? def __float__(self):
... ? ? ? ? return 3.14
...
>>> finish_class(Test2)
>>>
>>> int(Test2())
3
>>> float(Test2())
3.14

From vstinner at redhat.com  Thu Mar 14 13:01:32 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 14 Mar 2019 18:01:32 +0100
Subject: [Python-Dev] Initial updates to PEP 1 for Steering Council
 based governance
In-Reply-To: <CADiSq7e4TFv2XquDGasDZ72n2NFGs4zjfRnaFEE8qO0mrVhKyw@mail.gmail.com>
References: <CADiSq7e4TFv2XquDGasDZ72n2NFGs4zjfRnaFEE8qO0mrVhKyw@mail.gmail.com>
Message-ID: <CA+3bQGFH0N=4DhpX4vL5jU5K3gP3CfkehGwjLAPhxicKZWwJDA@mail.gmail.com>

Hi,

Le lun. 11 mars 2019 ? 13:26, Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> This is the smallest change to PEP 1 that we consider potentially viable: handling all PEPs through the BDFL-Delegate model, with the Steering Council's primary involvement being to appoint the delegates (or accept their volunteering), once a PEP has reached the point of being actively reviewed.

Does it mean that very controversial PEPs like PEP 572 would have a
BDFL-delegate as well? The delegate will be the only one taking the
final decision?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From guido at python.org  Thu Mar 14 13:30:54 2019
From: guido at python.org (Guido van Rossum)
Date: Thu, 14 Mar 2019 10:30:54 -0700
Subject: [Python-Dev] Typing Summit at PyCon?
Message-ID: <CAP7+vJJCarymRq0qhwC9rmTm4F1ddiBDu5C6yYT8JwDyf3UvbQ@mail.gmail.com>

If you're going to PyCon and interested in static typing for Python (e.g.
mypy, pytype, Pyre, PyCharm), I have something for you. I am hoping to
organize a Typing Summit, on Thursday afternoon, May 2nd. If you're
interested, please reply to me so I can gauge interest.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190314/1e77b5b4/attachment.html>

From brett at python.org  Thu Mar 14 13:33:01 2019
From: brett at python.org (Brett Cannon)
Date: Thu, 14 Mar 2019 10:33:01 -0700
Subject: [Python-Dev] Initial updates to PEP 1 for Steering Council
 based governance
In-Reply-To: <CA+3bQGFH0N=4DhpX4vL5jU5K3gP3CfkehGwjLAPhxicKZWwJDA@mail.gmail.com>
References: <CADiSq7e4TFv2XquDGasDZ72n2NFGs4zjfRnaFEE8qO0mrVhKyw@mail.gmail.com>
 <CA+3bQGFH0N=4DhpX4vL5jU5K3gP3CfkehGwjLAPhxicKZWwJDA@mail.gmail.com>
Message-ID: <CAP1=2W6+A7fpU6dZuG8qys1eyokttga=tvFcdAiDRXHtZzCBxw@mail.gmail.com>

On Thu, Mar 14, 2019 at 10:02 AM Victor Stinner <vstinner at redhat.com> wrote:

> Hi,
>
> Le lun. 11 mars 2019 ? 13:26, Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> > This is the smallest change to PEP 1 that we consider potentially
> viable: handling all PEPs through the BDFL-Delegate model, with the
> Steering Council's primary involvement being to appoint the delegates (or
> accept their volunteering), once a PEP has reached the point of being
> actively reviewed.
>
> Does it mean that very controversial PEPs like PEP 572 would have a
> BDFL-delegate as well? The delegate will be the only one taking the
> final decision?
>

Probably not. In cases where no clear BDFL Delegate can be identified the
SC will probably handle the decision.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190314/2ab3e6be/attachment.html>

From jigar.1994 at gmail.com  Thu Mar 14 14:39:04 2019
From: jigar.1994 at gmail.com (Jigar Bhalodia)
Date: Thu, 14 Mar 2019 14:39:04 -0400
Subject: [Python-Dev] How to create a child process using multiprocessing in
 Python2.7.10 without the child sharing resources with parent?
Message-ID: <CAAiHz+nRy=y_hbH83o040AbF3JxU7oVqAQ_7eu+TYv6wfV1V-Q@mail.gmail.com>

Hello,

We are trying to move our python 2.7.10 codebase from Windows to Linux. We
recently discovered that multiprocessing library in Python 2.7 behaves
differently on Windows vs Linux. We have found many articles like this one
<https://rhodesmill.org/brandon/2010/python-multiprocessing-linux-windows/>
describing
the problem however, we are unable to find a solution online for Python
2.7. This is a fix
<https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method>
for
this issue in Python 3.4 however, we are unable to upgrade to Python 3.4.
Is there any way to use multiprocessing in Python 2.7 on Linux without the
child and parent sharing memory? We can also use guidance on modifying
forking.py
<https://github.com/python/cpython/blob/2.7/Lib/multiprocessing/forking.py>
code
in python 2.7 to ensure child and parent process aren't sharing memory and
doing Copy-on-Write. Thanks!

https://stackoverflow.com/questions/55168149/how-to-create-a-child-process-using-multiprocessing-in-python2-7-10-without-the

Thanks,
Jigar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190314/99d09ac9/attachment.html>

From steve at pearwood.info  Thu Mar 14 22:45:42 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Mar 2019 13:45:42 +1100
Subject: [Python-Dev] int() and math.trunc don't accept objects that
 only define __index__
In-Reply-To: <CA+jiYFAhZH562xwqP_j3JZR9AcA1ceEZEoYSfgNvbsPkw+1FPw@mail.gmail.com>
References: <CA+jiYFCY-Fboaz=4+hvN_5LG4o=eHyOxUafGaJV6Yc51mOgtVw@mail.gmail.com>
 <q4obm4$1d65$1@blaine.gmane.org>
 <CADiSq7ezAk=vbKfi37wRjTAOE+iUB2Ay7eLFSusruaO_stLQXg@mail.gmail.com>
 <CA+jiYFAhZH562xwqP_j3JZR9AcA1ceEZEoYSfgNvbsPkw+1FPw@mail.gmail.com>
Message-ID: <20190315024541.GE12502@ando.pearwood.info>

On Wed, Mar 13, 2019 at 03:21:31AM -0700, R?mi Lapeyre wrote:

> When __index__ is defined it means that there is a lossless conversion
> to int possible. In this case, this means a lossless conversion to
> float and complex is also possible 

That's not correct:

py> n = 2**64 + 1
py> n == int(float(n))
False

Python floats (C doubles) can lose digits when converting from ints 
over 2**53 or so.


> (with the exception of overflows
> but anyone doing float(var) should expect them).

I don't. I expect float(var) to overflow to infinity, if it is going to 
overflow, and always forget that it can raise.


py> float("9e9999")
inf

py> float(str(9*10**9999))
inf

But:

py> float(9*10**9999)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: int too large to convert to float

This never fails to surprise me.




-- 
Steven

From solipsis at pitrou.net  Fri Mar 15 06:50:43 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 15 Mar 2019 11:50:43 +0100
Subject: [Python-Dev] Register-based VM [Was: Possible performance
 regression]
References: <20190225115452.30ed11bb@fsol>
 <9D28FA05-1FDA-43E0-9CD1-DCDDBCE05A03@gmail.com>
 <CALFfu7C=xpvRwbhnnB4fDEFxY7eezV0QHrxUEoX9Tkq3SgGexg@mail.gmail.com>
 <CALFfu7CT9KpV-6bDgi9pPeX0vESu8fOcRT=fkB83jQCYLXKD3g@mail.gmail.com>
 <20190226193512.m5wzrecaxt3yfm7l@python.ca>
 <CA+3bQGGXjMAJ6FQb=65OAjRW+p783ePzGFTcaHWLONZNFs7KJA@mail.gmail.com>
 <20190226205841.wo6a6bc65igmxpk3@python.ca>
 <CA+3bQGGfZ6nRkKS==ArkK8FcZe_8xdrV_bmcaoHOWPpWKVzq5w@mail.gmail.com>
 <5C75BE81.4060209@canterbury.ac.nz>
 <CA+3bQGEOc53H3VwnAj6Dy1o+txoc244AFnZJKTcWvQDJCg4G3w@mail.gmail.com>
 <20190311215629.y2j6szbqzzi7jcds@python.ca>
 <CAEbHw4ZsBtA5WukU1-VmcdM+-W4BbCrUsz6Op0ON39LoXXWDXg@mail.gmail.com>
Message-ID: <20190315115043.56255e6d@fsol>

On Mon, 11 Mar 2019 18:03:26 -0400
David Mertz <mertz at gnosis.cx> wrote:
> Parrot got rather further along than rattlesnake as a register based VM. I
> don't think it every really beat CPython in speed though.
> 
> http://parrot.org/

But Parrot also had a "generic" design that was supposed to cater for
all dynamic programming languages.  AFAIU, it was heavily
over-engineered and under-staffed, and the design documents were
difficult to understand.

Regards

Antoine.



From remi.lapeyre at henki.fr  Fri Mar 15 06:59:18 2019
From: remi.lapeyre at henki.fr (=?UTF-8?Q?R=C3=A9mi_Lapeyre?=)
Date: Fri, 15 Mar 2019 06:59:18 -0400
Subject: [Python-Dev] int() and math.trunc don't accept objects that
 only define __index__
In-Reply-To: <20190315024541.GE12502@ando.pearwood.info>
References: <CA+jiYFCY-Fboaz=4+hvN_5LG4o=eHyOxUafGaJV6Yc51mOgtVw@mail.gmail.com>
 <q4obm4$1d65$1@blaine.gmane.org>
 <CADiSq7ezAk=vbKfi37wRjTAOE+iUB2Ay7eLFSusruaO_stLQXg@mail.gmail.com>
 <CA+jiYFAhZH562xwqP_j3JZR9AcA1ceEZEoYSfgNvbsPkw+1FPw@mail.gmail.com>
 <20190315024541.GE12502@ando.pearwood.info>
 <20190315024541.GE12502@ando.pearwood.info>
Message-ID: <CA+jiYFCY02FjjZTVxQdqKtReOZy1sOXndmmCUd6m0bPSzZWn6g@mail.gmail.com>

Le 15 mars 2019 ? 03:49:19, Steven D'Aprano
(steve at pearwood.info(mailto:steve at pearwood.info)) a ?crit:

> On Wed, Mar 13, 2019 at 03:21:31AM -0700, R?mi Lapeyre wrote:
>
> > When __index__ is defined it means that there is a lossless conversion
> > to int possible. In this case, this means a lossless conversion to
> > float and complex is also possible
>
> That's not correct:
>
> py> n = 2**64 + 1
> py> n == int(float(n))
> False

Thanks, I should have thought of that. Do you think __index__ should
only define
__int__ then?


> Python floats (C doubles) can lose digits when converting from ints
> over 2**53 or so.
>
>
> > (with the exception of overflows
> > but anyone doing float(var) should expect them).
>
> I don't. I expect float(var) to overflow to infinity, if it is going to
> overflow, and always forget that it can raise.
>
>
> py> float("9e9999")
> inf
>
> py> float(str(9*10**9999))
> inf
>
> But:
>
> py> float(9*10**9999)
> Traceback (most recent call last):
> File "", line 1, in
> OverflowError: int too large to convert to float
>
> This never fails to surprise me.
>
>
>
>
> --
> Steven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/remi.lapeyre%40henki.fr

From solipsis at pitrou.net  Fri Mar 15 07:06:08 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 15 Mar 2019 12:06:08 +0100
Subject: [Python-Dev] (Licensing question) PSF license
References: <CAFYqXL9syH6xiiQh6SyK192UnSScV_3s_eCAkptcRCcqL-LNZQ@mail.gmail.com>
 <CAEfz+TyGaHv3DaAbxMfz5dyT2HsrmSp4Cx2S5Cv3h4cyMiOhKA@mail.gmail.com>
 <q67fvk$1154$1@blaine.gmane.org>
Message-ID: <20190315120608.18c3225e@fsol>

On Tue, 12 Mar 2019 01:27:14 -0400
Terry Reedy <tjreedy at udel.edu> wrote:
> > First of all, I'm sorry if I'm wrong.  I'm not lawyer.
> > 
> > You can use both of GPL and MIT.  Users can use your package under it.
> > 
> > On the other hand, when you publish your package, *you* should follow
> > PSF license.
> > Read this.  https://docs.python.org/3/license.html
> > 
> > """
> > 3. In the event Licensee prepares a derivative work that is based on or
> >     incorporates Python 3.7.2 or any part thereof, and wants to make the
> >     derivative work available to others as provided herein, then Licensee hereby
> >     agrees to include in any such work a brief summary of the changes
> > made to Python
> >     3.7.2.
> > """
> > 
> > As you can see, PSF license doesn't force you to use PSF license. (not
> > "copyleft")  
> 
> In fact, the PSF lawyer says one should not use the 'PSF license' as it 
> is specilized for the PSF and Python.

Interesting.  I did use the PSF license for the pickle5 backport and I
suspect I'm not the only one to use it (though I don't know how to do
a per-license search on PyPI).
https://pypi.org/project/pickle5/

Regards

Antoine.



From status at bugs.python.org  Fri Mar 15 14:08:26 2019
From: status at bugs.python.org (Python tracker)
Date: Fri, 15 Mar 2019 18:08:26 +0000 (UTC)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190315180826.131E052B22B@bugs.ams1.psf.io>


ACTIVITY SUMMARY (2019-03-08 - 2019-03-15)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    7048 (+17)
  closed 41009 (+46)
  total  48057 (+63)

Open issues with patches: 2798 


Issues opened (48)
==================

#4459: bdist_rpm should enable --fix-python by default
https://bugs.python.org/issue4459  reopened by vstinner

#10948: Trouble with dir_util created dir cache
https://bugs.python.org/issue10948  reopened by eric.araujo

#30040: new empty dict can be more small
https://bugs.python.org/issue30040  reopened by rhettinger

#33376: [pysqlite] Duplicate rows can be returned after rolling back a
https://bugs.python.org/issue33376  reopened by benjamin.peterson

#36245: PCBuild/build.bat errors, probably from space characters in pa
https://bugs.python.org/issue36245  opened by Jess

#36246: csv.writer lineterminator affects csv escaping
https://bugs.python.org/issue36246  opened by flow2k

#36247: zipfile - extract truncates (existing) file when bad password 
https://bugs.python.org/issue36247  opened by CristiFati

#36250: pdb: interaction might cause "ValueError: signal only works in
https://bugs.python.org/issue36250  opened by blueyed

#36253: Use after free in ctypes test suite
https://bugs.python.org/issue36253  opened by btharper

#36254: Fix invalid uses of %d in format strings in C
https://bugs.python.org/issue36254  opened by serhiy.storchaka

#36255: Provide a simple way to delete and edit python's welcome messa
https://bugs.python.org/issue36255  opened by benp

#36256: parser module fails on legal input
https://bugs.python.org/issue36256  opened by A. Skrobov

#36257: configure with --with-icc --with-cxx-main=icpc
https://bugs.python.org/issue36257  opened by aminiussi

#36258: Incorrect docstring of the ssl module
https://bugs.python.org/issue36258  opened by Ofekmeister

#36259: exception text is being sourced from the wrong file
https://bugs.python.org/issue36259  opened by rhubarbdog x

#36260: Cpython/Lib vulnerability found and request a patch submission
https://bugs.python.org/issue36260  opened by krnick

#36261: email examples should not gratuitously mess with preamble
https://bugs.python.org/issue36261  opened by era

#36263: test_hashlib.test_scrypt() fails on Fedora 29
https://bugs.python.org/issue36263  opened by vstinner

#36265: Remove ABCs from collections
https://bugs.python.org/issue36265  opened by jwilk

#36266: Which module could not be found?
https://bugs.python.org/issue36266  opened by Phillip.M.Feldman at gmail.com

#36267: User input to argparse raises Index_Error:  "-a=" on a 'store_
https://bugs.python.org/issue36267  opened by paul.j3

#36268: Change default tar format to modern POSIX 2001 (pax) for bette
https://bugs.python.org/issue36268  opened by CAM-Gerlach

#36270: DOC: Add link to sys.exc_info for "Reference Manual"
https://bugs.python.org/issue36270  opened by cheryl.sabella

#36271: '_io.TextIOWrapper' object has no attribute 'mode'
https://bugs.python.org/issue36271  opened by nemeskeyd

#36272: Recursive logging crashes Interpreter in Python 3
https://bugs.python.org/issue36272  opened by Saim Raza

#36273: test_thread leaks a core dump on PPC64 AIX 3.x
https://bugs.python.org/issue36273  opened by vstinner

#36274: http.client cannot send non-ASCII request lines
https://bugs.python.org/issue36274  opened by tburke

#36275: DOC: venv.create doesn't include prompt parameter
https://bugs.python.org/issue36275  opened by cheryl.sabella

#36276: Python urllib CRLF injection vulnerability
https://bugs.python.org/issue36276  opened by ragdoll.guo

#36277: pdb's recursive debug command is not listed in the docs
https://bugs.python.org/issue36277  opened by Antony.Lee

#36279: os.wait3() leaks some uninitialized stack when no processes ex
https://bugs.python.org/issue36279  opened by dw

#36281: OSError: handle is closed for ProcessPoolExecutor and run_in_e
https://bugs.python.org/issue36281  opened by basnijholt

#36283: eval is needlessly limited
https://bugs.python.org/issue36283  opened by bup

#36285: Integer overflow in array.array.remove()
https://bugs.python.org/issue36285  opened by sth

#36286: Random failure in test_idle
https://bugs.python.org/issue36286  opened by serhiy.storchaka

#36287: Make ast.dump() not output optional default fields
https://bugs.python.org/issue36287  opened by serhiy.storchaka

#36290: _ast.ast_type_init does not handle args and kwargs correctly.
https://bugs.python.org/issue36290  opened by remi.lapeyre

#36292: Coverity scan: Resource leaks in longobject.c
https://bugs.python.org/issue36292  opened by cstratak

#36293: Nonblocking read sys.stdin raises error
https://bugs.python.org/issue36293  opened by cykerway

#36295: Need to yield (sleep(0)) twice in asyncio
https://bugs.python.org/issue36295  opened by Assaf Dayan

#36297: Remove unicode_internal codec
https://bugs.python.org/issue36297  opened by inada.naoki

#36298: Lib/pyclbr.py crashes when the package spec cannot be determin
https://bugs.python.org/issue36298  opened by mental

#36299: Deprecate 'u' type in array module
https://bugs.python.org/issue36299  opened by inada.naoki

#36300: eval of generator expressions cannot access local variables
https://bugs.python.org/issue36300  opened by Martin Hosken

#36301: Add _Py_PreInitialize() function
https://bugs.python.org/issue36301  opened by vstinner

#36302: distutils creates unreproducible .so files
https://bugs.python.org/issue36302  opened by bmwiedemann

#36304: When using bz2 and lzma in mode 'wt', the BOM is not written
https://bugs.python.org/issue36304  opened by janluke

#36305: Inconsistent behavior of pathlib.WindowsPath with drive paths
https://bugs.python.org/issue36305  opened by kmaork



Most recent 15 issues with no replies (15)
==========================================

#36302: distutils creates unreproducible .so files
https://bugs.python.org/issue36302

#36299: Deprecate 'u' type in array module
https://bugs.python.org/issue36299

#36298: Lib/pyclbr.py crashes when the package spec cannot be determin
https://bugs.python.org/issue36298

#36295: Need to yield (sleep(0)) twice in asyncio
https://bugs.python.org/issue36295

#36290: _ast.ast_type_init does not handle args and kwargs correctly.
https://bugs.python.org/issue36290

#36285: Integer overflow in array.array.remove()
https://bugs.python.org/issue36285

#36281: OSError: handle is closed for ProcessPoolExecutor and run_in_e
https://bugs.python.org/issue36281

#36279: os.wait3() leaks some uninitialized stack when no processes ex
https://bugs.python.org/issue36279

#36275: DOC: venv.create doesn't include prompt parameter
https://bugs.python.org/issue36275

#36274: http.client cannot send non-ASCII request lines
https://bugs.python.org/issue36274

#36270: DOC: Add link to sys.exc_info for "Reference Manual"
https://bugs.python.org/issue36270

#36267: User input to argparse raises Index_Error:  "-a=" on a 'store_
https://bugs.python.org/issue36267

#36258: Incorrect docstring of the ssl module
https://bugs.python.org/issue36258

#36257: configure with --with-icc --with-cxx-main=icpc
https://bugs.python.org/issue36257

#36253: Use after free in ctypes test suite
https://bugs.python.org/issue36253



Most recent 15 issues waiting for review (15)
=============================================

#36302: distutils creates unreproducible .so files
https://bugs.python.org/issue36302

#36301: Add _Py_PreInitialize() function
https://bugs.python.org/issue36301

#36297: Remove unicode_internal codec
https://bugs.python.org/issue36297

#36292: Coverity scan: Resource leaks in longobject.c
https://bugs.python.org/issue36292

#36287: Make ast.dump() not output optional default fields
https://bugs.python.org/issue36287

#36285: Integer overflow in array.array.remove()
https://bugs.python.org/issue36285

#36279: os.wait3() leaks some uninitialized stack when no processes ex
https://bugs.python.org/issue36279

#36274: http.client cannot send non-ASCII request lines
https://bugs.python.org/issue36274

#36272: Recursive logging crashes Interpreter in Python 3
https://bugs.python.org/issue36272

#36263: test_hashlib.test_scrypt() fails on Fedora 29
https://bugs.python.org/issue36263

#36254: Fix invalid uses of %d in format strings in C
https://bugs.python.org/issue36254

#36253: Use after free in ctypes test suite
https://bugs.python.org/issue36253

#36250: pdb: interaction might cause "ValueError: signal only works in
https://bugs.python.org/issue36250

#36247: zipfile - extract truncates (existing) file when bad password 
https://bugs.python.org/issue36247

#36245: PCBuild/build.bat errors, probably from space characters in pa
https://bugs.python.org/issue36245



Top 10 most discussed issues (10)
=================================

#36085: Enable better DLL resolution
https://bugs.python.org/issue36085  18 msgs

#30040: new empty dict can be more small
https://bugs.python.org/issue30040  15 msgs

#36010: Please provide a .zip Windows release of Python that is not cr
https://bugs.python.org/issue36010  10 msgs

#36272: Recursive logging crashes Interpreter in Python 3
https://bugs.python.org/issue36272   8 msgs

#35121: Cookie domain check returns incorrect results
https://bugs.python.org/issue35121   7 msgs

#35647: Cookie path check returns incorrect results
https://bugs.python.org/issue35647   7 msgs

#36276: Python urllib CRLF injection vulnerability
https://bugs.python.org/issue36276   7 msgs

#36256: parser module fails on legal input
https://bugs.python.org/issue36256   6 msgs

#36263: test_hashlib.test_scrypt() fails on Fedora 29
https://bugs.python.org/issue36263   6 msgs

#21314: Document '/' in signatures
https://bugs.python.org/issue21314   5 msgs



Issues closed (44)
==================

#12974: array module: deprecate '__int__' conversion support for array
https://bugs.python.org/issue12974  closed by serhiy.storchaka

#26018: documentation of ZipFile file name encoding
https://bugs.python.org/issue26018  closed by serhiy.storchaka

#27916: Use time.monotonic instead of time.time where appropriate
https://bugs.python.org/issue27916  closed by cheryl.sabella

#28956: return list of modes for a multimodal distribution instead of 
https://bugs.python.org/issue28956  closed by steven.daprano

#31973: Incomplete DeprecationWarning for async/await keywords
https://bugs.python.org/issue31973  closed by barry

#32846: Deletion of large sets of strings is extra slow
https://bugs.python.org/issue32846  closed by inada.naoki

#33002: Making a class formattable as hex/oct integer with printf-styl
https://bugs.python.org/issue33002  closed by serhiy.storchaka

#35132: python-gdb error: Python Exception <class 'RuntimeError'> Type
https://bugs.python.org/issue35132  closed by vstinner

#35661: Store the venv prompt in pyvenv.cfg
https://bugs.python.org/issue35661  closed by cheryl.sabella

#35807: Update bundled pip to 19.0
https://bugs.python.org/issue35807  closed by ncoghlan

#35830: building multiple (binary) packages from a single project
https://bugs.python.org/issue35830  closed by stefan

#35832: Installation error
https://bugs.python.org/issue35832  closed by steve.dower

#35843: importlib.util docs for namespace packages innaccurate
https://bugs.python.org/issue35843  closed by brett.cannon

#35854: EnvBuilder and venv symlinks do not work on Windows on 3.7.2
https://bugs.python.org/issue35854  closed by steve.dower

#35892: Fix awkwardness of statistics.mode() for multimodal datasets
https://bugs.python.org/issue35892  closed by rhettinger

#35944: Python 3.7 install error
https://bugs.python.org/issue35944  closed by steve.dower

#36029: Use title-case HTTP header fields
https://bugs.python.org/issue36029  closed by mdk

#36042: Setting __init_subclass__ and __class_getitem__ methods are in
https://bugs.python.org/issue36042  closed by inada.naoki

#36174: Remove licenseUrl field from nuget packages
https://bugs.python.org/issue36174  closed by steve.dower

#36176: Fix IDLE Autocomplete / Calltip Window Colors
https://bugs.python.org/issue36176  closed by terry.reedy

#36212: [2.7] Coverity scan: Modules/_hotshot.c , Variable "s1" going 
https://bugs.python.org/issue36212  closed by vstinner

#36216: CVE-2019-9636: urlsplit does not handle NFKC normalization
https://bugs.python.org/issue36216  closed by steve.dower

#36229: Avoid unnecessary copies for list, set, and bytearray ops.
https://bugs.python.org/issue36229  closed by rhettinger

#36234: test_os: add tests for invalid uid type
https://bugs.python.org/issue36234  closed by vstinner

#36241: MD5 checksum is not valid for v2.7.16 "Windows x86-64 MSI inst
https://bugs.python.org/issue36241  closed by SilentGhost

#36244: Lock release fails under windows
https://bugs.python.org/issue36244  closed by Konrad Ciecierski

#36248: document about `or`, `and` operator.
https://bugs.python.org/issue36248  closed by SilentGhost

#36249: f-string should be the default placeholder
https://bugs.python.org/issue36249  closed by serhiy.storchaka

#36251: Invalid format specifiers in MatchObject and StdPrinter repr
https://bugs.python.org/issue36251  closed by serhiy.storchaka

#36252: update to Unicode 12
https://bugs.python.org/issue36252  closed by benjamin.peterson

#36262: Coverity scan: Python/dtoa.c resource leak
https://bugs.python.org/issue36262  closed by vstinner

#36264: os.path.expanduser should not use HOME on windows
https://bugs.python.org/issue36264  closed by steve.dower

#36269: post install in setup.py does not work when executed through p
https://bugs.python.org/issue36269  closed by sabakauser

#36278: Truncate method
https://bugs.python.org/issue36278  closed by josh.r

#36280: Add kind field to ast.Constant, to distinguish u"..." from "..
https://bugs.python.org/issue36280  closed by gvanrossum

#36282: Not accurate error message for exact number of positional argu
https://bugs.python.org/issue36282  closed by serhiy.storchaka

#36284: importlib.import_module() not thread safe if Exception is rais
https://bugs.python.org/issue36284  closed by serhiy.storchaka

#36288: Incorrect answer when using round()
https://bugs.python.org/issue36288  closed by zach.ware

#36289: [2.7] Coverity scan: Modules/_io/bufferedio.c leaked_storage: 
https://bugs.python.org/issue36289  closed by vstinner

#36291: [2.7] Coverity Scan: Modules/_json.c: leaked_storage: Variable
https://bugs.python.org/issue36291  closed by vstinner

#36294: `io.BufferedIOBase` returns `None`
https://bugs.python.org/issue36294  closed by martin.panter

#36296: distutils.version.StrictVersion objects cannot be compared wit
https://bugs.python.org/issue36296  closed by eric.araujo

#36303: Trying to change values (fe. "To", "From") of email.mime.text.
https://bugs.python.org/issue36303  closed by Fotoblysk

#36306: inspect.signature(math.log) raises ValueError
https://bugs.python.org/issue36306  closed by remi.lapeyre

From brett at python.org  Fri Mar 15 17:24:13 2019
From: brett at python.org (Brett Cannon)
Date: Fri, 15 Mar 2019 14:24:13 -0700
Subject: [Python-Dev] Rejecting PEP 473: Adding structured data to built-in
 exceptions
Message-ID: <CAP1=2W7PT1xqGdemEineaCKU_J5Tv22+qbRYbS_9x6Vo_5KfVg@mail.gmail.com>

The steering council felt the PEP was too broad and not focused enough.
Discussions about adding more attributes to built-in exceptions can
continue on the issue tracker on a per-exception basis (and obviously here
for any broader points, e.g. performance implications as I know that has
come up before when the idea of storing relevant objects on exceptions).

Thanks to Sebastian Kreft for taking the time to write the PEP in the first
place.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190315/e2f3550e/attachment.html>

From brett at python.org  Fri Mar 15 17:47:26 2019
From: brett at python.org (Brett Cannon)
Date: Fri, 15 Mar 2019 14:47:26 -0700
Subject: [Python-Dev] Reject PEP 472: Support for indexing with keyword
 arguments
Message-ID: <CAP1=2W7RHwOhSqMmYyRZjHJJGL55AX1dBEhEFDvORO_T5hgo_w@mail.gmail.com>

The idea never seemed to gain any traction over its near 5 years in
existence as a PEP.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190315/4b40946b/attachment.html>

From brett at python.org  Fri Mar 15 18:32:26 2019
From: brett at python.org (Brett Cannon)
Date: Fri, 15 Mar 2019 15:32:26 -0700
Subject: [Python-Dev] Deferring PEP 536 -- Final Grammar for Literal String
 Interpolation
Message-ID: <CAP1=2W5ELaSEuTaAc1_j+NYAgocRwoeX=oE0LxNgDqSNDL+b+w@mail.gmail.com>

The steering council decided to defer PEP 536 until an implementation is
available.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190315/73148f1b/attachment.html>

From brett at python.org  Fri Mar 15 18:42:49 2019
From: brett at python.org (Brett Cannon)
Date: Fri, 15 Mar 2019 15:42:49 -0700
Subject: [Python-Dev] Rejecting PEP 542 -- Dot Notation Assignment In
 Function Header
Message-ID: <CAP1=2W7pr-TAQcXJMCMbJnnWFLP7GHc0Ao6wZ7buHVYKyx10xA@mail.gmail.com>

The steering council has decided to reject PEP 542 as the idea never seemed
to gain traction.

Thanks to Markus Meskanen for taking the time to write the PEP.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190315/8b508c82/attachment.html>

From ncoghlan at gmail.com  Sat Mar 16 00:58:55 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Mar 2019 14:58:55 +1000
Subject: [Python-Dev] Initial updates to PEP 1 for Steering Council
 based governance
In-Reply-To: <CA+3bQGFH0N=4DhpX4vL5jU5K3gP3CfkehGwjLAPhxicKZWwJDA@mail.gmail.com>
References: <CADiSq7e4TFv2XquDGasDZ72n2NFGs4zjfRnaFEE8qO0mrVhKyw@mail.gmail.com>
 <CA+3bQGFH0N=4DhpX4vL5jU5K3gP3CfkehGwjLAPhxicKZWwJDA@mail.gmail.com>
Message-ID: <CADiSq7ekBzTezDbwVj=iFAG8STAP3xVGewJ98CqGr91aa96Z8g@mail.gmail.com>

On Fri, 15 Mar 2019 at 03:01, Victor Stinner <vstinner at redhat.com> wrote:
>
> Hi,
>
> Le lun. 11 mars 2019 ? 13:26, Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> > This is the smallest change to PEP 1 that we consider potentially viable: handling all PEPs through the BDFL-Delegate model, with the Steering Council's primary involvement being to appoint the delegates (or accept their volunteering), once a PEP has reached the point of being actively reviewed.
>
> Does it mean that very controversial PEPs like PEP 572 would have a
> BDFL-delegate as well? The delegate will be the only one taking the
> final decision?

Steering Council members can be BDFL-Delegates, so for particularly
controversial PEPs, it's most likely that:

1. the BD will be a Steering Council member
2. they'll be serving as a spokesperson for the SC in relation to that
PEP and consulting with the rest of the SC before making any
pronouncements, rather than operating independently

That seems likely to provide clearer communication both inside and
outside the SC than if we leave the responsibility for communicating
the status of affected PEPs to the SC as a whole.

However, we don't think it makes sense to handle *every* PEP that way,
since there's non-trivial overhead in making and communicating
decisions as a 5-person team, rather than as a more autonomous
individual decision maker.

Even when the BD isn't a Steering Council member themselves though,
the intent is for them to rely on the SC members for advice and
support if they feel they need it, as described in this paragraph from
the revised PEP 1:

============
The final authority for PEP approval is the Steering Council. However,
whenever a new PEP is put forward, any core developer that believes
they are suitably experienced to make the final decision on that PEP
may offer to serve as the BDFL-Delegate for that PEP, and they will
then have the authority to approve (or reject) that PEP.
Individuals taking on this responsibility are free to seek additional
guidance from the Steering Council at any time, and are also expected
to take the advice and perspectives of other core developers into
account.
============

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 16 01:25:13 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Mar 2019 15:25:13 +1000
Subject: [Python-Dev] PEP 581: Using GitHub Issues for CPython
In-Reply-To: <CAAjsFLRbO9myg_2VhOZJP+_EvTg3cXE3nKf+t6KOxa4sV6jOpg@mail.gmail.com>
References: <CAGbohnZMUfGwzfhzqp1QEyhuuVEFeAWe8XVs+=p_MqZ2zkq+ew@mail.gmail.com>
 <CAAjsFLRbO9myg_2VhOZJP+_EvTg3cXE3nKf+t6KOxa4sV6jOpg@mail.gmail.com>
Message-ID: <CADiSq7dk5BsbXOMBCEvABHn2H9gEBo1oz56URZv=yEo_xAPP5Q@mail.gmail.com>

On Fri, 8 Mar 2019 at 16:52, Karthikeyan <tir.karthi at gmail.com> wrote:
> Personally, I think more people will love it once they get to use it so if something like 100 issues can be migrated to a sample repo with labels, content etc.

We're already using GitHub issues for pretty much everything in Python
core development that *isn't* CPython itself, and even for CPython,
folks experience the core issue management interface in the overview
page for pull requests.

One of the requests we made at the Language Summit discussion last
year was for Mariatta to enhance PEP 581 with additional discussion of
what would need to change to bring the Roundup instance up to the
point of being competitive with the GitHub issues user experience,
which she added:

* https://www.python.org/dev/peps/pep-0581/#issues-with-roundup-bpo
* https://www.python.org/dev/peps/pep-0581/#why-not-focus-on-improving-roundup-bpo

There's been plenty of time since then for folks to put forward a
competing proposal to modernise the Roundup UI directly instead of
migrating to a different issue tracking tool, but so far no such
proposal has emerged.

One of the other blockers was the fact that the Contributor Licensing
Agreement process was tightly coupled to some custom fields in b.p.o
[1], and that is now very close to being resolved thanks to Mariatta's
efforts (and provides a nice workflow improvement in its own right).

Cheers,
Nick.

[1] https://www.python.org/dev/peps/pep-0581/#update-the-cla-host

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vstinner at redhat.com  Sat Mar 16 06:06:23 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Sat, 16 Mar 2019 11:06:23 +0100
Subject: [Python-Dev] Rejecting PEP 473: Adding structured data to built-in
 exceptions
In-Reply-To: <CAP1=2W7PT1xqGdemEineaCKU_J5Tv22+qbRYbS_9x6Vo_5KfVg@mail.gmail.com>
References: <CAP1=2W7PT1xqGdemEineaCKU_J5Tv22+qbRYbS_9x6Vo_5KfVg@mail.gmail.com>
Message-ID: <CA+3bQGHCs7Uor4C=_6qjwmGODhvuBRZ1hLAv==oFCXrdzzixNg@mail.gmail.com>

Hi,

While I would like to get such new attributes, I see drawbacks as blocker
issues and so I am fine with this PEP rejection. Performance is too
critical for most common exceptions.

For me one blocker issue is the high risk of creating reference cycles. And
the weak reference API isn't the most practical :-(

Moreover I expect slowdown, whereas exceptions are already expensive :-(
Recently, internal code to get an attribute has been reworked to avoid
exception whenever possible, and it made the code faster.

Victor

Le vendredi 15 mars 2019, Brett Cannon <brett at python.org> a ?crit :
> The steering council felt the PEP was too broad and not focused enough.
Discussions about adding more attributes to built-in exceptions can
continue on the issue tracker on a per-exception basis (and obviously here
for any broader points, e.g. performance implications as I know that has
come up before when the idea of storing relevant objects on exceptions).
> Thanks to Sebastian Kreft for taking the time to write the PEP in the
first place.
>

-- 
Night gathers, and now my watch begins. It shall not end until my death.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190316/7b0e0a07/attachment.html>

From vano at mail.mipt.ru  Sat Mar 16 03:10:24 2019
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Sat, 16 Mar 2019 10:10:24 +0300
Subject: [Python-Dev] Tests for internal functionality
Message-ID: <87f2d4c9-b9e8-6a63-e4a8-e464a0d63be6@mail.mipt.ru>

In https://github.com/python/cpython/pull/6541 , I was requested to add tests for an internal C function.

As I wrote in https://github.com/python/cpython/pull/6541#issuecomment-445514807 , it's not clear from the codebase

1) where tests for internal (as opposed to public) functionality should be located
 ??? * I only ran across very few tests for private functionality and they were located alongside the public tests. See e.g. 
https://github.com/python/cpython/pull/12140 -- site._getuserbase is a private function.

2) what spec one should test against since there's no official docs for internal functionality

That time, they let it slide, but of course this is not something to make a habit of. It still bothers me that I left unfinished business there.
Any thoughts?

-- 
Regards,
Ivan


From phil.angerer at gmail.com  Sat Mar 16 10:12:38 2019
From: phil.angerer at gmail.com (Philipp Angerer)
Date: Sat, 16 Mar 2019 15:12:38 +0100
Subject: [Python-Dev] Deferring PEP 536 -- Final Grammar for Literal
 String Interpolation
In-Reply-To: <CAP1=2W5ELaSEuTaAc1_j+NYAgocRwoeX=oE0LxNgDqSNDL+b+w@mail.gmail.com>
References: <CAP1=2W5ELaSEuTaAc1_j+NYAgocRwoeX=oE0LxNgDqSNDL+b+w@mail.gmail.com>
Message-ID: <CACSGjE-YXB0ByOh4B2VWv159Kai3AySACNh_c2UNyKOvGwzHRQ@mail.gmail.com>

Sorry for not having one, but my C is really not up to speed.

Help is therefore very welcome of course!

Am Fr., 15. M?rz 2019 um 23:32 Uhr schrieb Brett Cannon <brett at python.org>:

> The steering council decided to defer PEP 536 until an implementation is
> available.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190316/d1991953/attachment.html>

From stefano.borini at gmail.com  Sat Mar 16 11:46:26 2019
From: stefano.borini at gmail.com (Stefano Borini)
Date: Sat, 16 Mar 2019 15:46:26 +0000
Subject: [Python-Dev] Reject PEP 472: Support for indexing with keyword
 arguments
In-Reply-To: <CAP1=2W7RHwOhSqMmYyRZjHJJGL55AX1dBEhEFDvORO_T5hgo_w@mail.gmail.com>
References: <CAP1=2W7RHwOhSqMmYyRZjHJJGL55AX1dBEhEFDvORO_T5hgo_w@mail.gmail.com>
Message-ID: <CAFp+ze1EDBPMA+G8Cb5gv81e=a7qqy=_BAJJ-FAohQsnauboww@mail.gmail.com>

Hello, I am one of the authors of the PEP.
I agree with your sentiment. I still think it might be useful in some
contexts and I got some positive reactions here and there, but as you said
it does not seem to provide a lot of added value.


On Fri, 15 Mar 2019 at 21:50, Brett Cannon <brett at python.org> wrote:

> The idea never seemed to gain any traction over its near 5 years in
> existence as a PEP.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/makepython%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190316/9e1242b3/attachment.html>

From tjreedy at udel.edu  Sat Mar 16 17:23:30 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 16 Mar 2019 17:23:30 -0400
Subject: [Python-Dev] Tests for internal functionality
In-Reply-To: <87f2d4c9-b9e8-6a63-e4a8-e464a0d63be6@mail.mipt.ru>
References: <87f2d4c9-b9e8-6a63-e4a8-e464a0d63be6@mail.mipt.ru>
Message-ID: <q6jpgk$2uhn$1@blaine.gmane.org>

On 3/16/2019 3:10 AM, Ivan Pozdeev via Python-Dev wrote:
> In https://github.com/python/cpython/pull/6541 , I was requested to add 
> tests for an internal C function.
> 
> As I wrote in 
> https://github.com/python/cpython/pull/6541#issuecomment-445514807 , 
> it's not clear from the codebase
> 
> 1) where tests for internal (as opposed to public) functionality should 
> be located

In the same file where they would be if public, which is what you seemed 
to find.  Label somehow, such as 'class InternalsTest(TestCase):' or 
'test__private' (with double __ indicating test of _private).

>  ??? * I only ran across very few tests for private functionality and 
> they were located alongside the public tests. See e.g. 
> https://github.com/python/cpython/pull/12140 -- site._getuserbase is a 
> private function.
> 
> 2) what spec one should test against since there's no official docs for 
> internal functionality

Is there an informal spec in the form of docstring or comments?  Besides 
that, test against current behavior.  The main purpose of retro-testing 
is to detect *unintended* changes of behavior (regressions).

> That time, they let it slide, but of course this is not something to 
> make a habit of. It still bothers me that I left unfinished business there.
> Any thoughts?

-- 
Terry Jan Reedy



From Peixing.Xin at windriver.com  Mon Mar 18 03:31:47 2019
From: Peixing.Xin at windriver.com (Xin, Peixing)
Date: Mon, 18 Mar 2019 07:31:47 +0000
Subject: [Python-Dev] Anyone can help to proceed these 2 PRs?
Message-ID: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9E134@ALA-MBD.corp.ad.wrs.com>

Hi, Experts:

I have 2 PRs pending there to get review for days. Anyone can give your hands? Thanks in advance. BTW, they are about CPython working on VxWorks 7.

https://github.com/python/cpython/pull/12118
https://github.com/python/cpython/pull/12305

Thanks,
Peixing

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190318/cdd89be9/attachment.html>

From larry at hastings.org  Mon Mar 18 16:38:41 2019
From: larry at hastings.org (Larry Hastings)
Date: Mon, 18 Mar 2019 13:38:41 -0700
Subject: [Python-Dev] Python 3.5.7 is now available
Message-ID: <d3a2addd-46f6-eff9-cc0e-c53acc990faf@hastings.org>


On behalf of the Python development community, I'm chuffed to announce 
the availability of Python 3.5.7.

Python 3.5 is in "security fixes only" mode.? It only accepts security 
fixes, not conventional bug fixes, and the release is source-only.

And you can find Python 3.5.7rc1 here:

    https://www.python.org/downloads/release/python-357/


Best wishes,



//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190318/4101ad26/attachment.html>

From larry at hastings.org  Mon Mar 18 16:42:31 2019
From: larry at hastings.org (Larry Hastings)
Date: Mon, 18 Mar 2019 13:42:31 -0700
Subject: [Python-Dev] Python 3.4.10 is now available
Message-ID: <c062a27c-3be9-58de-b7cd-4a706f736433@hastings.org>


On behalf of the Python development community, I'm proud--if slightly 
sad--to announce the availability of Python 3.4.10.

Python 3.4.10 was released in "security fixes only" mode.? It only 
contains security fixes, not conventional bug fixes, and it is a 
source-only release.

Python 3.4.10 is the final release in the Python 3.4 series. As of this 
release, the 3.4 branch has been retired, no further changes to 3.4 will 
be accepted, and no new releases will be made.? This is standard Python 
policy; Python releases get five years of support and are then retired.

If you're still using Python 3.4, you should consider upgrading to the 
current version--3.7.2 as of this writing.? Newer versions of Python
have many new features, performance improvements, and bug fixes, which 
should all serve to enhance your Python programming experience.

We in the Python core development community thank you for your interest 
in 3.4, and we wish you all the best!


You can find Python 3.4.10 here:

    https://www.python.org/downloads/release/python-3410/


One I completely finish the Python 3.4.10 release process, I will retire 
as Python 3.4 Release Manager.? I'll still be Python 3.5 Release Manager 
for another eighteen months or so.


Python 3.4 is OVER!
https://www.youtube.com/watch?v=YlGqN3AKOsA



//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190318/64974083/attachment.html>

From Paul.Monson at microsoft.com  Mon Mar 18 16:10:53 2019
From: Paul.Monson at microsoft.com (Paul Monson)
Date: Mon, 18 Mar 2019 20:10:53 +0000
Subject: [Python-Dev] Testing cross-compiled python (windows arm32) from
 build bot
Message-ID: <CY4PR2101MB080437C3EE50F1184D4496D6F9470@CY4PR2101MB0804.namprd21.prod.outlook.com>

Hello,

For windows arm32 I would like to be able run tests in the buildbot.
Windows arm32 programs built with MSVC are always cross-compiled.
This means I need to build cpython on x86/x64 and then deploy the build artifacts and run the tests on an ARM32 device.

Is there any documentation or examples of how to do this?
If not, are there any recommendations for how to achieve this?

Thanks,
Paul Monson

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190318/a50835e0/attachment.html>

From doko at ubuntu.com  Mon Mar 18 17:32:02 2019
From: doko at ubuntu.com (Matthias Klose)
Date: Mon, 18 Mar 2019 22:32:02 +0100
Subject: [Python-Dev] Testing cross-compiled python (windows arm32) from
 build bot
In-Reply-To: <CY4PR2101MB080437C3EE50F1184D4496D6F9470@CY4PR2101MB0804.namprd21.prod.outlook.com>
References: <CY4PR2101MB080437C3EE50F1184D4496D6F9470@CY4PR2101MB0804.namprd21.prod.outlook.com>
Message-ID: <61bfdffc-e3cc-ee02-2f55-93f8658d23ce@ubuntu.com>

On 18.03.19 21:10, Paul Monson via Python-Dev wrote:
> Hello,
> 
> For windows arm32 I would like to be able run tests in the buildbot.
> Windows arm32 programs built with MSVC are always cross-compiled.
> This means I need to build cpython on x86/x64 and then deploy the build artifacts and run the tests on an ARM32 device.
> 
> Is there any documentation or examples of how to do this?
> If not, are there any recommendations for how to achieve this?

I'm not aware of any documentation. For arm-linux-gnueabihf I once had a setup
with two chroots, the first one for cross-building python, and then the second
one for running the tests using qemu.  Two separate chroots to make sure that I
don't run any target code during the build.  The build location bind mounted in
both chroots to avoid the copy step.

Matthias

From brett at python.org  Mon Mar 18 17:42:14 2019
From: brett at python.org (Brett Cannon)
Date: Mon, 18 Mar 2019 14:42:14 -0700
Subject: [Python-Dev] Reject PEP 472: Support for indexing with keyword
 arguments
In-Reply-To: <CAFp+ze1EDBPMA+G8Cb5gv81e=a7qqy=_BAJJ-FAohQsnauboww@mail.gmail.com>
References: <CAP1=2W7RHwOhSqMmYyRZjHJJGL55AX1dBEhEFDvORO_T5hgo_w@mail.gmail.com>
 <CAFp+ze1EDBPMA+G8Cb5gv81e=a7qqy=_BAJJ-FAohQsnauboww@mail.gmail.com>
Message-ID: <CAP1=2W4C=bhCmqtDnoF2QVaeqw253sCJfEhkU489fJTy8apCMg@mail.gmail.com>

And thanks for the PEP, Stefano!

(I meant to say "thanks" in the email but I simply forgot :( ).

On Sat, Mar 16, 2019 at 8:46 AM Stefano Borini <stefano.borini at gmail.com>
wrote:

> Hello, I am one of the authors of the PEP.
> I agree with your sentiment. I still think it might be useful in some
> contexts and I got some positive reactions here and there, but as you said
> it does not seem to provide a lot of added value.
>
>
> On Fri, 15 Mar 2019 at 21:50, Brett Cannon <brett at python.org> wrote:
>
>> The idea never seemed to gain any traction over its near 5 years in
>> existence as a PEP.
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/makepython%40gmail.com
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190318/2ba8248b/attachment.html>

From raymond.hettinger at gmail.com  Mon Mar 18 18:41:15 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 18 Mar 2019 15:41:15 -0700
Subject: [Python-Dev] Is XML serialization output guaranteed to be bytewise
 identical forever?
Message-ID: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>

We're having a super interesting discussion on https://bugs.python.org/issue34160 .  It is now marked as a release blocker and warrants a broader discussion.

Our problem is that at least two distinct and important users have written tests that depend on exact byte-by-byte comparisons of the final serialization.  So any changes to the XML modules will break those tests (not the applications themselves, just the test cases that assume the output will be forever, byte-by-byte identical).  

In theory, the tests are incorrectly designed and should not treat the module output as a canonical normal form.  In practice, doing an equality test on the output is the simplest, most obvious approach, and likely is being done in other packages we don't know about yet.

With pickle, json, and __repr__, the usual way to write a test is to verify a roundtrip:  assert pickle.loads(pickle.dumps(data)) == data.  With XML, the problem is that the DOM doesn't have an equality operator.  The user is left with either testing specific fragments with element.find(xpath) or with using a standards compliant canonicalization package (not available from us). Neither option is pleasant.

The code in the current 3.8 alpha differs from 3.7 in that it removes attribute sorting and instead preserves the order the user specified when creating an element.  As far as I can tell, there is no objection to this as a feature.  The problem is what to do about the existing tests in third-party code, what guarantees we want to make going forward, and what do we recommend as a best practice for testing XML generation.

Things we can do:

1) Revert back to the 3.7 behavior. This of course, makes all the test pass :-)  The downside is that it perpetuates the practice of bytewise equality tests and locks in all implementation quirks forever.  I don't know of anyone advocating this option, but it is the simplest thing to do.

2). Go into every XML module and add attribute sorting options to each function that generate xml.  This gives users a way to make their tests pass for now. There are several downsides. a) It grows the API in a way that is inconsistent with all the other XML packages I've seen. b) We'll have to test, maintain, and document the API forever -- the API is already large and time consuming to teach. c) It perpetuates the notion that bytewise equality tests are the right thing to do, so we'll have this problem again if substitute in another code generator or alter any of the other implementation quirks (i.e. how CDATA sections are serialized).

3) Add a standards compliant canonicalization tool (see https://en.wikipedia.org/wiki/Canonical_XML ).  This is likely to be the right-way-to-do-it but takes time and energy.

4) Fix the tests in the third-party modules to be more focused on their actual test objectives, the semantics of the generated XML rather than the exact serialization.  This option would seem like the right-thing-to-do but it isn't trivial because the entire premise of the existing test is invalid.  For every case, we'll actually have to think through what the test objective really is.

Of these, option 2 is my least preferred.  Ideally, we don't guarantee bytewise identical output across releases, and ideally we don't grow a new API that perpetuates the issue. That said, I'm not wedded to any of these options and just want us to do what is best for the users in the long run.

Regardless of option chosen, we should make explicit whether on not the Python standard library modules guarantee cross-release bytewise identical output for XML. That is really the core issue here.  Had we had an explicit notice one way or the other, there wouldn't be an issue now.

Any thoughts?



Raymond Hettinger


P.S.   Stefan Behnel is planning to remove attribute sorting from lxml.  On the bug tracker, he has clearly articulated his reasons.



From njs at pobox.com  Mon Mar 18 19:15:30 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 18 Mar 2019 16:15:30 -0700
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
Message-ID: <CAPJVwB=Z0zA=U+BsO50aphqYUtzLO_ROEb1RUYPz-ttwLAHyuQ@mail.gmail.com>

I noticed that your list doesn't include "add a DOM equality operator".
That seems potentially simpler to implement than canonical XML
serialization, and like a useful thing to have in any case. Would it make
sense as an option?

On Mon, Mar 18, 2019, 15:46 Raymond Hettinger <raymond.hettinger at gmail.com>
wrote:

> We're having a super interesting discussion on
> https://bugs.python.org/issue34160 .  It is now marked as a release
> blocker and warrants a broader discussion.
>
> Our problem is that at least two distinct and important users have written
> tests that depend on exact byte-by-byte comparisons of the final
> serialization.  So any changes to the XML modules will break those tests
> (not the applications themselves, just the test cases that assume the
> output will be forever, byte-by-byte identical).
>
> In theory, the tests are incorrectly designed and should not treat the
> module output as a canonical normal form.  In practice, doing an equality
> test on the output is the simplest, most obvious approach, and likely is
> being done in other packages we don't know about yet.
>
> With pickle, json, and __repr__, the usual way to write a test is to
> verify a roundtrip:  assert pickle.loads(pickle.dumps(data)) == data.  With
> XML, the problem is that the DOM doesn't have an equality operator.  The
> user is left with either testing specific fragments with
> element.find(xpath) or with using a standards compliant canonicalization
> package (not available from us). Neither option is pleasant.
>
> The code in the current 3.8 alpha differs from 3.7 in that it removes
> attribute sorting and instead preserves the order the user specified when
> creating an element.  As far as I can tell, there is no objection to this
> as a feature.  The problem is what to do about the existing tests in
> third-party code, what guarantees we want to make going forward, and what
> do we recommend as a best practice for testing XML generation.
>
> Things we can do:
>
> 1) Revert back to the 3.7 behavior. This of course, makes all the test
> pass :-)  The downside is that it perpetuates the practice of bytewise
> equality tests and locks in all implementation quirks forever.  I don't
> know of anyone advocating this option, but it is the simplest thing to do.
>
> 2). Go into every XML module and add attribute sorting options to each
> function that generate xml.  This gives users a way to make their tests
> pass for now. There are several downsides. a) It grows the API in a way
> that is inconsistent with all the other XML packages I've seen. b) We'll
> have to test, maintain, and document the API forever -- the API is already
> large and time consuming to teach. c) It perpetuates the notion that
> bytewise equality tests are the right thing to do, so we'll have this
> problem again if substitute in another code generator or alter any of the
> other implementation quirks (i.e. how CDATA sections are serialized).
>
> 3) Add a standards compliant canonicalization tool (see
> https://en.wikipedia.org/wiki/Canonical_XML ).  This is likely to be the
> right-way-to-do-it but takes time and energy.
>
> 4) Fix the tests in the third-party modules to be more focused on their
> actual test objectives, the semantics of the generated XML rather than the
> exact serialization.  This option would seem like the right-thing-to-do but
> it isn't trivial because the entire premise of the existing test is
> invalid.  For every case, we'll actually have to think through what the
> test objective really is.
>
> Of these, option 2 is my least preferred.  Ideally, we don't guarantee
> bytewise identical output across releases, and ideally we don't grow a new
> API that perpetuates the issue. That said, I'm not wedded to any of these
> options and just want us to do what is best for the users in the long run.
>
> Regardless of option chosen, we should make explicit whether on not the
> Python standard library modules guarantee cross-release bytewise identical
> output for XML. That is really the core issue here.  Had we had an explicit
> notice one way or the other, there wouldn't be an issue now.
>
> Any thoughts?
>
>
>
> Raymond Hettinger
>
>
> P.S.   Stefan Behnel is planning to remove attribute sorting from lxml.
> On the bug tracker, he has clearly articulated his reasons.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190318/2736d92b/attachment-0001.html>

From raymond.hettinger at gmail.com  Mon Mar 18 19:33:51 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 18 Mar 2019 16:33:51 -0700
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CAPJVwB=Z0zA=U+BsO50aphqYUtzLO_ROEb1RUYPz-ttwLAHyuQ@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CAPJVwB=Z0zA=U+BsO50aphqYUtzLO_ROEb1RUYPz-ttwLAHyuQ@mail.gmail.com>
Message-ID: <E2BF691B-636F-4B62-ACC0-A4F574C9118C@gmail.com>



> On Mar 18, 2019, at 4:15 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> I noticed that your list doesn't include "add a DOM equality operator". That seems potentially simpler to implement than canonical XML serialization, and like a useful thing to have in any case. Would it make sense as an option?

Time machine!  St?phane Wirtel just posted a basic semantic comparison between two streams.?   Presumably, there would need to be a range of options for specifying what constitutes equivalence but this is a nice start.

Raymond


? https://bugs.python.org/file48217/test_xml_compare.py


From brian.kuhl at windriver.com  Mon Mar 18 20:15:45 2019
From: brian.kuhl at windriver.com (Kuhl, Brian)
Date: Tue, 19 Mar 2019 00:15:45 +0000
Subject: [Python-Dev] Testing cross-compiled python (windows arm32) from
 build bot
In-Reply-To: <61bfdffc-e3cc-ee02-2f55-93f8658d23ce@ubuntu.com>
References: <CY4PR2101MB080437C3EE50F1184D4496D6F9470@CY4PR2101MB0804.namprd21.prod.outlook.com>
 <61bfdffc-e3cc-ee02-2f55-93f8658d23ce@ubuntu.com>
Message-ID: <6C486AB80682E24095E3E77A408D68F20149A4947B@ALA-MBD.corp.ad.wrs.com>


We (Wind River) are doing it for VxWorks, which is also cross-compiled, and has no concept of chroot you see in Linux. 
The trick is to insert a little switch from local to a remote shell between the build and the test run via ssh (or telnet).
Don't think we gotten anywhere near upstreaming that little change though?
CPython's buildbot is actually easier than most to open source test suites to adapt to cross compile, because everything is run in python after the build, so you only have to make one switch in context.

 Brian

> -----Original Message-----
> From: Python-Dev [mailto:python-dev-
> bounces+brian.kuhl=windriver.com at python.org] On Behalf Of Matthias Klose
> Sent: Monday, March 18, 2019 5:32 PM
> To: Paul Monson <Paul.Monson at microsoft.com>; python-dev at python.org
> Subject: Re: [Python-Dev] Testing cross-compiled python (windows arm32) from
> build bot
> 
> On 18.03.19 21:10, Paul Monson via Python-Dev wrote:
> > Hello,
> >
> > For windows arm32 I would like to be able run tests in the buildbot.
> > Windows arm32 programs built with MSVC are always cross-compiled.
> > This means I need to build cpython on x86/x64 and then deploy the build
> artifacts and run the tests on an ARM32 device.
> >
> > Is there any documentation or examples of how to do this?
> > If not, are there any recommendations for how to achieve this?
> 
> I'm not aware of any documentation. For arm-linux-gnueabihf I once had a
> setup with two chroots, the first one for cross-building python, and then the
> second one for running the tests using qemu.  Two separate chroots to make
> sure that I don't run any target code during the build.  The build location bind
> mounted in both chroots to avoid the copy step.
> 
> Matthias
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-
> dev/brian.kuhl%40windriver.com

From Peixing.Xin at windriver.com  Mon Mar 18 22:21:58 2019
From: Peixing.Xin at windriver.com (Xin, Peixing)
Date: Tue, 19 Mar 2019 02:21:58 +0000
Subject: [Python-Dev] Can I submit more support of standard library for
 VxWorks after 3.8.0 beta1?
Message-ID: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EE94@ALA-MBD.corp.ad.wrs.com>

Hi, Experts:

Seeing from the Python 3.8.0 schedule(https://www.python.org/dev/peps/pep-0569/#schedule), new features will not be allowed to submit after 3.8.0 beta1. For VxWorks RTOS platform supporting CPython, we are using bpo-31904(https://bugs.python.org/issue31904) for PRs to submit our codes. Now I want to know whether we can add more standard library support for VxWorks AFTER 3.8.0 beta1?

[cid:image001.png at 01D4DE3B.DF628700]

Thanks,
Peixing

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/8abd81fa/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 50332 bytes
Desc: image001.png
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/8abd81fa/attachment-0001.png>

From Peixing.Xin at windriver.com  Mon Mar 18 22:54:29 2019
From: Peixing.Xin at windriver.com (Xin, Peixing)
Date: Tue, 19 Mar 2019 02:54:29 +0000
Subject: [Python-Dev] What is the workflow for announcing tier 2 platform
 support?
Message-ID: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EF06@ALA-MBD.corp.ad.wrs.com>

Hi, Experts:

VxWorks RTOS will release a new version at the end of June. For this new version, we(Wind River) plan to announce the Python support officially here (https://www.python.org/download/other/) for VxWorks as tier 2 platform. Now my 2 questions come:

1.       To be qualified as tier 2 platform here https://www.python.org/download/other/, do I need to get all test cases pass on VxWorks RTOS? Now we have the following PRs for VxWorks. With these PRs merged, we can get around 70% test cases PASS.
[cid:image003.png at 01D4DE3F.A336F1E0]

2.       If we reach the announcement level, what is the workflow to update the website statements here https://www.python.org/download/other/. The following picture shows what we want to show.

[cid:image002.png at 01D4DE42.1EB0FB70]

Thanks,
Peixing

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/65b8207e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 15039 bytes
Desc: image003.png
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/65b8207e/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 130305 bytes
Desc: image002.png
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/65b8207e/attachment-0003.png>

From tjreedy at udel.edu  Tue Mar 19 00:44:30 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 19 Mar 2019 00:44:30 -0400
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
Message-ID: <q6ps3g$2e1q$1@blaine.gmane.org>

On 3/18/2019 6:41 PM, Raymond Hettinger wrote:
> We're having a super interesting discussion on https://bugs.python.org/issue34160 .  It is now marked as a release blocker and warrants a broader discussion.
> 
> Our problem is that at least two distinct and important users have written tests that depend on exact byte-by-byte comparisons of the final serialization.  So any changes to the XML modules will break those tests (not the applications themselves, just the test cases that assume the output will be forever, byte-by-byte identical).
> 
> In theory, the tests are incorrectly designed and should not treat the module output as a canonical normal form.  In practice, doing an equality test on the output is the simplest, most obvious approach, and likely is being done in other packages we don't know about yet.
> 
> With pickle, json, and __repr__, the usual way to write a test is to verify a roundtrip:  assert pickle.loads(pickle.dumps(data)) == data.  With XML, the problem is that the DOM doesn't have an equality operator.  The user is left with either testing specific fragments with element.find(xpath) or with using a standards compliant canonicalization package (not available from us). Neither option is pleasant.
> 
> The code in the current 3.8 alpha differs from 3.7 in that it removes attribute sorting and instead preserves the order the user specified when creating an element.  As far as I can tell, there is no objection to this as a feature.  The problem is what to do about the existing tests in third-party code, what guarantees we want to make going forward, and what do we recommend as a best practice for testing XML generation.
> 
> Things we can do:
> 
> 1) Revert back to the 3.7 behavior. This of course, makes all the test pass :-)  The downside is that it perpetuates the practice of bytewise equality tests and locks in all implementation quirks forever.  I don't know of anyone advocating this option, but it is the simplest thing to do.

If it comes down to doing *something* to unblock the release ...
1b) Revert to 3.7 *and* document that byte equality with current ouput 
is *not* guaranteed.

> 2). Go into every XML module and add attribute sorting options to each function that generate xml.  This gives users a way to make their tests pass for now. There are several downsides. a) It grows the API in a way that is inconsistent with all the other XML packages I've seen. b) We'll have to test, maintain, and document the API forever -- the API is already large and time consuming to teach. c) It perpetuates the notion that bytewise equality tests are the right thing to do, so we'll have this problem again if substitute in another code generator or alter any of the other implementation quirks (i.e. how CDATA sections are serialized).
> 
> 3) Add a standards compliant canonicalization tool (see https://en.wikipedia.org/wiki/Canonical_XML ).  This is likely to be the right-way-to-do-it but takes time and energy.
> 
> 4) Fix the tests in the third-party modules to be more focused on their actual test objectives, the semantics of the generated XML rather than the exact serialization.  This option would seem like the right-thing-to-do but it isn't trivial because the entire premise of the existing test is invalid.  For every case, we'll actually have to think through what the test objective really is.
> 
> Of these, option 2 is my least preferred.  Ideally, we don't guarantee bytewise identical output across releases, and ideally we don't grow a new API that perpetuates the issue. That said, I'm not wedded to any of these options and just want us to do what is best for the users in the long run.

The point of 1b would be to give us time to do that if more is needed.

> Regardless of option chosen, we should make explicit whether on not the Python standard library modules guarantee cross-release bytewise identical output for XML. That is really the core issue here.  Had we had an explicit notice one way or the other, there wouldn't be an issue now.

I have not read the XML docs but based on this and the issue discussion 
and what I think our general guarantee policy has been, I would consider 
that there is not one.  (I am thinking about things like garbage 
collection, stable sorting, and set/dict iteration order.)

-- 
Terry Jan Reedy


From greg at krypto.org  Tue Mar 19 02:30:43 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 18 Mar 2019 23:30:43 -0700
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <q6ps3g$2e1q$1@blaine.gmane.org>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6ps3g$2e1q$1@blaine.gmane.org>
Message-ID: <CAGE7PNJxv0T74mEM-Pq1vdMdQaQQiLwnMujHtTT=OVFdwMO6mA@mail.gmail.com>

On Mon, Mar 18, 2019 at 9:44 PM Terry Reedy <tjreedy at udel.edu> wrote:

> On 3/18/2019 6:41 PM, Raymond Hettinger wrote:
> > We're having a super interesting discussion on
> https://bugs.python.org/issue34160 .  It is now marked as a release
> blocker and warrants a broader discussion.
> >
> > Our problem is that at least two distinct and important users have
> written tests that depend on exact byte-by-byte comparisons of the final
> serialization.  So any changes to the XML modules will break those tests
> (not the applications themselves, just the test cases that assume the
> output will be forever, byte-by-byte identical).
> >
> > In theory, the tests are incorrectly designed and should not treat the
> module output as a canonical normal form.  In practice, doing an equality
> test on the output is the simplest, most obvious approach, and likely is
> being done in other packages we don't know about yet.
> >
> > With pickle, json, and __repr__, the usual way to write a test is to
> verify a roundtrip:  assert pickle.loads(pickle.dumps(data)) == data.  With
> XML, the problem is that the DOM doesn't have an equality operator.  The
> user is left with either testing specific fragments with
> element.find(xpath) or with using a standards compliant canonicalization
> package (not available from us). Neither option is pleasant.
> >
> > The code in the current 3.8 alpha differs from 3.7 in that it removes
> attribute sorting and instead preserves the order the user specified when
> creating an element.  As far as I can tell, there is no objection to this
> as a feature.  The problem is what to do about the existing tests in
> third-party code, what guarantees we want to make going forward, and what
> do we recommend as a best practice for testing XML generation.
> >
> > Things we can do:
> >
> > 1) Revert back to the 3.7 behavior. This of course, makes all the test
> pass :-)  The downside is that it perpetuates the practice of bytewise
> equality tests and locks in all implementation quirks forever.  I don't
> know of anyone advocating this option, but it is the simplest thing to do.
>
> If it comes down to doing *something* to unblock the release ...
> 1b) Revert to 3.7 *and* document that byte equality with current ouput
> is *not* guaranteed.
>
> > 2). Go into every XML module and add attribute sorting options to each
> function that generate xml.  This gives users a way to make their tests
> pass for now. There are several downsides. a) It grows the API in a way
> that is inconsistent with all the other XML packages I've seen. b) We'll
> have to test, maintain, and document the API forever -- the API is already
> large and time consuming to teach. c) It perpetuates the notion that
> bytewise equality tests are the right thing to do, so we'll have this
> problem again if substitute in another code generator or alter any of the
> other implementation quirks (i.e. how CDATA sections are serialized).
> >
> > 3) Add a standards compliant canonicalization tool (see
> https://en.wikipedia.org/wiki/Canonical_XML ).  This is likely to be the
> right-way-to-do-it but takes time and energy.

>
> > 4) Fix the tests in the third-party modules to be more focused on their
> actual test objectives, the semantics of the generated XML rather than the
> exact serialization.  This option would seem like the right-thing-to-do but
> it isn't trivial because the entire premise of the existing test is
> invalid.  For every case, we'll actually have to think through what the
> test objective really is.

>
> > Of these, option 2 is my least preferred.  Ideally, we don't guarantee
> bytewise identical output across releases, and ideally we don't grow a new
> API that perpetuates the issue. That said, I'm not wedded to any of these
> options and just want us to do what is best for the users in the long run.
>

For (1) - don't revert in 3.8 - Do not worry about order or formatting of
serialized data changing between major Python releases.  change in 3.8?
that's 100% okay.  This already happens all the time between Python
releases.  We've changed dict iteration order between releases twice this
decade.

Within point releases of stable versions, ie 3.7.x? Up to the release
manager; it is semi-rude to change something like this within a stable
release unless there is a good reason, but we *believe* have done it
before. A general rule of thumb is to try not to without good reason though
unless the code to avoid doing so would be over complicated.

It is always the user code depending on the non-declared ordering within
output that is wrong, when we preserve it we're only doing them a temporary
favor that ultimately allows more problems to grow in the future.  Nobody
should use a text comparison on serialized data not explicitly stated as
canonical and call that test good by any standard unless you are writing a
test that for canonical output by a library that explicitly guarantees its
output will be canonical.

Agreed that your option (2) is not good for the world. The best thing to do
API wise is intentionally force some randomness of emitted order so that
people do not accidentally stumble into writing tests like this in the
presence of multiple attributes being emitted. If people need some
canonical form of output they need to explicitly ask for it.

The hash randomization work we did years ago for dicts exposed real bugs in
code all around the world when it was enabled (we turned it on internally
at work soon after it was implemented and had thousands of tests to fix
including exposing several hidden actual bugs in code).  A lot of the fixes
were typical "parse the structured data and compare structures" cleanups of
code that was previously cheating and getting away with a golden value
string comparison.  (people are always going to write code like that, it is
trivial to write, and if it doesn't appear to be flaky they'll consider it
good enough and leave future breakage for the next maintainers)

Option (3) seems better served via PyPI modules.

Option (4): fix code testing golden value str/bytes equality of serialized
formatted data is always a good thing to do.  sometimes easy, occasionally
quite frustrating as you find fundamentally flawed tests or, worse, flawed
API designs that relied on the non-guaranteed order.  can we fix everyones
code?  nope.  it isn't our responsibility, but if we've identified widely
used projects with the issue at least give them a heads up about their
bug(s) as soon as possible with repro instructions in the relevant issue
trackers.

my 2cents,
-gps


>
> The point of 1b would be to give us time to do that if more is needed.
>
> > Regardless of option chosen, we should make explicit whether on not the
> Python standard library modules guarantee cross-release bytewise identical
> output for XML. That is really the core issue here.  Had we had an explicit
> notice one way or the other, there wouldn't be an issue now.
>
> I have not read the XML docs but based on this and the issue discussion
> and what I think our general guarantee policy has been, I would consider
> that there is not one.  (I am thinking about things like garbage
> collection, stable sorting, and set/dict iteration order.)
>
> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190318/1208eee0/attachment.html>

From storchaka at gmail.com  Tue Mar 19 04:13:00 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 19 Mar 2019 10:13:00 +0200
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
Message-ID: <q6q8af$3i67$1@blaine.gmane.org>

19.03.19 00:41, Raymond Hettinger ????:
> 3) Add a standards compliant canonicalization tool (see https://en.wikipedia.org/wiki/Canonical_XML ).  This is likely to be the right-way-to-do-it but takes time and energy.
> 
> 4) Fix the tests in the third-party modules to be more focused on their actual test objectives, the semantics of the generated XML rather than the exact serialization.  This option would seem like the right-thing-to-do but it isn't trivial because the entire premise of the existing test is invalid.  For every case, we'll actually have to think through what the test objective really is.

I think the combination of options 3 and 4 is the right thing. Not 
always the stable output is needed, in these cases option 4 should be 
considered. But there are valid use cases for the stable output, in 
these cases we need to provide an alternative in the stdlib. I am 
working on this.


From ned at nedbatchelder.com  Tue Mar 19 07:53:11 2019
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Tue, 19 Mar 2019 07:53:11 -0400
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <q6q8af$3i67$1@blaine.gmane.org>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6q8af$3i67$1@blaine.gmane.org>
Message-ID: <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>

On 3/19/19 4:13 AM, Serhiy Storchaka wrote:
> 19.03.19 00:41, Raymond Hettinger ????:
>> 3) Add a standards compliant canonicalization tool (see 
>> https://en.wikipedia.org/wiki/Canonical_XML ).? This is likely to be 
>> the right-way-to-do-it but takes time and energy.
>>
>> 4) Fix the tests in the third-party modules to be more focused on 
>> their actual test objectives, the semantics of the generated XML 
>> rather than the exact serialization.? This option would seem like the 
>> right-thing-to-do but it isn't trivial because the entire premise of 
>> the existing test is invalid.? For every case, we'll actually have to 
>> think through what the test objective really is.
>
> I think the combination of options 3 and 4 is the right thing. Not 
> always the stable output is needed, in these cases option 4 should be 
> considered. But there are valid use cases for the stable output, in 
> these cases we need to provide an alternative in the stdlib. I am 
> working on this.

Option 4 is misleading.? Is anyone here really offering to "fix the 
tests in third-party modules"?? Option 4 is actually, "do nothing, and 
let a multitude of projects figure out how to fix their tests, slowing 
progress in those projects as they try to support Python 3.8."

In my case, the test code has a generic function to compare an actual 
directory of files to an expected directory of files, so it isn't quite 
as simple as "just use the right XML comparison."? And I support Python 
2.7, 3.5, etc, so tests still need to work under those versions.? None 
of this is impossible, but please try not to preach to us maintainers 
that we are doing it wrong, that it will be easy to fix, etc.? Using 
language like "the entire premise of the test is invalid" seems 
needlessly condescending.

As one of the suggested solutions, a DOM comparison is not enough. I 
don't just want to know that my actual XML is different than my expected 
XML.? I want to know where and how it differs.

Textual comparison may be the "wrong" way to check XML, but it gives me 
many tools for working with the test results.? It was simple and it 
worked.? Now in Python 3.8, because Python doesn't want to add an 
optional flag to continue doing what it has always done, I need to 
re-engineer my tests.

--Ned.

>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com

From mertz at gnosis.cx  Tue Mar 19 08:11:34 2019
From: mertz at gnosis.cx (David Mertz)
Date: Tue, 19 Mar 2019 08:11:34 -0400
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
Message-ID: <CAEbHw4b3JkvO-OkAMMdCzGfUiJXqwte1XhVj-JhndSHQO6FStg@mail.gmail.com>

In my opinion, any test that relied on a non-promised accident of
serialization is broken today. Very often, such bad tests mask bad
production code that makes the same unsafe assumptions.

This is similar to tests that assumed a certain dictionary order, before we
got guaranteed insertion order. Or like tests that rely on object identity
of short strings or small ints. Or like non-guaranteed identities in
pickles across versions.

In a way, this case makes bugs worse because they are not only a Python
internal matter. XML is used to communicate among many tools and
programming languages, and relying on assumptions those other tools will
not follow us a bad habit. Sure, most tests probably don't get to the point
of touching those external tools themselves, but staying from bad
assumptions about the domain isn't best practices.

That said, I think aN XML canonicalization function is generally a good
thing for Python to have. But it shouldn't be a stopper in releases.

On Mon, Mar 18, 2019, 6:47 PM Raymond Hettinger <raymond.hettinger at gmail.com>
wrote:

> We're having a super interesting discussion on
> https://bugs.python.org/issue34160 .  It is now marked as a release
> blocker and warrants a broader discussion.
>
> Our problem is that at least two distinct and important users have written
> tests that depend on exact byte-by-byte comparisons of the final
> serialization.  So any changes to the XML modules will break those tests
> (not the applications themselves, just the test cases that assume the
> output will be forever, byte-by-byte identical).
>
> In theory, the tests are incorrectly designed and should not treat the
> module output as a canonical normal form.  In practice, doing an equality
> test on the output is the simplest, most obvious approach, and likely is
> being done in other packages we don't know about yet.
>
> With pickle, json, and __repr__, the usual way to write a test is to
> verify a roundtrip:  assert pickle.loads(pickle.dumps(data)) == data.  With
> XML, the problem is that the DOM doesn't have an equality operator.  The
> user is left with either testing specific fragments with
> element.find(xpath) or with using a standards compliant canonicalization
> package (not available from us). Neither option is pleasant.
>
> The code in the current 3.8 alpha differs from 3.7 in that it removes
> attribute sorting and instead preserves the order the user specified when
> creating an element.  As far as I can tell, there is no objection to this
> as a feature.  The problem is what to do about the existing tests in
> third-party code, what guarantees we want to make going forward, and what
> do we recommend as a best practice for testing XML generation.
>
> Things we can do:
>
> 1) Revert back to the 3.7 behavior. This of course, makes all the test
> pass :-)  The downside is that it perpetuates the practice of bytewise
> equality tests and locks in all implementation quirks forever.  I don't
> know of anyone advocating this option, but it is the simplest thing to do.
>
> 2). Go into every XML module and add attribute sorting options to each
> function that generate xml.  This gives users a way to make their tests
> pass for now. There are several downsides. a) It grows the API in a way
> that is inconsistent with all the other XML packages I've seen. b) We'll
> have to test, maintain, and document the API forever -- the API is already
> large and time consuming to teach. c) It perpetuates the notion that
> bytewise equality tests are the right thing to do, so we'll have this
> problem again if substitute in another code generator or alter any of the
> other implementation quirks (i.e. how CDATA sections are serialized).
>
> 3) Add a standards compliant canonicalization tool (see
> https://en.wikipedia.org/wiki/Canonical_XML ).  This is likely to be the
> right-way-to-do-it but takes time and energy.
>
> 4) Fix the tests in the third-party modules to be more focused on their
> actual test objectives, the semantics of the generated XML rather than the
> exact serialization.  This option would seem like the right-thing-to-do but
> it isn't trivial because the entire premise of the existing test is
> invalid.  For every case, we'll actually have to think through what the
> test objective really is.
>
> Of these, option 2 is my least preferred.  Ideally, we don't guarantee
> bytewise identical output across releases, and ideally we don't grow a new
> API that perpetuates the issue. That said, I'm not wedded to any of these
> options and just want us to do what is best for the users in the long run.
>
> Regardless of option chosen, we should make explicit whether on not the
> Python standard library modules guarantee cross-release bytewise identical
> output for XML. That is really the core issue here.  Had we had an explicit
> notice one way or the other, there wouldn't be an issue now.
>
> Any thoughts?
>
>
>
> Raymond Hettinger
>
>
> P.S.   Stefan Behnel is planning to remove attribute sorting from lxml.
> On the bug tracker, he has clearly articulated his reasons.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/0288c706/attachment-0001.html>

From solipsis at pitrou.net  Tue Mar 19 08:50:56 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 19 Mar 2019 13:50:56 +0100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
Message-ID: <20190319135056.42957ec2@fsol>


Hi Raymond,

As long as the new serialization order is deterministic (i.e. it's the
same every run and doesn't depend on e.g. hash randomization), then I
think it's fine to change it.

Some more comments / questions:

> 2). Go into every XML module and add attribute sorting options to each function that generate xml.

What do you mean with "every XML module"? Are there many of them?

> Regardless of option chosen, we should make explicit whether on not the Python standard library modules guarantee cross-release bytewise identical output for XML.

IMO we certainly shouldn't.  XML is a serialization format used for
machine interoperability (even though "human-editable" was one of its
selling points at the start, rather misguidingly).  However, the output
should ideally be stable and deterministic accross all releases of a
given bugfix branch.

(i.e., if I run the same code multiple times on all 3.7.x versions, I
should get always the same output)

Regards

Antoine.



From stephane at wirtel.be  Tue Mar 19 09:03:11 2019
From: stephane at wirtel.be (=?UTF-8?Q?St=c3=a9phane_Wirtel?=)
Date: Tue, 19 Mar 2019 14:03:11 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
Message-ID: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>

Hi,

Context: raise a warning or remove tempfile.mktemp()
BPO: https://bugs.python.org/issue36309

Since 2.3, this function is deprecated in the documentation, just in the
documentation. In the code, there is a commented RuntimeWarning.
Commented by Guido in 2002, because the warning was too annoying (and I
understand ;-)).

So, in this BPO, we start to discuss about the future of this function
and Serhiy proposed to discuss on the Python-dev mailing list.

Question: Should we drop it or add a (Pending)DeprecationWarning?

Suggestion and timeline:

3.8, we raise a PendingDeprecationWarning
    * update the code
    * update the documentation
    * update the tests
      (check a PendingDeprecationWarning if sys.version_info == 3.8)

3.9, we change PendingDeprecationWarning to DeprecationWarning
      (check DeprecationWarning if sys.version_info == 3.9)

3.9+, we drop tempfile.mktemp()

What do you suggest?

Have a nice day and thank you for your feedback.

From timothy.c.delaney at gmail.com  Tue Mar 19 09:10:32 2019
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 20 Mar 2019 00:10:32 +1100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CAEbHw4b3JkvO-OkAMMdCzGfUiJXqwte1XhVj-JhndSHQO6FStg@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CAEbHw4b3JkvO-OkAMMdCzGfUiJXqwte1XhVj-JhndSHQO6FStg@mail.gmail.com>
Message-ID: <CAN8CLgkB=206yqkvBc7dGXNRR7YmyCERJT-XjbyjvCPgqeGdVg@mail.gmail.com>

On Tue, 19 Mar 2019 at 23:13, David Mertz <mertz at gnosis.cx> wrote:

> In a way, this case makes bugs worse because they are not only a Python
> internal matter. XML is used to communicate among many tools and
> programming languages, and relying on assumptions those other tools will
> not follow us a bad habit.
>

I have a recent example I encountered where the 3.7 behaviour (sorting
attributes) results in a third-party tool behaving incorrectly, whereas
maintaining attribute order works correctly. The particular case was using
HTML <meta> tags for importing into Calibre for converting to an ebook. The
most common symptom was that series indexes were sometimes being correctly
imported, and sometimes not. Occasionally other <meta> tags would also fail
to be correctly imported.

Turns out that <meta name="series_index" content="3"/> gave consistently
correct results, whilst <meta content="3" name="series_index"/> was
erratic. And whilst I'd specified the <meta> tags with the name attribute
first, I was then passing the HTML through BeautifulSoup, which sorted the
attributes.

Now Calibre is definitely in the wrong here - it should be able to import
regardless of the order of attributes. But the fact is that there are a lot
of tools out there that are semi-broken in a similar manner.

This to me is an argument to default to maintaining order, but provide a
way for the caller to control the order of attributes when formatting e.g.
pass an ordering function. If you want sorted attributes, pass the built-in
sorted function as your ordering function. But I think that's getting
beyond the scope of this discussion.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190320/1a496c7e/attachment.html>

From storchaka at gmail.com  Tue Mar 19 09:12:18 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 19 Mar 2019 15:12:18 +0200
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6q8af$3i67$1@blaine.gmane.org>
 <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
Message-ID: <q6qprl$2ebs$1@blaine.gmane.org>

19.03.19 13:53, Ned Batchelder ????:
> Option 4 is misleading.? Is anyone here really offering to "fix the 
> tests in third-party modules"?? Option 4 is actually, "do nothing, and 
> let a multitude of projects figure out how to fix their tests, slowing 
> progress in those projects as they try to support Python 3.8."

Any option except option 1 (and option 2 with sorting by default) 
requires changing third-party code. You should either pass additional 
argument to serialization functions, or use special canonization functions.

We should look at the problem from long perspective. Freezing the 
current behavior forever does not look good. If we need to break the 
user code, we should minimize the harm and provide convenient tools for 
reproducing the current behavior. And this is an opportunity to rewrite 
user tests in more appropriate form. In your case textual comparison may 
be the most appropriate form, but this may be not so in other cases.

> Now in Python 3.8, because Python doesn't want to add an 
> optional flag to continue doing what it has always done, I need to 
> re-engineer my tests.

Please wait yet some time. I hope to add canonicalization before the 
first beta.


From storchaka at gmail.com  Tue Mar 19 09:19:04 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 19 Mar 2019 15:19:04 +0200
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <20190319135056.42957ec2@fsol>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <20190319135056.42957ec2@fsol>
Message-ID: <q6qq8a$2vn0$1@blaine.gmane.org>

19.03.19 14:50, Antoine Pitrou ????:
>> 2). Go into every XML module and add attribute sorting options to each function that generate xml.
> 
> What do you mean with "every XML module"? Are there many of them?

ElementTree and minidom. Maybe xmlrpc. And perhaps we need to add 
arguments in calls at higher level where these modules are used.


From solipsis at pitrou.net  Tue Mar 19 09:23:39 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 19 Mar 2019 14:23:39 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
Message-ID: <20190319142339.76cfdc47@fsol>


-1.  Please don't remove tempfile.mktemp().  mktemp() is useful to
create a temporary *name*.  All other tempfile functions create an
actual file and impose additional burden, for example by making the
file unaccessible by other processes.  But sometimes all I want is a
temporary name that an *other* program will create / act on, not Python.
It's a very common use case when writing scripts.

The only reasonable workaround I can think of is to first create a
temporary directory using mkdtemp(), then use a well-known name inside
that directory.  But that has the same security implications AFAICT,
since another process can come and create the file / symlink first.

Regards

Antoine.


On Tue, 19 Mar 2019 14:03:11 +0100
St?phane Wirtel <stephane at wirtel.be> wrote:
> Hi,
> 
> Context: raise a warning or remove tempfile.mktemp()
> BPO: https://bugs.python.org/issue36309
> 
> Since 2.3, this function is deprecated in the documentation, just in the
> documentation. In the code, there is a commented RuntimeWarning.
> Commented by Guido in 2002, because the warning was too annoying (and I
> understand ;-)).
> 
> So, in this BPO, we start to discuss about the future of this function
> and Serhiy proposed to discuss on the Python-dev mailing list.
> 
> Question: Should we drop it or add a (Pending)DeprecationWarning?
> 
> Suggestion and timeline:
> 
> 3.8, we raise a PendingDeprecationWarning
>     * update the code
>     * update the documentation
>     * update the tests
>       (check a PendingDeprecationWarning if sys.version_info == 3.8)
> 
> 3.9, we change PendingDeprecationWarning to DeprecationWarning
>       (check DeprecationWarning if sys.version_info == 3.9)
> 
> 3.9+, we drop tempfile.mktemp()
> 
> What do you suggest?
> 
> Have a nice day and thank you for your feedback.




From storchaka at gmail.com  Tue Mar 19 09:23:35 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 19 Mar 2019 15:23:35 +0200
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CAN8CLgkB=206yqkvBc7dGXNRR7YmyCERJT-XjbyjvCPgqeGdVg@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CAEbHw4b3JkvO-OkAMMdCzGfUiJXqwte1XhVj-JhndSHQO6FStg@mail.gmail.com>
 <CAN8CLgkB=206yqkvBc7dGXNRR7YmyCERJT-XjbyjvCPgqeGdVg@mail.gmail.com>
Message-ID: <q6qqgr$3tpp$1@blaine.gmane.org>

19.03.19 15:10, Tim Delaney ????:
> Now Calibre is definitely in the wrong here - it should be able to 
> import regardless of the order of attributes. But the fact is that there 
> are a lot of tools out there that are semi-broken in a similar manner.

Is not Calibre going to seat on Python 2 forever? This makes it 
non-relevant to the discussion about Python 3.8.


From storchaka at gmail.com  Tue Mar 19 09:32:25 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 19 Mar 2019 15:32:25 +0200
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
Message-ID: <q6qr1d$5497$1@blaine.gmane.org>

19.03.19 15:03, St?phane Wirtel ????:
> Suggestion and timeline:
> 
> 3.8, we raise a PendingDeprecationWarning
>      * update the code
>      * update the documentation
>      * update the tests
>        (check a PendingDeprecationWarning if sys.version_info == 3.8)
> 
> 3.9, we change PendingDeprecationWarning to DeprecationWarning
>        (check DeprecationWarning if sys.version_info == 3.9)
> 
> 3.9+, we drop tempfile.mktemp()

This plan LGTM.

Currently mkdir() is widely used in distutils, Sphinx, pip, setuptools, 
virtualenv, and many other third-party projects, so it will take time to 
fix all these places. But we should do this, because all this code 
likely contains security flaws.


From rosuav at gmail.com  Tue Mar 19 09:37:56 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 20 Mar 2019 00:37:56 +1100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <20190319142339.76cfdc47@fsol>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
Message-ID: <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>

On Wed, Mar 20, 2019 at 12:25 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>
> -1.  Please don't remove tempfile.mktemp().  mktemp() is useful to
> create a temporary *name*.  All other tempfile functions create an
> actual file and impose additional burden, for example by making the
> file unaccessible by other processes.  But sometimes all I want is a
> temporary name that an *other* program will create / act on, not Python.
> It's a very common use case when writing scripts.
>
> The only reasonable workaround I can think of is to first create a
> temporary directory using mkdtemp(), then use a well-known name inside
> that directory.  But that has the same security implications AFAICT,
> since another process can come and create the file / symlink first.

Can't you create a NamedTemporaryFile and permit the other program to
use it? I just tried that (with TiMidity, even though it's quite
capable of just writing to stdout) and it worked fine.

>>> f = tempfile.NamedTemporaryFile(suffix=".flac")
>>> subprocess.check_call(["timidity", "-OF", "-o", f.name, "Music/gp_peers.mid"])
... snip ...
Wrote 29645816/55940900 bytes(52.9949% compressed)
>>> data = f.read()
>>> len(data)
29645816

ChrisA

From solipsis at pitrou.net  Tue Mar 19 09:39:46 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 19 Mar 2019 14:39:46 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <q6qr1d$5497$1@blaine.gmane.org>
Message-ID: <20190319143946.3df5a605@fsol>

On Tue, 19 Mar 2019 15:32:25 +0200
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 19.03.19 15:03, St?phane Wirtel ????:
> > Suggestion and timeline:
> > 
> > 3.8, we raise a PendingDeprecationWarning
> >      * update the code
> >      * update the documentation
> >      * update the tests
> >        (check a PendingDeprecationWarning if sys.version_info == 3.8)
> > 
> > 3.9, we change PendingDeprecationWarning to DeprecationWarning
> >        (check DeprecationWarning if sys.version_info == 3.9)
> > 
> > 3.9+, we drop tempfile.mktemp()  
> 
> This plan LGTM.
> 
> Currently mkdir() is widely used in distutils, Sphinx, pip, setuptools, 
> virtualenv, and many other third-party projects, so it will take time to 
> fix all these places. But we should do this, because all this code 
> likely contains security flaws.

The fact that many projects, including well-maintained ones such Sphinx
or pip, use mktemp(), may be a hint that replacing it is not as easy as
the people writing the Python documentation seem to think.

Regards

Antoine.



From solipsis at pitrou.net  Tue Mar 19 09:51:27 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 19 Mar 2019 14:51:27 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
Message-ID: <20190319145127.0824f196@fsol>

On Wed, 20 Mar 2019 00:37:56 +1100
Chris Angelico <rosuav at gmail.com> wrote:
> On Wed, Mar 20, 2019 at 12:25 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
> >
> >
> > -1.  Please don't remove tempfile.mktemp().  mktemp() is useful to
> > create a temporary *name*.  All other tempfile functions create an
> > actual file and impose additional burden, for example by making the
> > file unaccessible by other processes.  But sometimes all I want is a
> > temporary name that an *other* program will create / act on, not Python.
> > It's a very common use case when writing scripts.
> >
> > The only reasonable workaround I can think of is to first create a
> > temporary directory using mkdtemp(), then use a well-known name inside
> > that directory.  But that has the same security implications AFAICT,
> > since another process can come and create the file / symlink first.  
> 
> Can't you create a NamedTemporaryFile and permit the other program to
> use it? I just tried that (with TiMidity, even though it's quite
> capable of just writing to stdout) and it worked fine.

Does it always work? According to the docs, """Whether the name can be
used to open the file a second time, while the named temporary file is
still open, varies across platforms (it can be so used on Unix; it
cannot on Windows NT or later)""".

tempfile.mktemp() is portable.

Regards

Antoine.



From stephane at wirtel.be  Tue Mar 19 09:51:51 2019
From: stephane at wirtel.be (=?UTF-8?Q?St=c3=a9phane_Wirtel?=)
Date: Tue, 19 Mar 2019 14:51:51 +0100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <q6qprl$2ebs$1@blaine.gmane.org>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6q8af$3i67$1@blaine.gmane.org>
 <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
 <q6qprl$2ebs$1@blaine.gmane.org>
Message-ID: <0df5a99d-7aeb-183f-b5f6-87be1cc8f208@wirtel.be>

and why not with a very long PendingDeprecationWarning? this warning
could be used in this case.

From vstinner at redhat.com  Tue Mar 19 09:53:25 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 19 Mar 2019 14:53:25 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
Message-ID: <CA+3bQGEJ2pYn1Hmnwct7B0GYLd-8HLcPwyTbmFLHxnH4xKG32w@mail.gmail.com>

Hi,

I would prefer to keep tempfile.mktemp(), remove the deprecation, but
better explain the risk of race condition affecting security.

Le mar. 19 mars 2019 ? 14:41, Chris Angelico <rosuav at gmail.com> a ?crit :
> Can't you create a NamedTemporaryFile and permit the other program to
> use it? I just tried that (with TiMidity, even though it's quite
> capable of just writing to stdout) and it worked fine.

When I write tests, I don't really care of security, but
NamedTemporaryFile caused me many troubles on Windows: you cannot
delete a file if it's still open in a another program. It's way more
convenient to use tempfile.mktemp().

O_EXCL, open(tmpname, "wx"), os.open(tmpname, os.O_CREAT | os.O_EXCL |
os.O_WRONLY), etc. can be used to get an error if the file already
exists.

I agree that for production code where security matters,
tempfile.mktemp() must be avoided. But I would prefer to keep it for
tests.

"with NamedTemporaryFile() as tmp: name = tmp.name" isn't a great
replacement for tempfile.mktemp(): it creates the file and it opens
it, whereas I only want a file name and be the first file to create
and open it.

Victor

From stephane at wirtel.be  Tue Mar 19 10:10:40 2019
From: stephane at wirtel.be (=?UTF-8?Q?St=c3=a9phane_Wirtel?=)
Date: Tue, 19 Mar 2019 15:10:40 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <20190319143946.3df5a605@fsol>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <q6qr1d$5497$1@blaine.gmane.org> <20190319143946.3df5a605@fsol>
Message-ID: <15894f39-1298-4e78-ee59-5d28ca123b5e@wirtel.be>

totally agree with you but this function is deprecated (2002) since 2.3,
with a simle comment about a security issue.

2.3 -> 2.7, 3.0 -> 3.7, 13 releases and 17 years.

Maybe we could remove it with an official PendingDeprecationWarning.

Le 19/03/19 ? 14:39, Antoine Pitrou a ?crit?:
> The fact that many projects, including well-maintained ones such Sphinx
> or pip, use mktemp(), may be a hint that replacing it is not as easy as
> the people writing the Python documentation seem to think.
What's the relation with the people writing the Python documentation?

The suggestion about the deprecation warning was proposed by Brett
Cannon, and Serhiy has proposed to deprecate this function with some
releases.

The final release for 3.8 is scheduled for October 2019
(PendingDeprecationWarning).
Maybe 3.9 will be released 18 months later (DeprecationWarning).
and maybe 3.10 or 4.0 will be released 18 months after 3.9.

so, from today to 3.9+ there are approximatively 43 months -> 3,5 years.
I think it's enough in term of time for the big projects to improve
their code.


St?phane

From srittau at rittau.biz  Tue Mar 19 10:12:06 2019
From: srittau at rittau.biz (Sebastian Rittau)
Date: Tue, 19 Mar 2019 15:12:06 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CA+3bQGEJ2pYn1Hmnwct7B0GYLd-8HLcPwyTbmFLHxnH4xKG32w@mail.gmail.com>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
 <CA+3bQGEJ2pYn1Hmnwct7B0GYLd-8HLcPwyTbmFLHxnH4xKG32w@mail.gmail.com>
Message-ID: <8d4e50d4-5167-a419-c254-a1f0631d2755@rittau.biz>

Am 19.03.19 um 14:53 schrieb Victor Stinner:
>
> When I write tests, I don't really care of security, but
> NamedTemporaryFile caused me many troubles on Windows: you cannot
> delete a file if it's still open in a another program. It's way more
> convenient to use tempfile.mktemp().
>
> O_EXCL, open(tmpname, "wx"), os.open(tmpname, os.O_CREAT | os.O_EXCL |
> os.O_WRONLY), etc. can be used to get an error if the file already
> exists.
>
> I agree that for production code where security matters,
> tempfile.mktemp() must be avoided. But I would prefer to keep it for
> tests.

If there are valid use cases for mktemp(), I recommend renaming
it to mkname_unsafe() or something equally obvious. Experience
(and the list of packages still using mktemp() posted here) shows
that just adding a warning to documentation is not enough. Users
often discover functions by experimentation or looking at examples
on the internet.

mktemp() is also unfortunately named, as it does not create a temp
file as implied. This can also add to the impression that it is the
proper function to use.

Adding a new function and following the deprecation process for the
old one should only be a minor inconvenience for existing users that
need it, should wake up existing users that should not use it in the
first place, and still allows it to be used for relevant use cases.

I believe for security reasons sometimes inconvenient changes like
this are necessary.

 ?- Sebastian


From paul at ganssle.io  Tue Mar 19 10:21:35 2019
From: paul at ganssle.io (Paul Ganssle)
Date: Tue, 19 Mar 2019 10:21:35 -0400
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <20190319143946.3df5a605@fsol>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <q6qr1d$5497$1@blaine.gmane.org> <20190319143946.3df5a605@fsol>
Message-ID: <0b519940-06cd-32ee-538e-1c6effdd102a@ganssle.io>

I'm not sure the relationship with mkdir and mktemp here. I don't see
any uses of tempfile.mktemp in pip or setuptools, though they do use
os.mkdir (which is not deprecated).

Both pip and setuptools use pytest's tmpdir_factory.mktemp() in their
test suites, but I believe that is not the same thing.

On 3/19/19 9:39 AM, Antoine Pitrou wrote:
> On Tue, 19 Mar 2019 15:32:25 +0200
> Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 19.03.19 15:03, St?phane Wirtel ????:
>>> Suggestion and timeline:
>>>
>>> 3.8, we raise a PendingDeprecationWarning
>>>      * update the code
>>>      * update the documentation
>>>      * update the tests
>>>        (check a PendingDeprecationWarning if sys.version_info == 3.8)
>>>
>>> 3.9, we change PendingDeprecationWarning to DeprecationWarning
>>>        (check DeprecationWarning if sys.version_info == 3.9)
>>>
>>> 3.9+, we drop tempfile.mktemp()  
>> This plan LGTM.
>>
>> Currently mkdir() is widely used in distutils, Sphinx, pip, setuptools, 
>> virtualenv, and many other third-party projects, so it will take time to 
>> fix all these places. But we should do this, because all this code 
>> likely contains security flaws.
> The fact that many projects, including well-maintained ones such Sphinx
> or pip, use mktemp(), may be a hint that replacing it is not as easy as
> the people writing the Python documentation seem to think.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/f1aa8ebd/attachment.sig>

From solipsis at pitrou.net  Tue Mar 19 10:29:17 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 19 Mar 2019 15:29:17 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <q6qr1d$5497$1@blaine.gmane.org> <20190319143946.3df5a605@fsol>
 <15894f39-1298-4e78-ee59-5d28ca123b5e@wirtel.be>
Message-ID: <20190319152917.4d17d7af@fsol>

On Tue, 19 Mar 2019 15:10:40 +0100
St?phane Wirtel <stephane at wirtel.be> wrote:
> totally agree with you but this function is deprecated (2002) since 2.3,
> with a simle comment about a security issue.

"Deprecated" doesn't mean anything here.  It's just a mention in the
documentation.  It doesn't produce actual warnings when used.  And for
good reason: there are valid use cases.

> so, from today to 3.9+ there are approximatively 43 months -> 3,5 years.
> I think it's enough in term of time for the big projects to improve
> their code.

Please explain how the "improvement" would look like.  What is the
intended replacement for the use case I have explained, and how does it
improve on the statu quo?

And if there is an easy replacement, then how about re-implementing
mktemp() using that replacement, instead of removing it?

Regards

Antoine.



From guido at python.org  Tue Mar 19 10:52:12 2019
From: guido at python.org (Guido van Rossum)
Date: Tue, 19 Mar 2019 07:52:12 -0700
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <20190319142339.76cfdc47@fsol>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
Message-ID: <CAP7+vJKvJDs6EhZBKr0BBTWMkepYMmxTj-2rzrF_o9N2Gd1eSw@mail.gmail.com>

On Tue, Mar 19, 2019 at 6:27 AM Antoine Pitrou <solipsis at pitrou.net> wrote:

>
> -1.  Please don't remove tempfile.mktemp().  mktemp() is useful to
> create a temporary *name*.  All other tempfile functions create an
> actual file and impose additional burden, for example by making the
> file unaccessible by other processes.  But sometimes all I want is a
> temporary name that an *other* program will create / act on, not Python.
> It's a very common use case when writing scripts.
>
> The only reasonable workaround I can think of is to first create a
> temporary directory using mkdtemp(), then use a well-known name inside
> that directory.  But that has the same security implications AFAICT,
> since another process can come and create the file / symlink first.
>

If all you need is a random name, why not just use a random number
generator?
E.g. I see code like this:

    binascii.hexlify(os.urandom(8)).decode('ascii')

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/244ce09f/attachment.html>

From antoine at python.org  Tue Mar 19 10:57:15 2019
From: antoine at python.org (Antoine Pitrou)
Date: Tue, 19 Mar 2019 15:57:15 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CAP7+vJKvJDs6EhZBKr0BBTWMkepYMmxTj-2rzrF_o9N2Gd1eSw@mail.gmail.com>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAP7+vJKvJDs6EhZBKr0BBTWMkepYMmxTj-2rzrF_o9N2Gd1eSw@mail.gmail.com>
Message-ID: <363c5b82-dcd7-d8f5-b790-65eb9e950ca1@python.org>


Le 19/03/2019 ? 15:52, Guido van Rossum a ?crit?:
> 
> If all you need is a random name, why not just use a random number
> generator?
> E.g. I see code like this:
> 
> ??? binascii.hexlify(os.urandom(8)).decode('ascii')

mktemp() already does this, probably in a better way, including the
notion of prefix, suffix, and parent directory.  Why should I have to
reimplement it myself?

Regards

Antoine.

From solipsis at pitrou.net  Tue Mar 19 11:16:21 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 19 Mar 2019 16:16:21 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
 <CA+3bQGEJ2pYn1Hmnwct7B0GYLd-8HLcPwyTbmFLHxnH4xKG32w@mail.gmail.com>
 <8d4e50d4-5167-a419-c254-a1f0631d2755@rittau.biz>
Message-ID: <20190319161621.1bb75551@fsol>

On Tue, 19 Mar 2019 15:12:06 +0100
Sebastian Rittau <srittau at rittau.biz> wrote:
> Am 19.03.19 um 14:53 schrieb Victor Stinner:
> >
> > When I write tests, I don't really care of security, but
> > NamedTemporaryFile caused me many troubles on Windows: you cannot
> > delete a file if it's still open in a another program. It's way more
> > convenient to use tempfile.mktemp().
> >
> > O_EXCL, open(tmpname, "wx"), os.open(tmpname, os.O_CREAT | os.O_EXCL |
> > os.O_WRONLY), etc. can be used to get an error if the file already
> > exists.
> >
> > I agree that for production code where security matters,
> > tempfile.mktemp() must be avoided. But I would prefer to keep it for
> > tests.  
> 
> If there are valid use cases for mktemp(), I recommend renaming
> it to mkname_unsafe() or something equally obvious.
> [...]
> Adding a new function and following the deprecation process for the
> old one should only be a minor inconvenience for existing users that
> need it, should wake up existing users that should not use it in the
> first place, and still allows it to be used for relevant use cases.

That would be fine with me.

Regards

Antoine.



From ajm at flonidan.dk  Tue Mar 19 11:48:32 2019
From: ajm at flonidan.dk (Anders Munch)
Date: Tue, 19 Mar 2019 15:48:32 +0000
Subject: [Python-Dev] Remove tempfile.mktemp()
Message-ID: <HE1PR05MB321116F3C3C61D7BF97E0FCFB4400@HE1PR05MB3211.eurprd05.prod.outlook.com>

Antoine Pitrou:
> And if there is an easy replacement, then how about re-implementing
> mktemp() using that replacement, instead of removing it?

Indeed.  The principal security issue with mktemp is the difficulty in creating a user-specific thing under a shared /tmp folder in a multi-user setup.

But if it hurts when you use /tmp, why use /tmp? Use a path with no world-accessible ancestor, or at least no world-writable ancestor.

On Windows, that means creating it somewhere under the CSIDL_LOCAL_APPDATA folder. Which is already the default for %TEMP% and %TMP%.
On Unix, it's a $HOME subfolder with access 700 or 600.
How about switching mktemp over to use that?

regards, Anders


From g.rodola at gmail.com  Tue Mar 19 12:23:00 2019
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 19 Mar 2019 17:23:00 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
Message-ID: <CAFYqXL_wWKrJrc7JRQP2yUTRyrJ8aKAUmUiUaEBGkabG-oWD3g@mail.gmail.com>

On Tue, Mar 19, 2019 at 2:06 PM St?phane Wirtel <stephane at wirtel.be> wrote:
>
> Hi,
>
> Context: raise a warning or remove tempfile.mktemp()
> BPO: https://bugs.python.org/issue36309
>
> Since 2.3, this function is deprecated in the documentation, just in the
> documentation. In the code, there is a commented RuntimeWarning.
> Commented by Guido in 2002, because the warning was too annoying (and I
> understand ;-)).
>
> So, in this BPO, we start to discuss about the future of this function
> and Serhiy proposed to discuss on the Python-dev mailing list.
>
> Question: Should we drop it or add a (Pending)DeprecationWarning?
>
> Suggestion and timeline:
>
> 3.8, we raise a PendingDeprecationWarning
>     * update the code
>     * update the documentation
>     * update the tests
>       (check a PendingDeprecationWarning if sys.version_info == 3.8)
>
> 3.9, we change PendingDeprecationWarning to DeprecationWarning
>       (check DeprecationWarning if sys.version_info == 3.9)
>
> 3.9+, we drop tempfile.mktemp()
>
> What do you suggest?

I concur with others who think this should not be removed. It's used
in different stdlib and third party modules' test suites. I see
tempfile.mktemp() very similar to  test.support.find_unused_port() and
os.path.exists/isfile/isdir functions: they are all subject to race
conditions but still they are widely used and have reason to exist
(practicality beats purity). I suggest turning the doc deprecation
into a note:: or warning::, so that extra visibility is maintained.
Because the use case is legitimate and many fs-related APIs such as
this one are inevitably racy, I lean more towards a note:: rather than
a warning:: though, and we could probably do the same for os.path.is*
functions.

@Sebastian
> If there are valid use cases for mktemp(), I recommend renaming
> it to mkname_unsafe() or something equally obvious.

I'm -1 about adding an alias (there should be one and preferably only
one way to do it). Also mkstemp() and mkdtemp() are somewhat poorly
named IMO, but I wouldn't add an alias for them either.

-- 
Giampaolo - http://grodola.blogspot.com

From g.rodola at gmail.com  Tue Mar 19 12:41:28 2019
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 19 Mar 2019 17:41:28 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <363c5b82-dcd7-d8f5-b790-65eb9e950ca1@python.org>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAP7+vJKvJDs6EhZBKr0BBTWMkepYMmxTj-2rzrF_o9N2Gd1eSw@mail.gmail.com>
 <363c5b82-dcd7-d8f5-b790-65eb9e950ca1@python.org>
Message-ID: <CAFYqXL9wzPohqD_E+rqFnXcHowR9W5XrpmuWiR85T=+HFgvCCQ@mail.gmail.com>

On Tue, Mar 19, 2019 at 3:57 PM Antoine Pitrou <antoine at python.org> wrote:
>
>
> Le 19/03/2019 ? 15:52, Guido van Rossum a ?crit :
> >
> > If all you need is a random name, why not just use a random number
> > generator?
> > E.g. I see code like this:
> >
> >     binascii.hexlify(os.urandom(8)).decode('ascii')
>
> mktemp() already does this, probably in a better way, including the
> notion of prefix, suffix, and parent directory.  Why should I have to
> reimplement it myself?

On top of that mktemp() tries exists() in a loop, diminishing the risk
of names collision.

-- 
Giampaolo - http://grodola.blogspot.com

From srittau at rittau.biz  Tue Mar 19 12:44:19 2019
From: srittau at rittau.biz (Sebastian Rittau)
Date: Tue, 19 Mar 2019 17:44:19 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CAFYqXL_wWKrJrc7JRQP2yUTRyrJ8aKAUmUiUaEBGkabG-oWD3g@mail.gmail.com>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <CAFYqXL_wWKrJrc7JRQP2yUTRyrJ8aKAUmUiUaEBGkabG-oWD3g@mail.gmail.com>
Message-ID: <fcfbe3d1-1dda-5eb5-a17a-2ee13194a1c3@rittau.biz>

Am 19.03.19 um 17:23 schrieb Giampaolo Rodola':
> @Sebastian
>> If there are valid use cases for mktemp(), I recommend renaming
>> it to mkname_unsafe() or something equally obvious.
> I'm -1 about adding an alias (there should be one and preferably only
> one way to do it). Also mkstemp() and mkdtemp() are somewhat poorly
> named IMO, but I wouldn't add an alias for them either.
>
Just to clarify: I was not suggesting creating an alias, I was suggesting
renaming the function, but keeping the old name for a normal
deprecation cycle.

But I had another thought: If I understand correctly, the exploitability
of mktemp() relies on the fact that between determining whether the
file exists and creation an attacker can create the file themselves.
Couldn't this problem be solved by generating a filename of sufficient
length using the secrets module? This way the filename should be
"unguessable" and safe.

 ?- Sebastian


From p.f.moore at gmail.com  Tue Mar 19 13:03:38 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 19 Mar 2019 17:03:38 +0000
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <fcfbe3d1-1dda-5eb5-a17a-2ee13194a1c3@rittau.biz>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <CAFYqXL_wWKrJrc7JRQP2yUTRyrJ8aKAUmUiUaEBGkabG-oWD3g@mail.gmail.com>
 <fcfbe3d1-1dda-5eb5-a17a-2ee13194a1c3@rittau.biz>
Message-ID: <CACac1F8QeAT0Y5r+VaE-6CNn4p9RKN9OAzSKUjLYg_WyNDJk8g@mail.gmail.com>

On Tue, 19 Mar 2019 at 16:47, Sebastian Rittau <srittau at rittau.biz> wrote:
> But I had another thought: If I understand correctly, the exploitability
> of mktemp() relies on the fact that between determining whether the
> file exists and creation an attacker can create the file themselves.
> Couldn't this problem be solved by generating a filename of sufficient
> length using the secrets module? This way the filename should be
> "unguessable" and safe.

IMO, there's not much point trying to "fix" mktemp(). The issues with
it are clear and there are far better alternatives already available
for people who need them. The question here is simply about removing
the function "because people might mistakenly use it and create
security risks".

Personally, I don't think we should break the code of people who are
using mktemp() correctly, in awareness of its limitations, just out of
some idea of protecting people from themselves. Certainly we should
provide safe library functions wherever possible, but we should have
better reasons for removing functions that have been around for many,
many years than just "people might be using it wrongly".

Paul

From g.rodola at gmail.com  Tue Mar 19 13:11:02 2019
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 19 Mar 2019 18:11:02 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <fcfbe3d1-1dda-5eb5-a17a-2ee13194a1c3@rittau.biz>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <CAFYqXL_wWKrJrc7JRQP2yUTRyrJ8aKAUmUiUaEBGkabG-oWD3g@mail.gmail.com>
 <fcfbe3d1-1dda-5eb5-a17a-2ee13194a1c3@rittau.biz>
Message-ID: <CAFYqXL9C8X465iT-dDJSb4rhfGxXXwA1Q5=cVjs=tQ0jgf5F_g@mail.gmail.com>

On Tue, 19 Mar 2019 at 17:47, Sebastian Rittau <srittau at rittau.biz> wrote:

> Am 19.03.19 um 17:23 schrieb Giampaolo Rodola':
> > @Sebastian
> >> If there are valid use cases for mktemp(), I recommend renaming
> >> it to mkname_unsafe() or something equally obvious.
> > I'm -1 about adding an alias (there should be one and preferably only
> > one way to do it). Also mkstemp() and mkdtemp() are somewhat poorly
> > named IMO, but I wouldn't add an alias for them either.
> >
> Just to clarify: I was not suggesting creating an alias, I was suggesting
> renaming the function, but keeping the old name for a normal
> deprecation cycle.
>
> But I had another thought: If I understand correctly, the exploitability
> of mktemp() relies on the fact that between determining whether the
> file exists and creation an attacker can create the file themselves.
> Couldn't this problem be solved by generating a filename of sufficient
> length using the secrets module? This way the filename should be
> "unguessable" and safe.


Technically you cannot make it 100% safe, only less likely to occur. And on
a second thought (I retract :)) since this could be used in real apps other
than tests (I was too focused on that) I think this should be a doc warning
after all, not info. Doc may suggest to use mode=x when creating the file,
in order to remove the security implications.

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/a8c7cd8a/attachment.html>

From guido at python.org  Tue Mar 19 13:21:11 2019
From: guido at python.org (Guido van Rossum)
Date: Tue, 19 Mar 2019 10:21:11 -0700
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CAFYqXL9C8X465iT-dDJSb4rhfGxXXwA1Q5=cVjs=tQ0jgf5F_g@mail.gmail.com>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <CAFYqXL_wWKrJrc7JRQP2yUTRyrJ8aKAUmUiUaEBGkabG-oWD3g@mail.gmail.com>
 <fcfbe3d1-1dda-5eb5-a17a-2ee13194a1c3@rittau.biz>
 <CAFYqXL9C8X465iT-dDJSb4rhfGxXXwA1Q5=cVjs=tQ0jgf5F_g@mail.gmail.com>
Message-ID: <CAP7+vJ+BKNdnLMKSnO9GfB5oeYfD21fR1iHaWpu1o930OUD-DA@mail.gmail.com>

On Tue, Mar 19, 2019 at 10:14 AM Giampaolo Rodola' <g.rodola at gmail.com>
wrote:

>
> On Tue, 19 Mar 2019 at 17:47, Sebastian Rittau <srittau at rittau.biz> wrote:
>
>> Am 19.03.19 um 17:23 schrieb Giampaolo Rodola':
>> > @Sebastian
>> >> If there are valid use cases for mktemp(), I recommend renaming
>> >> it to mkname_unsafe() or something equally obvious.
>> > I'm -1 about adding an alias (there should be one and preferably only
>> > one way to do it). Also mkstemp() and mkdtemp() are somewhat poorly
>> > named IMO, but I wouldn't add an alias for them either.
>> >
>> Just to clarify: I was not suggesting creating an alias, I was suggesting
>> renaming the function, but keeping the old name for a normal
>> deprecation cycle.
>>
>> But I had another thought: If I understand correctly, the exploitability
>> of mktemp() relies on the fact that between determining whether the
>> file exists and creation an attacker can create the file themselves.
>> Couldn't this problem be solved by generating a filename of sufficient
>> length using the secrets module? This way the filename should be
>> "unguessable" and safe.
>
>
> Technically you cannot make it 100% safe, only less likely to occur.
>

That seems unnecessarily pedantic. A cryptographic random generator, like
the one in the secrets module, *is* safe enough for all practical purposes
(with a large margin).


> And on a second thought (I retract :)) since this could be used in real
> apps other than tests (I was too focused on that) I think this should be a
> doc warning after all, not info. Doc may suggest to use mode=x when
> creating the file, in order to remove the security implications.
>

Hm, the random sequence (implemented in tempfile._RandomNameSequence) is
currently derived from the random module, which is not cryptographically
secure. Maybe all we need to do is replace its source of randomness with
one derived from the secrets module. That seems a one-line change.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/b26fc74e/attachment.html>

From sebastian at realpath.org  Tue Mar 19 13:16:29 2019
From: sebastian at realpath.org (Sebastian Krause)
Date: Tue, 19 Mar 2019 18:16:29 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CAP7+vJKvJDs6EhZBKr0BBTWMkepYMmxTj-2rzrF_o9N2Gd1eSw@mail.gmail.com>
 (Guido van Rossum's message of "Tue, 19 Mar 2019 07:52:12 -0700")
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAP7+vJKvJDs6EhZBKr0BBTWMkepYMmxTj-2rzrF_o9N2Gd1eSw@mail.gmail.com>
Message-ID: <86bm26wzrm.fsf@realpath.org>

Guido van Rossum <guido at python.org> wrote:
> If all you need is a random name, why not just use a random number
> generator?
> E.g. I see code like this:
>
>     binascii.hexlify(os.urandom(8)).decode('ascii')

I tend to use os.path.join(dir, str(uuid.uuid4())) if I need to
create a random filename to pass to another program. However, it
would be nice to have a standard helper function that also allows me
to specify a prefix and suffix. Shouldn't it be possible to just
modify tempfile.mktemp() to generate much longer random filenames so
that it is virtually impossible that another program has already
created a file with the same name? Then the security problem is
gone, there is no need to continue deprecating this function and all
programs using it should continue to work.

From vstinner at redhat.com  Tue Mar 19 13:37:42 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 19 Mar 2019 18:37:42 +0100
Subject: [Python-Dev] What is the workflow for announcing tier 2
 platform support?
In-Reply-To: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EF06@ALA-MBD.corp.ad.wrs.com>
References: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EF06@ALA-MBD.corp.ad.wrs.com>
Message-ID: <CA+3bQGGZegoCOGRzqu6fQSg36Bba7a5XJJ+ohBwr10AY+77qAQ@mail.gmail.com>

Hi,

I don't think that we can say that Python supports VxWorks yet. Many PRs
are not merged yet. You are free to disrribute a modified version with your
changes.

The PEP 11 lists conditions to fully support a platform:
https://www.python.org/dev/peps/pep-0011/

My summary is: all tests must pass (it's ok to skip a few), a buildbot must
run and a core dev should handle any regression.

I expect that VxWorks will better fit in the "Best Effort" category: only
fix issues when someone proposes a fix.

See also my notes:
https://pythondev.readthedocs.io/platforms.html

Victor


Le mardi 19 mars 2019, Xin, Peixing <Peixing.Xin at windriver.com> a ?crit :
> Hi, Experts:
>
>
>
> VxWorks RTOS will release a new version at the end of June. For this new
version, we(Wind River) plan to announce the Python support officially here
(https://www.python.org/download/other/) for VxWorks as tier 2 platform.
Now my 2 questions come:
>
> 1.       To be qualified as tier 2 platform here
https://www.python.org/download/other/, do I need to get all test cases
pass on VxWorks RTOS? Now we have the following PRs for VxWorks. With these
PRs merged, we can get around 70% test cases PASS.
>
> </mail/u/0/s/?view=att&th=16993e40c3ae8b1e&attid=0.0.1&disp=emb&zw&atsh=1>
>
> 2.       If we reach the announcement level, what is the workflow to
update the website statements here https://www.python.org/download/other/.
The following picture shows what we want to show.
>
>
>
> </mail/u/0/s/?view=att&th=16993e40c3ae8b1e&attid=0.0.2&disp=emb&zw&atsh=1>
>
>
>
> Thanks,
>
> Peixing
>
>

-- 
Night gathers, and now my watch begins. It shall not end until my death.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/d0e48a72/attachment.html>

From vstinner at redhat.com  Tue Mar 19 13:39:55 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Tue, 19 Mar 2019 18:39:55 +0100
Subject: [Python-Dev] Can I submit more support of standard library for
 VxWorks after 3.8.0 beta1?
In-Reply-To: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EE94@ALA-MBD.corp.ad.wrs.com>
References: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EE94@ALA-MBD.corp.ad.wrs.com>
Message-ID: <CA+3bQGH_+Duh=eett7m1Tr0mtp-dwQRu+dT8sDzXzaBVeakhaQ@mail.gmail.com>

You have to find someone to review your PRs. It takes time. We are all
volunteers. I looked and merged some VxWorks PRs.

Would it be possible to pay a core dev to do the reviews? That's an open
question for core devs and for WindRiver.

Victor

Le mardi 19 mars 2019, Xin, Peixing <Peixing.Xin at windriver.com> a ?crit :
> Hi, Experts:
>
>
>
> Seeing from the Python 3.8.0 schedule(
https://www.python.org/dev/peps/pep-0569/#schedule), new features will not
be allowed to submit after 3.8.0 beta1. For VxWorks RTOS platform
supporting CPython, we are using bpo-31904(
https://bugs.python.org/issue31904) for PRs to submit our codes. Now I want
to know whether we can add more standard library support for VxWorks AFTER
3.8.0 beta1?
>
>
>
> </mail/u/0/s/?view=att&th=16993c4c52478e78&attid=0.0.1&disp=emb&zw&atsh=1>
>
>
>
> Thanks,
>
> Peixing
>
>

-- 
Night gathers, and now my watch begins. It shall not end until my death.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/bb6fb7f8/attachment-0001.html>

From brett at python.org  Tue Mar 19 14:30:07 2019
From: brett at python.org (Brett Cannon)
Date: Tue, 19 Mar 2019 11:30:07 -0700
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <q6qprl$2ebs$1@blaine.gmane.org>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6q8af$3i67$1@blaine.gmane.org>
 <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
 <q6qprl$2ebs$1@blaine.gmane.org>
Message-ID: <CAP1=2W6GtXJ=TkEJ3nixt=FmmeAHrA0XjHvMpvx5bnTeee5+fg@mail.gmail.com>

On Tue, Mar 19, 2019 at 6:15 AM Serhiy Storchaka <storchaka at gmail.com>
wrote:

> 19.03.19 13:53, Ned Batchelder ????:
> > Option 4 is misleading.  Is anyone here really offering to "fix the
> > tests in third-party modules"?  Option 4 is actually, "do nothing, and
> > let a multitude of projects figure out how to fix their tests, slowing
> > progress in those projects as they try to support Python 3.8."
>
> Any option except option 1 (and option 2 with sorting by default)
> requires changing third-party code. You should either pass additional
> argument to serialization functions, or use special canonization functions.
>
> We should look at the problem from long perspective. Freezing the
> current behavior forever does not look good. If we need to break the
> user code, we should minimize the harm and provide convenient tools for
> reproducing the current behavior. And this is an opportunity to rewrite
> user tests in more appropriate form. In your case textual comparison may
> be the most appropriate form, but this may be not so in other cases.
>

In situations like this I think it's best to bite the bullet sooner rather
than later while acknowledging that folks like Ned are in a bind when they
have support older versions and thus have long-term support costs, too, and
try to make the transition as painless as possible (my guess is Ned's need
to support older versions will drop off faster than us having to support
the xml libraries in the stdlib going forward, hence my viewpoint).


>
> > Now in Python 3.8, because Python doesn't want to add an
> > optional flag to continue doing what it has always done, I need to
> > re-engineer my tests.
>
> Please wait yet some time. I hope to add canonicalization before the
> first beta.
>

For me I think canonicalization/stable pretty-print is the best option,
especially if we can put the canonicalization code up on PyPI for
supporting older versions of Python. Otherwise a function that does
something like an XOR to help diagnose what differs between 2 XML documents
is also seems like a good option to me.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/79f682a8/attachment.html>

From stefan_ml at behnel.de  Tue Mar 19 14:35:24 2019
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 Mar 2019 19:35:24 +0100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CAPJVwB=Z0zA=U+BsO50aphqYUtzLO_ROEb1RUYPz-ttwLAHyuQ@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CAPJVwB=Z0zA=U+BsO50aphqYUtzLO_ROEb1RUYPz-ttwLAHyuQ@mail.gmail.com>
Message-ID: <q6rcpf$54gj$1@blaine.gmane.org>

Nathaniel Smith schrieb am 19.03.19 um 00:15:
> That seems potentially simpler to implement than canonical XML
> serialization

C14N is already implemented for ElementTree, just needs to be ported to
Py3.8 and merged.

https://bugs.python.org/issue13611

Stefan


From brett at python.org  Tue Mar 19 14:39:13 2019
From: brett at python.org (Brett Cannon)
Date: Tue, 19 Mar 2019 11:39:13 -0700
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CAP7+vJ+BKNdnLMKSnO9GfB5oeYfD21fR1iHaWpu1o930OUD-DA@mail.gmail.com>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <CAFYqXL_wWKrJrc7JRQP2yUTRyrJ8aKAUmUiUaEBGkabG-oWD3g@mail.gmail.com>
 <fcfbe3d1-1dda-5eb5-a17a-2ee13194a1c3@rittau.biz>
 <CAFYqXL9C8X465iT-dDJSb4rhfGxXXwA1Q5=cVjs=tQ0jgf5F_g@mail.gmail.com>
 <CAP7+vJ+BKNdnLMKSnO9GfB5oeYfD21fR1iHaWpu1o930OUD-DA@mail.gmail.com>
Message-ID: <CAP1=2W6iRN3GZaSFE2b+9ZP+NZ9+9fwgizVbvpN8oXW6EydriA@mail.gmail.com>

On Tue, Mar 19, 2019 at 10:22 AM Guido van Rossum <guido at python.org> wrote:

> On Tue, Mar 19, 2019 at 10:14 AM Giampaolo Rodola' <g.rodola at gmail.com>
> wrote:
>
>>
>> On Tue, 19 Mar 2019 at 17:47, Sebastian Rittau <srittau at rittau.biz>
>> wrote:
>>
>>> Am 19.03.19 um 17:23 schrieb Giampaolo Rodola':
>>> > @Sebastian
>>> >> If there are valid use cases for mktemp(), I recommend renaming
>>> >> it to mkname_unsafe() or something equally obvious.
>>> > I'm -1 about adding an alias (there should be one and preferably only
>>> > one way to do it). Also mkstemp() and mkdtemp() are somewhat poorly
>>> > named IMO, but I wouldn't add an alias for them either.
>>> >
>>> Just to clarify: I was not suggesting creating an alias, I was suggesting
>>> renaming the function, but keeping the old name for a normal
>>> deprecation cycle.
>>>
>>> But I had another thought: If I understand correctly, the exploitability
>>> of mktemp() relies on the fact that between determining whether the
>>> file exists and creation an attacker can create the file themselves.
>>> Couldn't this problem be solved by generating a filename of sufficient
>>> length using the secrets module? This way the filename should be
>>> "unguessable" and safe.
>>
>>
>> Technically you cannot make it 100% safe, only less likely to occur.
>>
>
> That seems unnecessarily pedantic. A cryptographic random generator, like
> the one in the secrets module, *is* safe enough for all practical purposes
> (with a large margin).
>
>
>> And on a second thought (I retract :)) since this could be used in real
>> apps other than tests (I was too focused on that) I think this should be a
>> doc warning after all, not info. Doc may suggest to use mode=x when
>> creating the file, in order to remove the security implications.
>>
>
> Hm, the random sequence (implemented in tempfile._RandomNameSequence) is
> currently derived from the random module, which is not cryptographically
> secure. Maybe all we need to do is replace its source of randomness with
> one derived from the secrets module. That seems a one-line change.
>

If the only security vulnerability is due to the ability to guess a path
that would make sense (but I honestly don't know since I'm not a security
expert).

If Guido's suggestion isn't enough then I think that long with a rename of
the function to make it obvious that there's potential issues and
deprecating the old name makes the most sense.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/51c6a45c/attachment.html>

From brett at python.org  Tue Mar 19 14:42:25 2019
From: brett at python.org (Brett Cannon)
Date: Tue, 19 Mar 2019 11:42:25 -0700
Subject: [Python-Dev] Can I submit more support of standard library for
 VxWorks after 3.8.0 beta1?
In-Reply-To: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EE94@ALA-MBD.corp.ad.wrs.com>
References: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EE94@ALA-MBD.corp.ad.wrs.com>
Message-ID: <CAP1=2W6mB2W2rnXNdu7ye6mzpeJCTthUa9MSPqam+oOgdPD4og@mail.gmail.com>

On Mon, Mar 18, 2019 at 7:22 PM Xin, Peixing <Peixing.Xin at windriver.com>
wrote:

> Hi, Experts:
>
>
>
> Seeing from the Python 3.8.0 schedule(
> https://www.python.org/dev/peps/pep-0569/#schedule), new features will
> not be allowed to submit after 3.8.0 beta1. For VxWorks RTOS platform
> supporting CPython, we are using bpo-31904(
> https://bugs.python.org/issue31904) for PRs to submit our codes. Now I
> want to know whether we can add more standard library support for VxWorks
> AFTER 3.8.0 beta1?
>

If the question is whether we as core devs will merge PRs that extend
support in the stdlib, that will be up to the release manager. Since it
technically isn't a feature enhancement but a "bugfix" for VxWorks then it
will probably come down to what changes are required.

-Brett


>
>
>
>
> Thanks,
>
> Peixing
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/587771e7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 50332 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/587771e7/attachment-0001.png>

From stefan_ml at behnel.de  Tue Mar 19 14:42:39 2019
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 Mar 2019 19:42:39 +0100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6q8af$3i67$1@blaine.gmane.org>
 <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
Message-ID: <q6rd72$6crv$1@blaine.gmane.org>

Ned Batchelder schrieb am 19.03.19 um 12:53:
> I need to re-engineer my tests.

? or sort the attributes before serialisation, or use C14N always, or
change your code to create the attributes in sorted-by-name order. The new
behaviour allows for a couple of ways to deal with the issue of backwards
compatibility.

Stefan


From raymond.hettinger at gmail.com  Tue Mar 19 14:55:51 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 19 Mar 2019 11:55:51 -0700
Subject: [Python-Dev] Best way to specify docstrings for member objects
Message-ID: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>

I'm working on ways to make improve help() by giving docstrings to member objects.

One way to do it is to wait until after the class definition and then make individual, direct assignments to __doc__ attributes.This way widely the separates docstrings from their initial __slots__ definition.   Working downstream from the class definition feels awkward and doesn't look pretty.  

There's another way I would like to propose?.  The __slots__ definition already works with any iterable including a dictionary (the dict values are ignored), so we could use the values for the  docstrings.   

This keeps all the relevant information in one place (much like we already do with property() objects).  This way already works, we just need a few lines in pydoc to check to see if a dict if present.  This way also looks pretty and doesn't feel awkward.

I've included worked out examples below.  What do you all think about the proposal?


Raymond


? https://bugs.python.org/issue36326


====== Desired help() output ======

>>> help(NormalDist)
Help on class NormalDist in module __main__:

class NormalDist(builtins.object)
 |  NormalDist(mu=0.0, sigma=1.0)
 |  
 |  Normal distribution of a random variable
 |  
 |  Methods defined here:
 |  
 |  __init__(self, mu=0.0, sigma=1.0)
 |      NormalDist where mu is the mean and sigma is the standard deviation.
 |  
 |  cdf(self, x)
 |      Cumulative distribution function.  P(X <= x)
 |  
 |  pdf(self, x)
 |      Probability density function.  P(x <= X < x+dx) / dx
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |  
 |  mu
 |      Arithmetic mean.
 |  
 |  sigma
 |      Standard deviation.
 |  
 |  variance
 |      Square of the standard deviation.



====== Example of assigning docstrings after the class definition ======

class NormalDist:
    'Normal distribution of a random variable'

    __slots__ = ('mu', 'sigma')

    def __init__(self, mu=0.0, sigma=1.0):
        'NormalDist where mu is the mean and sigma is the standard deviation.'
        self.mu = mu
        self.sigma = sigma

    @property
    def variance(self):
        'Square of the standard deviation.'
        return self.sigma ** 2.

    def pdf(self, x):
        'Probability density function.  P(x <= X < x+dx) / dx'
        variance = self.variance
        return exp((x - self.mu)**2.0 / (-2.0*variance)) / sqrt(tau * variance)

    def cdf(self, x):
        'Cumulative distribution function.  P(X <= x)'
        return 0.5 * (1.0 + erf((x - self.mu) / (self.sigma * sqrt(2.0))))

NormalDist.mu.__doc__ = 'Arithmetic mean'
NormalDist.sigma.__doc__ = 'Standard deviation'



====== Example of assigning docstrings with a dict =====

class NormalDist:
    'Normal distribution of a random variable'

    __slots__ = {'mu' : 'Arithmetic mean.', 'sigma': 'Standard deviation.'}

    def __init__(self, mu=0.0, sigma=1.0):
        'NormalDist where mu is the mean and sigma is the standard deviation.'
        self.mu = mu
        self.sigma = sigma

    @property
    def variance(self):
        'Square of the standard deviation.'
        return self.sigma ** 2.

    def pdf(self, x):
        'Probability density function.  P(x <= X < x+dx) / dx'
        variance = self.variance
        return exp((x - self.mu)**2.0 / (-2.0*variance)) / sqrt(tau * variance)

    def cdf(self, x):
        'Cumulative distribution function.  P(X <= x)'
        return 0.5 * (1.0 + erf((x - self.mu) / (self.sigma * sqrt(2.0))))


From arj.python at gmail.com  Tue Mar 19 15:02:24 2019
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Tue, 19 Mar 2019 23:02:24 +0400
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
Message-ID: <CADrxXX=_UaTxJy8VmWvoKEL8_12EbND-_3MJYHqKqeAycbLu6g@mail.gmail.com>

I have the impression that the line between variables and docs is a tidbit
too much blurred.

Yours,

Abdur-Rahmaan Janhangeer
Mauritius
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/a0090419/attachment.html>

From timothy.c.delaney at gmail.com  Tue Mar 19 16:27:13 2019
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 20 Mar 2019 07:27:13 +1100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <q6qqgr$3tpp$1@blaine.gmane.org>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CAEbHw4b3JkvO-OkAMMdCzGfUiJXqwte1XhVj-JhndSHQO6FStg@mail.gmail.com>
 <CAN8CLgkB=206yqkvBc7dGXNRR7YmyCERJT-XjbyjvCPgqeGdVg@mail.gmail.com>
 <q6qqgr$3tpp$1@blaine.gmane.org>
Message-ID: <CAN8CLgn3SyMFS_J5N=eJPe5TV9_7htdW5JLsFWTAgs_nACDzMQ@mail.gmail.com>

On Wed, 20 Mar 2019 at 00:29, Serhiy Storchaka <storchaka at gmail.com> wrote:

> 19.03.19 15:10, Tim Delaney ????:
> > Now Calibre is definitely in the wrong here - it should be able to
> > import regardless of the order of attributes. But the fact is that there
> > are a lot of tools out there that are semi-broken in a similar manner.
>
> Is not Calibre going to seat on Python 2 forever? This makes it
> non-relevant to the discussion about Python 3.8.
>

I was simply using Calibre as an example of a tool I'd encountered recently
that works correctly with input files with attributes in one order, but not
the other. That it happens to be using Python (of any vintage) is
irrelevant - could have been written in C, Go, Lua ... same problem that
XML libraries that arbitrarily sort (or otherwise manipulate the order of)
attributes can result in files that may not work with third-party tools.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190320/d74b31f9/attachment.html>

From python at mrabarnett.plus.com  Tue Mar 19 16:52:02 2019
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 19 Mar 2019 20:52:02 +0000
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
Message-ID: <3d4c1184-a8e2-9fe5-a8f5-31c91b7ef1e9@mrabarnett.plus.com>

On 2019-03-19 18:55, Raymond Hettinger wrote:
> I'm working on ways to make improve help() by giving docstrings to member objects.
> 
> One way to do it is to wait until after the class definition and then make individual, direct assignments to __doc__ attributes.This way widely the separates docstrings from their initial __slots__ definition.   Working downstream from the class definition feels awkward and doesn't look pretty.
> 
> There's another way I would like to propose?.  The __slots__ definition already works with any iterable including a dictionary (the dict values are ignored), so we could use the values for the  docstrings.
> 
> This keeps all the relevant information in one place (much like we already do with property() objects).  This way already works, we just need a few lines in pydoc to check to see if a dict if present.  This way also looks pretty and doesn't feel awkward.
> 
> I've included worked out examples below.  What do you all think about the proposal?
> 
[snip]

Thinking ahead, could there ever be anything else that you might want 
also to attach to member objects?

I suppose that if that's ever the case, the value could itself be 
expanded to be a dict, something like this:

__slots__ = {'mu' : {'__doc__': 'Arithmetic mean.'}, 'sigma': 
{'__doc__': 'Standard deviation.'}}

But that could be left to the future...

From raymond.hettinger at gmail.com  Tue Mar 19 17:00:41 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 19 Mar 2019 14:00:41 -0700
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <3d4c1184-a8e2-9fe5-a8f5-31c91b7ef1e9@mrabarnett.plus.com>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
 <3d4c1184-a8e2-9fe5-a8f5-31c91b7ef1e9@mrabarnett.plus.com>
Message-ID: <429BB6F0-9D18-412B-9EC7-CB8322FAE43E@gmail.com>



> On Mar 19, 2019, at 1:52 PM, MRAB <python at mrabarnett.plus.com> wrote:
> 
> Thinking ahead, could there ever be anything else that you might want also to attach to member objects?

Our experience with property object suggests that once docstrings are supported, there don't seem to be any other needs.   But then, you never can tell ;-)


Raymond


"Difficult to see. Always in motion is the future." -- Master Yoda



From greg at krypto.org  Tue Mar 19 17:36:53 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 19 Mar 2019 14:36:53 -0700
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6q8af$3i67$1@blaine.gmane.org>
 <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
Message-ID: <CAGE7PNJjWYDc6zPQcFtDtTqjxgVV6euK3Uq27Fq9j2RQ0v+D=Q@mail.gmail.com>

On Tue, Mar 19, 2019, 4:53 AM Ned Batchelder <ned at nedbatchelder.com> wrote:

> On 3/19/19 4:13 AM, Serhiy Storchaka wrote:
> > 19.03.19 00:41, Raymond Hettinger ????:
> >> 4) Fix the tests in the third-party modules to be more focused on
> >> their actual test objectives, the semantics of the generated XML
> >> rather than the exact serialization.  This option would seem like the
> >> right-thing-to-do but it isn't trivial because the entire premise of
> >> the existing test is invalid.  For every case, we'll actually have to
> >> think through what the test objective really is.
> >
>
> Option 4 is misleading.  Is anyone here really offering to "fix the
> tests in third-party modules"?  Option 4 is actually, "do nothing, and
> let a multitude of projects figure out how to fix their tests, slowing
> progress in those projects as they try to support Python 3.8."
>

We've done Option 4 for every past behavior change of any form on feature
.Next releases.  We do try to encourage projects to run their tests on the
3.Next betas so that they can be ready before 3.Next.0 lands, some of us
even do it ourselves when we're interested.  Many things won't get ready
ahead of time, but the actual .0 release forces the issue as their users
start demanding it on occasion offering patches.  We don't bock a release
on existing user code being ready for it.

In my case, the test code has a generic function to compare an actual
> directory of files to an expected directory of files, so it isn't quite
> as simple as "just use the right XML comparison."  And I support Python
> 2.7, 3.5, etc, so tests still need to work under those versions.  None
> of this is impossible, but please try not to preach to us maintainers
> that we are doing it wrong, that it will be easy to fix, etc.  Using
> language like "the entire premise of the test is invalid" seems
> needlessly condescending.
>

Agreed, that was poor wording.  Lets not let that type of wording escape
python-dev into docs about a behavior change.

Wording aside, a test relying on undefined behavior is testing for things
the code under test doesn't actually need to care about being true, even if
it has happened to work for years.  Such a test is overspecified.
Potentially without the authors previously consciously realizing that.
It'll need refactoring to loosen its requirements.  How to loosen it is
always an implementation detail based on the constraints imposed upon the
test.  Difficulty lies within range(0, "Port Mercurial to Python 3").  But
the end result is nice: The code is healthier as tests focus more on what
was actually important rather than what was quicker to write that got the
original job done many years ago.

One of the suggested solutions, a DOM comparison is not enough. I
> don't just want to know that my actual XML is different than my expected
> XML.  I want to know where and how it differs.
>
> Textual comparison may be the "wrong" way to check XML, but it gives me
> many tools for working with the test results.  It was simple and it
> worked.  Now in Python 3.8, because Python doesn't want to add an
> optional flag to continue doing what it has always done, I need to
> re-engineer my tests.
>
> --Ned.
>

I understand that from a code owners perspective having to do any work, no
matter what the reason, is counted as re-engineering.  But that doesn't
make it wrong.  If "what it has always done" was unspecified and arbitrary
despite happening to not change in the past rather than something easy to
continue the stability of such as sorted attributes, it is a burden to
maintain that **unspecifiable** behavior in the language or library going
forward.

(Note that I have no idea what order the xml code in question happened to
impose upon attributes; if it went from sorted to not a "fix" to provide
users is clear: provide a way to keep it sorted for those who need that.
If it relied on insertion order or hash table iteration order or the phase
of the moon when the release was cut, we are right to refuse to maintain
unspecifiable implementation side effect behavior)

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/396956a6/attachment.html>

From greg.ewing at canterbury.ac.nz  Tue Mar 19 18:25:53 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 20 Mar 2019 11:25:53 +1300
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <20190319145127.0824f196@fsol>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
 <20190319145127.0824f196@fsol>
Message-ID: <5C916C71.5070102@canterbury.ac.nz>

Antoine Pitrou wrote:
> Does it always work? According to the docs, """Whether the name can be
> used to open the file a second time, while the named temporary file is
> still open, varies across platforms

So use NamedTemporaryFile(delete = False) and close it before passing
it to the other program.

-- 
Greg

From g.rodola at gmail.com  Tue Mar 19 18:41:30 2019
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 19 Mar 2019 23:41:30 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CAP7+vJ+BKNdnLMKSnO9GfB5oeYfD21fR1iHaWpu1o930OUD-DA@mail.gmail.com>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <CAFYqXL_wWKrJrc7JRQP2yUTRyrJ8aKAUmUiUaEBGkabG-oWD3g@mail.gmail.com>
 <fcfbe3d1-1dda-5eb5-a17a-2ee13194a1c3@rittau.biz>
 <CAFYqXL9C8X465iT-dDJSb4rhfGxXXwA1Q5=cVjs=tQ0jgf5F_g@mail.gmail.com>
 <CAP7+vJ+BKNdnLMKSnO9GfB5oeYfD21fR1iHaWpu1o930OUD-DA@mail.gmail.com>
Message-ID: <CAFYqXL_Gj8WsMcs0TQ_LZt4u+rLM+S4UqD-pPUzYhHj30V8=5w@mail.gmail.com>

On Tue, Mar 19, 2019 at 6:21 PM Guido van Rossum <guido at python.org> wrote:

>> On Tue, Mar 19, 2019 at 10:14 AM Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>> Technically you cannot make it 100% safe, only less likely to occur.
> That seems unnecessarily pedantic. A cryptographic random generator, like the one in the secrets module, *is* safe enough for all practical purposes (with a large margin).
> [...]
> Hm, the random sequence (implemented in tempfile._RandomNameSequence) is currently derived from the random module, which is not cryptographically secure. Maybe all we need to do is replace its source of randomness with one derived from the secrets module. That seems a one-line change.

Seems to make sense to me. Currently the random string is 8 chars
long, meaning (if I'm not mistaken) max 40320 possible combinations:

    >>> len(list(itertools.permutations('wuoa736m'))))
    40320

We may use 9 chars and get to 362880 and increase "security".
Regardless, it seems to me the original deprecation warning for
mktemp() was not really justified and should be removed. IMO it
probably makes more sense to just clarify in the doc that
NamedTemporaryFile is the right / safe way to create a file with a
unique name and avoid the theoretical, rare name collision you'd get
by using open(mktemp(), 'w') instead. Also I'm not sure I understand
what the code sample provided in the doc aims to achieve:
https://docs.python.org/3/library/tempfile.html#tempfile.mktemp
The way I see it, the main (or "correct") use case for mktemp() is
when you explicitly want a file name which does not exist, in order to
exercise legitimate scenarios like these ones:

    >>> self.assertRaises(FileNotFoundError, os.unlink, tempfile.mktemp())

    >>> dst = tempfile.mktemp()
    >>> os.rename(src, dst)
    >>> assert not os.path.exists(src) and os.path.exists(dst)

-- 
Giampaolo - http://grodola.blogspot.com

From eryksun at gmail.com  Tue Mar 19 18:49:43 2019
From: eryksun at gmail.com (eryk sun)
Date: Tue, 19 Mar 2019 17:49:43 -0500
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CA+3bQGEJ2pYn1Hmnwct7B0GYLd-8HLcPwyTbmFLHxnH4xKG32w@mail.gmail.com>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
 <CA+3bQGEJ2pYn1Hmnwct7B0GYLd-8HLcPwyTbmFLHxnH4xKG32w@mail.gmail.com>
Message-ID: <CACL+1av3p+5KvcAcpxK6a-p-eRdyA9CGvRh4rgrkMYG2M90z=w@mail.gmail.com>

On 3/19/19, Victor Stinner <vstinner at redhat.com> wrote:
>
> When I write tests, I don't really care of security, but
> NamedTemporaryFile caused me many troubles on Windows: you cannot
> delete a file if it's still open in a another program. It's way more
> convenient to use tempfile.mktemp().

Opening the file again for normal access is problematic.
NamedTemporaryFile opens it with delete access, but Python's open()
function doesn't support delete-access sharing unless an opener is
used that calls CreateFileW.

NamedTemporaryFile does open files with delete-access sharing, so any
process can delete the file if it's allowed by the file's security and
attributes. You may be thinking of unlinking. In Windows versions
prior to 10, that's always a two-step process. A file with its delete
disposition set doesn't get unlinked until all references for all open
instances are closed.

In Windows 10 (release 1709+), we have the option of using
SetFileInformationByHandle: FileDispositionInfoEx (21) with
FILE_DISPOSITION_FLAG_POSIX_SEMANTICS (2) and
FILE_DISPOSITION_FLAG_DELETE (1). The online documentation hasn't been
updated to include this, but it's supported in the headers for
_WIN32_WINNT_WIN10_RS1 and later. This operation unlinks the file as
soon as we close our handle, even if it has existing references. This
is explained in the remarks for the underlying NT system call [1]. In
particular this resolves the race condition related to handles opened
by anti-malware programs.

It may be worth adding support for deleting files by handle that tries
FileDispositionInfoEx in 1709+. This will work in about half of all
Windows systems. (About 40% still run Windows 7.) It's not a panacea
for Windows file-deleting woes. We still need to be able to open the
file with delete access, which requires existing opens to share delete
access.

[1]: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntddk/ns-ntddk-_file_disposition_information_ex

From brian.kuhl at windriver.com  Tue Mar 19 16:58:05 2019
From: brian.kuhl at windriver.com (Kuhl, Brian)
Date: Tue, 19 Mar 2019 20:58:05 +0000
Subject: [Python-Dev] Can I submit more support of standard library for
 VxWorks after 3.8.0 beta1?
In-Reply-To: <CA+3bQGH_+Duh=eett7m1Tr0mtp-dwQRu+dT8sDzXzaBVeakhaQ@mail.gmail.com>
References: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EE94@ALA-MBD.corp.ad.wrs.com>
 <CA+3bQGH_+Duh=eett7m1Tr0mtp-dwQRu+dT8sDzXzaBVeakhaQ@mail.gmail.com>
Message-ID: <6C486AB80682E24095E3E77A408D68F20149A4AF01@ALA-MBD.corp.ad.wrs.com>

Hi Victor,
I pinged our product manager and he?s open to the idea of setting up a consulting  arrangement with a core developer to help move things along.
At least in principal.
If anyone on the core team is interested, and reasonably unaligned with Wind River?s competitors, please contact me off list, Brian dot Kuhl at Wind River dot com.

Brian

From: Python-Dev [mailto:python-dev-bounces+brian.kuhl=windriver.com at python.org] On Behalf Of Victor Stinner
Sent: Tuesday, March 19, 2019 1:40 PM
To: Xin, Peixing <Peixing.Xin at windriver.com>
Cc: python-dev at python.org
Subject: Re: [Python-Dev] Can I submit more support of standard library for VxWorks after 3.8.0 beta1?

You have to find someone to review your PRs. It takes time. We are all volunteers. I looked and merged some VxWorks PRs.

Would it be possible to pay a core dev to do the reviews? That's an open question for core devs and for WindRiver.

Victor

Le mardi 19 mars 2019, Xin, Peixing <Peixing.Xin at windriver.com<mailto:Peixing.Xin at windriver.com>> a ?crit :
> Hi, Experts:
>
>
>
> Seeing from the Python 3.8.0 schedule(https://www.python.org/dev/peps/pep-0569/#schedule), new features will not be allowed to submit after 3.8.0 beta1. For VxWorks RTOS platform supporting CPython, we are using bpo-31904(https://bugs.python.org/issue31904) for PRs to submit our codes. Now I want to know whether we can add more standard library support for VxWorks AFTER 3.8.0 beta1?
>
>
>
> </mail/u/0/s/?view=att&th=16993c4c52478e78&attid=0.0.1&disp=emb&zw&atsh=1>
>
>
>
> Thanks,
>
> Peixing
>
>

--
Night gathers, and now my watch begins. It shall not end until my death.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190319/5fd54eaa/attachment.html>

From njs at pobox.com  Tue Mar 19 19:33:37 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 19 Mar 2019 16:33:37 -0700
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CAFYqXL_Gj8WsMcs0TQ_LZt4u+rLM+S4UqD-pPUzYhHj30V8=5w@mail.gmail.com>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <CAFYqXL_wWKrJrc7JRQP2yUTRyrJ8aKAUmUiUaEBGkabG-oWD3g@mail.gmail.com>
 <fcfbe3d1-1dda-5eb5-a17a-2ee13194a1c3@rittau.biz>
 <CAFYqXL9C8X465iT-dDJSb4rhfGxXXwA1Q5=cVjs=tQ0jgf5F_g@mail.gmail.com>
 <CAP7+vJ+BKNdnLMKSnO9GfB5oeYfD21fR1iHaWpu1o930OUD-DA@mail.gmail.com>
 <CAFYqXL_Gj8WsMcs0TQ_LZt4u+rLM+S4UqD-pPUzYhHj30V8=5w@mail.gmail.com>
Message-ID: <CAPJVwBk=C08LhBBcHJZeqxwLv0SOME2FLhU6adzvpo-aBsMEJw@mail.gmail.com>

On Tue, Mar 19, 2019 at 3:43 PM Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>
> On Tue, Mar 19, 2019 at 6:21 PM Guido van Rossum <guido at python.org> wrote:
>
> >> On Tue, Mar 19, 2019 at 10:14 AM Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> >> Technically you cannot make it 100% safe, only less likely to occur.
> > That seems unnecessarily pedantic. A cryptographic random generator, like the one in the secrets module, *is* safe enough for all practical purposes (with a large margin).
> > [...]
> > Hm, the random sequence (implemented in tempfile._RandomNameSequence) is currently derived from the random module, which is not cryptographically secure. Maybe all we need to do is replace its source of randomness with one derived from the secrets module. That seems a one-line change.
>
> Seems to make sense to me. Currently the random string is 8 chars
> long, meaning (if I'm not mistaken) max 40320 possible combinations:
>
>     >>> len(list(itertools.permutations('wuoa736m'))))
>     40320

It's 8 chars taken from a-z0-9_, so there are 37**8 ~= 10**12 possible
names, or ~41 bits.

> We may use 9 chars and get to 362880 and increase "security".
> Regardless, it seems to me the original deprecation warning for
> mktemp() was not really justified and should be removed.

Historically, mktemp variants have caused *tons* of serious security
vulnerabilities. It's not a theoretical issue. See e.g.
https://www.owasp.org/index.php/Insecure_Temporary_File

I believe that if we used sufficient unguessable randomness to
generate the name, then that would resolve the issues and let us
un-deprecate it. Though like Brett I would like to double-check this
with security specialists :-).

This would also simplify the implementation a *lot*, down to just:

def mktemp(suffix='', prefix='tmp', dir=None):
    if dir is None:
        dir = gettempdir()
    return _os.path.join(dir, prefix +
secrets.token_urlsafe(ENTROPY_BYTES) + suffix)

The choice of ENTROPY_BYTES is an interesting question. 16 (= 128
bits) would be a nice "obviously safe" number, and gives 22-byte
filenames. We might be able to get away with fewer, if we had a
plausible cost model for the attack. This is another point where a
security specialist might be helpful :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From tjreedy at udel.edu  Wed Mar 20 00:45:56 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Mar 2019 00:45:56 -0400
Subject: [Python-Dev] Can I submit more support of standard library for
 VxWorks after 3.8.0 beta1?
In-Reply-To: <CA+3bQGH_+Duh=eett7m1Tr0mtp-dwQRu+dT8sDzXzaBVeakhaQ@mail.gmail.com>
References: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EE94@ALA-MBD.corp.ad.wrs.com>
 <CA+3bQGH_+Duh=eett7m1Tr0mtp-dwQRu+dT8sDzXzaBVeakhaQ@mail.gmail.com>
Message-ID: <q6sgi5$69f$1@blaine.gmane.org>

One main purpose of the beta period is to discover and fix bugs and 
otherwise tweek behavior in the new features included in the first beta. 
  This usually presupposes that the feature is thought to be 
'ready-to-go' as-is, absent new discoveries.

Interpretation and issuance of exceptions is usually up to the release 
manager.  For 3.8, this is ?ukasz Langa,  Victor Stinner is otherwise 
most involved with platform support.

FWIW, I personally might allow 'suppport added by beta 1' to include 
failures in up to 4 modules to be fixed as bugs as soon as possible.

On 3/19/2019 1:39 PM, Victor Stinner wrote:
> You have to find someone to review your PRs. It takes time. We are all 
> volunteers. I looked and merged some VxWorks PRs.
> 
> Would it be possible to pay a core dev to do the reviews? That's an open 
> question for core devs and for WindRiver.

I think that this is an excellent idea (as in 'likely needed to meet the 
deadline').  (And no, I am not a candidate for the contract.)

-- 
Terry Jan Reedy



From Peixing.Xin at windriver.com  Wed Mar 20 02:18:22 2019
From: Peixing.Xin at windriver.com (Xin, Peixing)
Date: Wed, 20 Mar 2019 06:18:22 +0000
Subject: [Python-Dev] Can I submit more support of standard library for
 VxWorks after 3.8.0 beta1?
In-Reply-To: <CAP1=2W6mB2W2rnXNdu7ye6mzpeJCTthUa9MSPqam+oOgdPD4og@mail.gmail.com>
References: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9EE94@ALA-MBD.corp.ad.wrs.com>
 <CAP1=2W6mB2W2rnXNdu7ye6mzpeJCTthUa9MSPqam+oOgdPD4og@mail.gmail.com>
Message-ID: <8488FBC4EAAC5941BA4B85DD1ECCF1870133B9FF16@ALA-MBD.corp.ad.wrs.com>

Hi, Brett:

Thanks for your comments. Yes, we want to submit more fixes on test_failures of stardard library after 3.8.0 beta1. Expect those can be accepted and get merged??

Thanks,
Peixing

From: Brett Cannon [mailto:brett at python.org]
Sent: Wednesday, March 20, 2019 2:42 AM
To: Xin, Peixing
Cc: python-dev at python.org
Subject: Re: [Python-Dev] Can I submit more support of standard library for VxWorks after 3.8.0 beta1?



On Mon, Mar 18, 2019 at 7:22 PM Xin, Peixing <Peixing.Xin at windriver.com<mailto:Peixing.Xin at windriver.com>> wrote:
Hi, Experts:

Seeing from the Python 3.8.0 schedule(https://www.python.org/dev/peps/pep-0569/#schedule), new features will not be allowed to submit after 3.8.0 beta1. For VxWorks RTOS platform supporting CPython, we are using bpo-31904(https://bugs.python.org/issue31904) for PRs to submit our codes. Now I want to know whether we can add more standard library support for VxWorks AFTER 3.8.0 beta1?

If the question is whether we as core devs will merge PRs that extend support in the stdlib, that will be up to the release manager. Since it technically isn't a feature enhancement but a "bugfix" for VxWorks then it will probably come down to what changes are required.

-Brett


[cid:image001.png at 01D4DF27.C517FFF0]

Thanks,
Peixing

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org<mailto:Python-Dev at python.org>
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190320/2fa986c4/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 50332 bytes
Desc: image001.png
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190320/2fa986c4/attachment-0001.png>

From storchaka at gmail.com  Wed Mar 20 03:41:34 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 20 Mar 2019 09:41:34 +0200
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
Message-ID: <q6sqrh$5318$1@blaine.gmane.org>

19.03.19 20:55, Raymond Hettinger ????:
> I'm working on ways to make improve help() by giving docstrings to member objects.
> 
> One way to do it is to wait until after the class definition and then make individual, direct assignments to __doc__ attributes.This way widely the separates docstrings from their initial __slots__ definition.   Working downstream from the class definition feels awkward and doesn't look pretty.
> 
> There's another way I would like to propose?.  The __slots__ definition already works with any iterable including a dictionary (the dict values are ignored), so we could use the values for the  docstrings.

I think it would be nice to separate docstrings from the bytecode. This 
would be allow to have several translated sets of docstrings and load an 
appropriate set depending on user preferences. This would help in 
teaching Python.

It is possible with docstrings of modules, classes, functions, methods 
and properties (created by using the decorator), because the compiler 
knows what string literal is a docstring. But this is impossible with 
namedtuple fields and any of the above ideas for slots.

It would be nice to allow to specify docstrings for slots as for methods 
and properties. Something like in the following pseudocode:

class NormalDist:
     slot mu:
	'''Arithmetic mean'''
     slot sigma:
         '''Standard deviation'''

It would be also nice to annotate slots and add default values (used 
when the slot value was not set).

class NormalDist:
     mu: float = 0.0
	'''Arithmetic mean'''
     sigma: float = 1.0
         '''Standard deviation'''


From storchaka at gmail.com  Wed Mar 20 04:04:56 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 20 Mar 2019 10:04:56 +0200
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <0b519940-06cd-32ee-538e-1c6effdd102a@ganssle.io>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <q6qr1d$5497$1@blaine.gmane.org> <20190319143946.3df5a605@fsol>
 <0b519940-06cd-32ee-538e-1c6effdd102a@ganssle.io>
Message-ID: <q6ss7c$hp$1@blaine.gmane.org>

19.03.19 16:21, Paul Ganssle ????:
> I'm not sure the relationship with mkdir and mktemp here. I don't see
> any uses of tempfile.mktemp in pip or setuptools, though they do use
> os.mkdir (which is not deprecated).
> 
> Both pip and setuptools use pytest's tmpdir_factory.mktemp() in their
> test suites, but I believe that is not the same thing.

My fault!

Initially I searched mktemp, and have found usages in distutils, tests, 
and few other modules in the stdlib. When I wrote the last message I 
repeat the search on the wider set of Python sources, but for *mkdir* 
instead of *mktemp*! Thank you for catching this mistake Paul.

Actually, seems mktemp is used exclusively in tests in third-party projects.


From storchaka at gmail.com  Wed Mar 20 04:06:27 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 20 Mar 2019 10:06:27 +0200
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <20190319143946.3df5a605@fsol>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <q6qr1d$5497$1@blaine.gmane.org> <20190319143946.3df5a605@fsol>
Message-ID: <q6ssa6$hp$2@blaine.gmane.org>

19.03.19 15:39, Antoine Pitrou ????:
> The fact that many projects, including well-maintained ones such Sphinx
> or pip, use mktemp(), may be a hint that replacing it is not as easy as
> the people writing the Python documentation seem to think.

Sorry, it was my mistake (searching mkdir instead of mktemp). mktemp is 
only used in few tests in third-party projects.


From ajm at flonidan.dk  Wed Mar 20 04:47:53 2019
From: ajm at flonidan.dk (Anders Munch)
Date: Wed, 20 Mar 2019 08:47:53 +0000
Subject: [Python-Dev] Remove tempfile.mktemp()
Message-ID: <HE1PR05MB3211F931DA855F9991E90693B4410@HE1PR05MB3211.eurprd05.prod.outlook.com>

Greg Ewing:
> So use NamedTemporaryFile(delete = False) and close it before passing it to the other program.

That's effectively the same as calling tempfile.mktemp.   While it does waste time opening and closing an unused file, that doesn't help with security.  If anything, it might worsen security.

If a secure implementation of mktemp is truly impossible, then the same could be said for NamedTemperatoryFile(delete=False). Should that be deprecated as well?

regards, Anders


From srittau at rittau.biz  Wed Mar 20 04:57:38 2019
From: srittau at rittau.biz (Sebastian Rittau)
Date: Wed, 20 Mar 2019 09:57:38 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <HE1PR05MB3211F931DA855F9991E90693B4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
References: <HE1PR05MB3211F931DA855F9991E90693B4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
Message-ID: <9bea28c8-3a77-710a-af65-14b051aa5a20@rittau.biz>


Am 20.03.19 um 09:47 schrieb Anders Munch:
> Greg Ewing:
>> So use NamedTemporaryFile(delete = False) and close it before passing it to the other program.
> That's effectively the same as calling tempfile.mktemp.   While it does waste time opening and closing an unused file, that doesn't help with security.  If anything, it might worsen security.

That is not actually true. The important difference is that with 
NamedTemporaryFile the file exists with appropriate access right (0600). 
This denies access of that file to other users. With mktemp() no file is 
created, so another user can "hijack" that name and cause programs to 
write potentially privileged data into or read manipulated data from 
that file.

 ?- Sebastian



From ajm at flonidan.dk  Wed Mar 20 05:53:13 2019
From: ajm at flonidan.dk (Anders Munch)
Date: Wed, 20 Mar 2019 09:53:13 +0000
Subject: [Python-Dev] Remove tempfile.mktemp()
Message-ID: <HE1PR05MB32116569E1371CC3DFD84960B4410@HE1PR05MB3211.eurprd05.prod.outlook.com>

Anders Munch:
>>> So use NamedTemporaryFile(delete = False) and close it before passing it to the other program.
>> That's effectively the same as calling tempfile.mktemp.   While it does waste time opening and closing an unused file, that doesn't help with security
Sebastian Rittau:
> That is not actually true. The important difference is that with NamedTemporaryFile the file exists with appropriate access right (0600).

You are right, I must have mentally reversed the polarity of the delete argument.  And I didn't realise that the access right on a file had the power to prevent itself from being removed from the folder that it's in.  I thought the access flags were a property of the file itself and not the directory entry. Not sure how that works.

But if NamedTemporaryFile(delete=False) is secure then why not use that to implement mktemp?

def mktemp(suffix="", prefix=template, dir=None):
    with NamedTemporaryFile(delete=False, suffix=suffix, prefix=prefix, dir=dir) as f:
        return f.name

Yes, it does leave an empty file if the name is not used, but the name is usually created with the intent to use it, so that is rarely going to be a problem. Just document that that's how it is.  It does mean that where there's an explicit file-exists check before writing the file, that code will break. But it will break a lot less code than removing mktemp entirely.

regards, Anders


From eryksun at gmail.com  Wed Mar 20 06:55:38 2019
From: eryksun at gmail.com (eryk sun)
Date: Wed, 20 Mar 2019 05:55:38 -0500
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <HE1PR05MB32116569E1371CC3DFD84960B4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
References: <HE1PR05MB32116569E1371CC3DFD84960B4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
Message-ID: <CACL+1as7+6do-RMmmHOBOs06RC9qJEDdr9N7tufJ-KKOxrK4rw@mail.gmail.com>

On 3/20/19, Anders Munch <ajm at flonidan.dk> wrote:
>
> You are right, I must have mentally reversed the polarity of the delete
> argument.  And I didn't realise that the access right on a file had the
> power to prevent itself from being removed from the folder that it's in.  I
> thought the access flags were a property of the file itself and not the
> directory entry. Not sure how that works.

In POSIX, it's secure so long as we use a directory that doesn't grant
write access to other users, or one that has the sticky bit set such
as "/tmp". A directory that has the sticky bit set allows only root
and the owner of the file to unlink the file.

In Windows, a user's default %TEMP% directory is only accessible by
the user, SYSTEM, and Administrators. The only way others can delete a
file there is if the file security is modified to allow it (possible
for individual files, unlike POSIX). This works even with no access to
the temp directory itself because users have SeChangeNotifyPrivilege,
which bypasses traverse (execute) access checks.

From ajm at flonidan.dk  Wed Mar 20 07:25:03 2019
From: ajm at flonidan.dk (Anders Munch)
Date: Wed, 20 Mar 2019 11:25:03 +0000
Subject: [Python-Dev] Remove tempfile.mktemp()
Message-ID: <HE1PR05MB32110BC69414568A25BE4B3DB4410@HE1PR05MB3211.eurprd05.prod.outlook.com>

Nathaniel J. Smith:
> Historically, mktemp variants have caused *tons* of serious security
> vulnerabilities. It's not a theoretical issue.

All the more reason to have a standard library function that gets it right.

> The choice of ENTROPY_BYTES is an interesting question. 16 (= 128 bits) would
> be a nice "obviously safe" number, and gives 22-byte filenames. We might be
> able to get away with fewer, if we had a plausible cost model for the
> attack. This is another point where a security specialist might be helpful :-).

I'm not a security specialist but I play one on TV.
Here's my take on it.

Any kind of brute force attack will require at least one syscall per try, to
create a file or check if a file by a given name exists.  It's a safe assumption
that names have to be tried individually, because if the attacker has a faster
way of enumerating existing file names, then the entropy of the filename is
worthless anyway.

That means even with only 41 bits of entry, the attacker will have make 2^40
tries on average.  For an individual short-lived file, that could be enough;
even with a billion syscalls per second, that's over a thousand seconds, leaving
plenty of time to initiate whatever writes the file.

However, there could be applications where the window of attack is very long,
hours or days even, or that are constantly writing new temporary files, and
where the attacker can keep trying at a rapid pace, and then 41 bits is
definitely not secure.

128 bits seems like overkill: There's no birthday attack because no-one keeps
2^(ENTROPY_BITS/2) files around, and the attack is running on the attackee's
system, so there's no using specialised accelerator hardware.  I'd say 64 bits
is enough under those circumstances, but I wouldn't be surprised if a better
security specialist could make a case for more.  So maybe go with 80 bits,
that's puts it at 15 or 16 characters.


Med venlig hilsen/Best regards

 Anders Munch
Chief Security Architect

T: +45 76266981  *  M: +45 51856626
ajm at flonidan.dk  *  www.flonidan.com 
 FLONIDAN A/S  *  Islandsvej 29  *  DK-8700 Horsens  *  CVR: 89919916    
Winner of the 2018 Frost & Sullivan Customer Leadership Award

From vstinner at redhat.com  Wed Mar 20 07:45:40 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 20 Mar 2019 12:45:40 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <HE1PR05MB32110BC69414568A25BE4B3DB4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
References: <HE1PR05MB32110BC69414568A25BE4B3DB4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
Message-ID: <CA+3bQGGx7wgf+HrUp3HE=2BTwJdacuVs0YUYhETzdwfSkPmS+Q@mail.gmail.com>

Hi,

I'm not really convinced that mktemp() should be made "more secure".
To be clear: mktemp() is vulnerable by design. It's not a matter of
entropy. You can watch the /tmp directory using inotify and "discover"
immediately the "secret" filename, it doesn't depend on the amount of
entropy used to generate the filename. A function is either unsafe or
secure.

Why mktemp() only uses 8 characters? Well, I guess that humans like to
be able to copy manually (type) a filename :-)

Note: For the ones who didn't notice, "mktemp()" name comes from a
function with the same name in the libc.
http://man7.org/linux/man-pages/man3/mktemp.3.html

Victor

Le mer. 20 mars 2019 ? 12:29, Anders Munch <ajm at flonidan.dk> a ?crit :
>
> Nathaniel J. Smith:
> > Historically, mktemp variants have caused *tons* of serious security
> > vulnerabilities. It's not a theoretical issue.
>
> All the more reason to have a standard library function that gets it right.
>
> > The choice of ENTROPY_BYTES is an interesting question. 16 (= 128 bits) would
> > be a nice "obviously safe" number, and gives 22-byte filenames. We might be
> > able to get away with fewer, if we had a plausible cost model for the
> > attack. This is another point where a security specialist might be helpful :-).
>
> I'm not a security specialist but I play one on TV.
> Here's my take on it.
>
> Any kind of brute force attack will require at least one syscall per try, to
> create a file or check if a file by a given name exists.  It's a safe assumption
> that names have to be tried individually, because if the attacker has a faster
> way of enumerating existing file names, then the entropy of the filename is
> worthless anyway.
>
> That means even with only 41 bits of entry, the attacker will have make 2^40
> tries on average.  For an individual short-lived file, that could be enough;
> even with a billion syscalls per second, that's over a thousand seconds, leaving
> plenty of time to initiate whatever writes the file.
>
> However, there could be applications where the window of attack is very long,
> hours or days even, or that are constantly writing new temporary files, and
> where the attacker can keep trying at a rapid pace, and then 41 bits is
> definitely not secure.
>
> 128 bits seems like overkill: There's no birthday attack because no-one keeps
> 2^(ENTROPY_BITS/2) files around, and the attack is running on the attackee's
> system, so there's no using specialised accelerator hardware.  I'd say 64 bits
> is enough under those circumstances, but I wouldn't be surprised if a better
> security specialist could make a case for more.  So maybe go with 80 bits,
> that's puts it at 15 or 16 characters.
>
>
> Med venlig hilsen/Best regards
>
>  Anders Munch
> Chief Security Architect
>
> T: +45 76266981  *  M: +45 51856626
> ajm at flonidan.dk  *  www.flonidan.com
>  FLONIDAN A/S  *  Islandsvej 29  *  DK-8700 Horsens  *  CVR: 89919916
> Winner of the 2018 Frost & Sullivan Customer Leadership Award
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From steve at pearwood.info  Wed Mar 20 07:57:10 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 20 Mar 2019 22:57:10 +1100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <HE1PR05MB32110BC69414568A25BE4B3DB4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
References: <HE1PR05MB32110BC69414568A25BE4B3DB4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
Message-ID: <20190320115710.GE12502@ando.pearwood.info>

On Wed, Mar 20, 2019 at 11:25:03AM +0000, Anders Munch wrote:

> 128 bits seems like overkill: There's no birthday attack because no-one keeps
> 2^(ENTROPY_BITS/2) files around, 

You haven't seen my Downloads folder... :-)

But seriously:

> and the attack is running on the attackee's
> system, so there's no using specialised accelerator hardware.  I'd say 64 bits
> is enough under those circumstances, but I wouldn't be surprised if a better
> security specialist could make a case for more.  So maybe go with 80 bits,
> that's puts it at 15 or 16 characters.

Why be so miserly with entropy? This probably isn't a token that goes to 
a human, who may have to type it into a web browser, or send it by SMS. 
Its likely to be a name used only by the machine. Using 128 bits is just 
22 characters using secrets.token_urlsafe().

The default entropy used by secrets is 32 bytes, which gives a 43 
character token. I have plenty of files with names longer than that:

"Funny video of cat playing piano while dog does backflips.mp4"

Of course, if you have some specific need for the file name to be 
shorter (or longer!) then there ought to be a way to set the entropy 
used. But I think the default secrets entropy is fine, and its better to 
have longer names than shorter ones, within reason. I don't think 40-50 
characters (plus any prefix or suffix) is excessive for a temporary file 
intended for use by an application rather than a human.



-- 
Steven

From J.Demeyer at UGent.be  Wed Mar 20 07:53:20 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Wed, 20 Mar 2019 12:53:20 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CA+3bQGGx7wgf+HrUp3HE=2BTwJdacuVs0YUYhETzdwfSkPmS+Q@mail.gmail.com>
References: <HE1PR05MB32110BC69414568A25BE4B3DB4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
 <CA+3bQGGx7wgf+HrUp3HE=2BTwJdacuVs0YUYhETzdwfSkPmS+Q@mail.gmail.com>
Message-ID: <5C9229B0.20501@UGent.be>

On 2019-03-20 12:45, Victor Stinner wrote:
> You can watch the /tmp directory using inotify and "discover"
> immediately the "secret" filename, it doesn't depend on the amount of
> entropy used to generate the filename.

That's not the problem. The security issue here is guessing the filename 
*before* it's created and putting a different file or symlink in place.

So I actually do think that mktemp() could be made secure by using a 
longer name generated by a secure random generator.

From steve at pearwood.info  Wed Mar 20 08:09:44 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 20 Mar 2019 23:09:44 +1100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <CA+3bQGGx7wgf+HrUp3HE=2BTwJdacuVs0YUYhETzdwfSkPmS+Q@mail.gmail.com>
References: <HE1PR05MB32110BC69414568A25BE4B3DB4410@HE1PR05MB3211.eurprd05.prod.outlook.com>
 <CA+3bQGGx7wgf+HrUp3HE=2BTwJdacuVs0YUYhETzdwfSkPmS+Q@mail.gmail.com>
Message-ID: <20190320120944.GF12502@ando.pearwood.info>

On Wed, Mar 20, 2019 at 12:45:40PM +0100, Victor Stinner wrote:
> Hi,
> 
> I'm not really convinced that mktemp() should be made "more secure".
> To be clear: mktemp() is vulnerable by design. It's not a matter of
> entropy. You can watch the /tmp directory using inotify and "discover"
> immediately the "secret" filename, it doesn't depend on the amount of
> entropy used to generate the filename. A function is either unsafe or
> secure.

Security is not a binary state, it is never either-or "unsafe" or 
"secure". Secure against what attacks? Unsafe under what circumstances?

I can use the unsafe mktemp on a stand alone single-user computer, 
disconnected from the internet, guaranteed to have nothing but trusted 
software, and it will be secure in practice.

Or I can use the "safe interfaces" and I'm still vulnerable to an 
Advanced Persistent Threat that has compromised the OS specifically to 
target my application. If the attacker controls the OS or the hardware, 
then effectively they've already won.


-- 
Steven

From ajm at flonidan.dk  Wed Mar 20 08:25:36 2019
From: ajm at flonidan.dk (Anders Munch)
Date: Wed, 20 Mar 2019 12:25:36 +0000
Subject: [Python-Dev] Remove tempfile.mktemp()
Message-ID: <HE1PR05MB321127366A723100297A04CBB4410@HE1PR05MB3211.eurprd05.prod.outlook.com>

Steven D'Aprano:
>> 128 bits seems like overkill: There's no birthday attack because 
>> no-one keeps 2^(ENTROPY_BITS/2) files around
> You haven't seen my Downloads folder... :-)

I put it to you that those files are not temporary :-)

> Why be so miserly with entropy?

I don't necessarily disagree.  

> Using 128 bits is just 22 characters using secrets.token_urlsafe().

A little more when you take into account case-insensitive file systems.

regards, Anders


From ajm at flonidan.dk  Wed Mar 20 08:30:50 2019
From: ajm at flonidan.dk (Anders Munch)
Date: Wed, 20 Mar 2019 12:30:50 +0000
Subject: [Python-Dev] Remove tempfile.mktemp()
Message-ID: <HE1PR05MB321197944958498AC3905264B4410@HE1PR05MB3211.eurprd05.prod.outlook.com>

Victor Stinner:
> To be clear: mktemp() is vulnerable by design

No: mktemp() is vulnerable by implementation.  Specifically, returning a file name in a world-accessible location, /tmp.

regards, Anders


From solipsis at pitrou.net  Wed Mar 20 10:22:31 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 20 Mar 2019 15:22:31 +0100
Subject: [Python-Dev] Remove tempfile.mktemp()
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
 <20190319145127.0824f196@fsol> <5C916C71.5070102@canterbury.ac.nz>
Message-ID: <20190320152231.7936863b@fsol>

On Wed, 20 Mar 2019 11:25:53 +1300
Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Antoine Pitrou wrote:
> > Does it always work? According to the docs, """Whether the name can be
> > used to open the file a second time, while the named temporary file is
> > still open, varies across platforms  
> 
> So use NamedTemporaryFile(delete = False) and close it before passing
> it to the other program.

How is it more secure than using mktemp()?

Regards

Antoine.



From ethan at stoneleaf.us  Wed Mar 20 18:24:03 2019
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 20 Mar 2019 15:24:03 -0700
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
Message-ID: <8f49ea84-37d8-d5c0-f19c-8eeb90e902e0@stoneleaf.us>

On 03/19/2019 11:55 AM, Raymond Hettinger wrote:

> I'm working on ways to make improve help() by giving docstrings
>  to member objects.

Cool!

> There's another way I would like to propose.  The __slots__
>  definition already works with any iterable including a
>  dictionary (the dict values are ignored), so we could use the
>  values for the  docstrings.
>
> [...]
> 
> What do you all think about the proposal?

This proposal only works with objects defining __slots__, and only the objects in __slots__?  Does it help Enum, dataclasses, or other enhanced classes/objects?

--
~Ethan~

From greg at krypto.org  Wed Mar 20 18:30:14 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 20 Mar 2019 15:30:14 -0700
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <q6sqrh$5318$1@blaine.gmane.org>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
 <q6sqrh$5318$1@blaine.gmane.org>
Message-ID: <CAGE7PN+K0amuQS+nkwrukwfJEmkzCjPXce_vZU8zsqyMWv-zUQ@mail.gmail.com>

(answers above and below the quoting)

I like the idea of documenting attributes, but we shouldn't force the user
to use __slots__ as that has significant side effects and is rarely
something people should bother to use.  There are multiple types of
attributes.  class and instance.  but regardless of where they are
initialized, they all define the API shape of a class (or instance).

Q: Where at runtime regardless of syntax chosen would such docstrings
live?  One (of many) common conventions today is to just put them into an
Attributes: or similar section of the class docstring.  We could actually
do that automatically by appending a section to the class docstring, but
that unstructures the data enshrining one format and could break existing
code for the users of the few but existing APIs that treat docstrings as
structured runtime data instead of documentation if someone were to try and
use attribute docstrings on subclasses of those library types.  (ply does
this, I believe some database abstraction APIs do as well).

On Wed, Mar 20, 2019 at 12:41 AM Serhiy Storchaka <storchaka at gmail.com>
wrote:

> 19.03.19 20:55, Raymond Hettinger ????:
> > I'm working on ways to make improve help() by giving docstrings to
> member objects.
> >
> > One way to do it is to wait until after the class definition and then
> make individual, direct assignments to __doc__ attributes.This way widely
> the separates docstrings from their initial __slots__ definition.   Working
> downstream from the class definition feels awkward and doesn't look pretty.
> >
> > There's another way I would like to propose?.  The __slots__ definition
> already works with any iterable including a dictionary (the dict values are
> ignored), so we could use the values for the  docstrings.
>
> I think it would be nice to separate docstrings from the bytecode. This
> would be allow to have several translated sets of docstrings and load an
> appropriate set depending on user preferences. This would help in
> teaching Python.
>
> It is possible with docstrings of modules, classes, functions, methods
> and properties (created by using the decorator), because the compiler
> knows what string literal is a docstring. But this is impossible with
> namedtuple fields and any of the above ideas for slots.
>
> It would be nice to allow to specify docstrings for slots as for methods
> and properties. Something like in the following pseudocode:
>
> class NormalDist:
>      slot mu:
>         '''Arithmetic mean'''
>      slot sigma:
>          '''Standard deviation'''
>

I don't think adding a 'slot' keyword even if limited in scope to class
body definition level is a good idea (very painful anytime we reserve a new
word that is already used in code and APIs).


> It would be also nice to annotate slots and add default values (used
> when the slot value was not set).
>
> class NormalDist:
>      mu: float = 0.0
>         '''Arithmetic mean'''
>      sigma: float = 1.0
>          '''Standard deviation'''
>
>
Something along these lines is more interesting to me.  And could be
applied to variables in _any_ scope.  though there wouldn't be a point in
using a string in context where the name isn't bound to a class or module.

The best practice today remains "just use the class docstring to document
your public class and instance attributes".  FWIW other languages tend to
generate their documentation from code via comments rather than requiring a
special in language runtime accessible syntax to declare it as
documentation.

It feels like Python is diverging from the norm if we were encourage more
of this __doc__ carried around at runtime implicit assignment than we
already have.  I'm not convinced that is a good thing.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190320/dfce7afd/attachment.html>

From vano at mail.mipt.ru  Wed Mar 20 18:47:27 2019
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Thu, 21 Mar 2019 01:47:27 +0300
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
Message-ID: <416e2cdf-7284-1be0-e4ff-3e40cf93a07a@mail.mipt.ru>

On 19.03.2019 21:55, Raymond Hettinger wrote:
> I'm working on ways to make improve help() by giving docstrings to member objects.
>
> One way to do it is to wait until after the class definition and then make individual, direct assignments to __doc__ attributes.This way widely the separates docstrings from their initial __slots__ definition.   Working downstream from the class definition feels awkward and doesn't look pretty.
>
> There's another way I would like to propose?.  The __slots__ definition already works with any iterable including a dictionary (the dict values are ignored), so we could use the values for the  docstrings.
>
> This keeps all the relevant information in one place (much like we already do with property() objects).  This way already works, we just need a few lines in pydoc to check to see if a dict if present.  This way also looks pretty and doesn't feel awkward.
>
> I've included worked out examples below.  What do you all think about the proposal?
>
>
> Raymond
>
>
> ? https://bugs.python.org/issue36326
>
>
> ====== Desired help() output ======
>
>>>> help(NormalDist)
> Help on class NormalDist in module __main__:
>
> class NormalDist(builtins.object)
>   |  NormalDist(mu=0.0, sigma=1.0)
>   |
>   |  Normal distribution of a random variable
>   |
>   |  Methods defined here:
>   |
>   |  __init__(self, mu=0.0, sigma=1.0)
>   |      NormalDist where mu is the mean and sigma is the standard deviation.
>   |
>   |  cdf(self, x)
>   |      Cumulative distribution function.  P(X <= x)
>   |
>   |  pdf(self, x)
>   |      Probability density function.  P(x <= X < x+dx) / dx
>   |
>   |  ----------------------------------------------------------------------
>   |  Data descriptors defined here:
>   |
>   |  mu
>   |      Arithmetic mean.
>   |
>   |  sigma
>   |      Standard deviation.
>   |
>   |  variance
>   |      Square of the standard deviation.
>
>
>
> ====== Example of assigning docstrings after the class definition ======
>
> class NormalDist:
>      'Normal distribution of a random variable'
>
>      __slots__ = ('mu', 'sigma')
>
>      def __init__(self, mu=0.0, sigma=1.0):
>          'NormalDist where mu is the mean and sigma is the standard deviation.'
>          self.mu = mu
>          self.sigma = sigma
>
>      @property
>      def variance(self):
>          'Square of the standard deviation.'
>          return self.sigma ** 2.
>
>      def pdf(self, x):
>          'Probability density function.  P(x <= X < x+dx) / dx'
>          variance = self.variance
>          return exp((x - self.mu)**2.0 / (-2.0*variance)) / sqrt(tau * variance)
>
>      def cdf(self, x):
>          'Cumulative distribution function.  P(X <= x)'
>          return 0.5 * (1.0 + erf((x - self.mu) / (self.sigma * sqrt(2.0))))
>
> NormalDist.mu.__doc__ = 'Arithmetic mean'
> NormalDist.sigma.__doc__ = 'Standard deviation'

IMO this is another manifestation of the problem that things in the class definition have no access to the class object.
Logically speaking, a definition item should be able to see everything that is defined before it.
For the same reason, we have to jump through hoops to use a class name in a class attribute definition -- see e.g. 
https://stackoverflow.com/questions/14513019/python-get-class-name

If that problem is resolved, you would be able to write something like:

class NormalDist:
     'Normal distribution of a random variable'

     __slots__ = ('mu', 'sigma')

     __self__.mu.__doc__= 'Arithmetic mean'
 ?   __self__.sigma.__doc__= 'Stndard deviation'


>
>
> ====== Example of assigning docstrings with a dict =====
>
> class NormalDist:
>      'Normal distribution of a random variable'
>
>      __slots__ = {'mu' : 'Arithmetic mean.', 'sigma': 'Standard deviation.'}
>
>      def __init__(self, mu=0.0, sigma=1.0):
>          'NormalDist where mu is the mean and sigma is the standard deviation.'
>          self.mu = mu
>          self.sigma = sigma
>
>      @property
>      def variance(self):
>          'Square of the standard deviation.'
>          return self.sigma ** 2.
>
>      def pdf(self, x):
>          'Probability density function.  P(x <= X < x+dx) / dx'
>          variance = self.variance
>          return exp((x - self.mu)**2.0 / (-2.0*variance)) / sqrt(tau * variance)
>
>      def cdf(self, x):
>          'Cumulative distribution function.  P(X <= x)'
>          return 0.5 * (1.0 + erf((x - self.mu) / (self.sigma * sqrt(2.0))))
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- 
Regards,
Ivan


From vano at mail.mipt.ru  Wed Mar 20 18:47:47 2019
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Thu, 21 Mar 2019 01:47:47 +0300
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
Message-ID: <8ac0ef6d-dcfd-54b2-ff89-e3655453e9d0@mail.mipt.ru>

Before we can say if something is "secure" or not, we need a threat model -- i.e we need to agree which use cases we are protecting and from 
what threats.

So far, I've seen these use cases:

1. File for the current process' private use
2. File/file name generated by the current process; written by another process, read by current one
3. File name generated by the current process; written by the current process, read by another one

And the following threats, three axes:

a. Processes run as other users
b. Processes run as the same user (or a user that otherwise automatically has access to all your files)

1. Accidental collision from a process that uses CREATE_NEW or equivalent
2. Accidental collision from a process that doesn't use CREATE_NEW or equivalent
3. Malicious code creating files at random
4. Malicious code actively monitoring file creation

-1. read
-2. write

E.g. for threat b-4), it's not safe to use named files for IPC at all, only case 1 can be secured (with exclusive open).

On 19.03.2019 16:03, St?phane Wirtel wrote:

> Hi,
>
> Context: raise a warning or remove tempfile.mktemp()
> BPO: https://bugs.python.org/issue36309
>
> Since 2.3, this function is deprecated in the documentation, just in the
> documentation. In the code, there is a commented RuntimeWarning.
> Commented by Guido in 2002, because the warning was too annoying (and I
> understand ;-)).
>
> So, in this BPO, we start to discuss about the future of this function
> and Serhiy proposed to discuss on the Python-dev mailing list.
>
> Question: Should we drop it or add a (Pending)DeprecationWarning?
>
> Suggestion and timeline:
>
> 3.8, we raise a PendingDeprecationWarning
>      * update the code
>      * update the documentation
>      * update the tests
>        (check a PendingDeprecationWarning if sys.version_info == 3.8)
>
> 3.9, we change PendingDeprecationWarning to DeprecationWarning
>        (check DeprecationWarning if sys.version_info == 3.9)
>
> 3.9+, we drop tempfile.mktemp()
>
> What do you suggest?
>
> Have a nice day and thank you for your feedback.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- 
Regards,
Ivan


From raymond.hettinger at gmail.com  Wed Mar 20 18:53:26 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Mar 2019 15:53:26 -0700
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <CAGE7PN+K0amuQS+nkwrukwfJEmkzCjPXce_vZU8zsqyMWv-zUQ@mail.gmail.com>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
 <q6sqrh$5318$1@blaine.gmane.org>
 <CAGE7PN+K0amuQS+nkwrukwfJEmkzCjPXce_vZU8zsqyMWv-zUQ@mail.gmail.com>
Message-ID: <1AC277CC-EF1F-4728-876A-ABB8934A5DF6@gmail.com>


> On Mar 20, 2019, at 3:30 PM, Gregory P. Smith <greg at krypto.org> wrote:
> 
> I like the idea of documenting attributes, but we shouldn't force the user to use __slots__ as that has significant side effects and is rarely something people should bother to use.

Member objects are like property objects in that they exist at the class level and show up in the help whether you want them to or not.   AFAICT, they are the only such objects to not have a way to attach docstrings.

For instance level attributes created by __init__, the usual way to document them is in either the class docstring or the __init__ docstring.  This is because they don't actually exist until  __init__ is run.

No one is forcing anyone to use slots.  I'm just proposing that for classes that do use them that there is currently no way to annotate them like we do for property objects (which people aren't being forced to use either).  The goal is to make help() better for whatever people are currently doing.  That shouldn't be controversial.  

Someone not liking or recommending slots is quite different from not wanting them documented.  In the examples I posted (taken from the standard library), the help() is clearly better with the annotations than without.


Raymond





From ethan at stoneleaf.us  Wed Mar 20 18:59:44 2019
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 20 Mar 2019 15:59:44 -0700
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <8f49ea84-37d8-d5c0-f19c-8eeb90e902e0@stoneleaf.us>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
 <8f49ea84-37d8-d5c0-f19c-8eeb90e902e0@stoneleaf.us>
Message-ID: <1475cd51-57a2-c5be-2800-034706ef7b32@stoneleaf.us>

On 03/20/2019 03:24 PM, Ethan Furman wrote:
> On 03/19/2019 11:55 AM, Raymond Hettinger wrote:

>> There's another way I would like to propose.  The __slots__
>>  definition already works with any iterable including a
>>  dictionary (the dict values are ignored), so we could use the
>>  values for the  docstrings.
>>
>> [...]
>>
>> What do you all think about the proposal?
> 
> This proposal only works with objects defining __slots__, and only
>  the objects in __slots__?  Does it help Enum, dataclasses, or other
>  enhanced classes/objects?

Hmm.  Said somewhat less snarkily, is there a more general solution to the problem of absent docstrings or do we have to attack this problem piece-by-piece?

--
~Ethan~

From raymond.hettinger at gmail.com  Wed Mar 20 19:05:26 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Mar 2019 16:05:26 -0700
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <416e2cdf-7284-1be0-e4ff-3e40cf93a07a@mail.mipt.ru>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
 <416e2cdf-7284-1be0-e4ff-3e40cf93a07a@mail.mipt.ru>
Message-ID: <C09BF55D-6EC7-46D7-90EB-D82B5D72DB58@gmail.com>


> On Mar 20, 2019, at 3:47 PM, Ivan Pozdeev via Python-Dev <python-dev at python.org> wrote:
> 
>> NormalDist.mu.__doc__ = 'Arithmetic mean'
>> NormalDist.sigma.__doc__ = 'Standard deviation'
> 
> IMO this is another manifestation of the problem that things in the class definition have no access to the class object.
> Logically speaking, a definition item should be able to see everything that is defined before it.

The member objects get created downstream by the type() metaclass.  So, there isn't a visibility issue because the objects don't exist yet.


Raymond


From raymond.hettinger at gmail.com  Wed Mar 20 19:17:13 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Mar 2019 16:17:13 -0700
Subject: [Python-Dev] Best way to specify docstrings for member objects
In-Reply-To: <1475cd51-57a2-c5be-2800-034706ef7b32@stoneleaf.us>
References: <B8318F69-46D0-43E5-9E60-B8CD1AE8E887@gmail.com>
 <8f49ea84-37d8-d5c0-f19c-8eeb90e902e0@stoneleaf.us>
 <1475cd51-57a2-c5be-2800-034706ef7b32@stoneleaf.us>
Message-ID: <3C4F62DF-AE45-4CEE-8E9C-E87892FCC3A0@gmail.com>


> On Mar 20, 2019, at 3:59 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> 
> Hmm.  Said somewhat less snarkily, is there a more general solution to the problem of absent docstrings or do we have to attack this problem piece-by-piece?

I think this is the last piece.  The pydoc help() utility already knows how to find docstrings for other class level descriptors:  property, class method, staticmethod.

Enum() already has nice looking help() output because the class variables are assigned values that have a nice __repr__, making them self documenting.

By design, dataclasses aren't special -- they just make regular classes, similar to or better than you would write by hand.


Raymond

From vstinner at redhat.com  Wed Mar 20 20:22:56 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 21 Mar 2019 01:22:56 +0100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
Message-ID: <CA+3bQGGqLJHEu6M6xrSPV2hB1FUwqTp-u1kTRNSEuWYWwoTRYA@mail.gmail.com>

Hi,

Le lun. 18 mars 2019 ? 23:41, Raymond Hettinger
<raymond.hettinger at gmail.com> a ?crit :
> We're having a super interesting discussion on https://bugs.python.org/issue34160 .  It is now marked as a release blocker and warrants a broader discussion.

Thanks for starting a thread on python-dev. I'm the one who raised the
priority to release blocker to trigger such discussion on python-dev.


> Our problem is that at least two distinct and important users have written tests that depend on exact byte-by-byte comparisons of the final serialization.

Sorry but I don't think that it's a good summary of the issue. IMHO
the issue is more general about how we introduce backward incompatible
in Python.

The migration from Python 2 to Python 3 took around ten years. That's
way too long and it caused a lot of troubles in the Python community.
IMHO one explanation is our patronizing behavior regarding to users
that I would like to summarize as "your code is wrong, you have to fix
it" (whereas the code was working well for 10 years with Python 2!).

I'm not opposed to backward incompatible changes, but I think that we
must very carefully prepare the migration and do our best to help
users to migrate their code.


> 2). Go into every XML module and add attribute sorting options to each function that generate xml. (...)

Written like that, it sounds painful and a huge project... But in
practice, the implementation looks simple and straightforward:
https://github.com/python/cpython/pull/12354/files

I don't understand why such simple solution has been rejected.

IMHO adding an optional sort parameter is just the *bare minimum* that
we can do for our users.

Alternatives have been proposed like a recipe to sort node attributes
before serialization, but honestly, it's way too complex. I don't want
to have to copy such recipe to every project. Add a new function,
import it, use it where XML is written into a file, etc. Taken alone,
maybe it's acceptable. But please remember that some companies are
still porting their large Python 2 code base to Python 3. This new
backward incompatible gets on top of the pile of other backward
incompatible changes between 2.7 and 3.8.

I would prefer to be able to "just add" sort=True. Don't forget that
tests like "if sys.version >= (3, 8):"  will be needed which makes the
overall fix more complicated.

Said differently, the stdlib should help the user to update Python.
The pain should not only be on the user side.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From raymond.hettinger at gmail.com  Wed Mar 20 20:26:18 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Mar 2019 17:26:18 -0700
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6q8af$3i67$1@blaine.gmane.org>
 <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
Message-ID: <FB1A02AF-3224-40FF-AF2E-4982495F149E@gmail.com>


> On Mar 19, 2019, at 4:53 AM, Ned Batchelder <ned at nedbatchelder.com> wrote:
> 
> None of this is impossible, but please try not to preach to us maintainers that we are doing it wrong, that it will be easy to fix, etc

There's no preaching and no judgment.  We can't have a conversation though if we can't state the crux of the problem: some existing tests in third-party modules depend on the XML serialization being byte-for-byte identical forever. The various respondents to this thread have indicated that the standard library should only make that guarantee within a single feature release and that it may to vary across feature releases.

For docutils, it may end-up being an easy fix (either with a semantic comparison or with regenerating the target files when point releases differ).  For Coverage, I don't make any presumption that reengineering the tests will be easy or fun.  Several mitigation strategies have been proposed:

* alter to element creation code to create the attributes in the desired order
* use a canonicalization tool to create output that is guarantee not to change
* generate new baseline files when a feature release changes
* apply Stefan's recipe for reordering attributes
* make a semantic level comparison

Will any other these work for you?


Raymond








From vstinner at redhat.com  Wed Mar 20 20:46:14 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 21 Mar 2019 01:46:14 +0100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <FB1A02AF-3224-40FF-AF2E-4982495F149E@gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6q8af$3i67$1@blaine.gmane.org>
 <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
 <FB1A02AF-3224-40FF-AF2E-4982495F149E@gmail.com>
Message-ID: <CA+3bQGFfUUWqBbXf8Vmp2yJU+agh+PLS8rgao+TsmAauy7RQzw@mail.gmail.com>

Le jeu. 21 mars 2019 ? 01:30, Raymond Hettinger
<raymond.hettinger at gmail.com> a ?crit :
> There's no preaching and no judgment.  We can't have a conversation though if we can't state the crux of the problem: some existing tests in third-party modules depend on the XML serialization being byte-for-byte identical forever. The various respondents to this thread have indicated that the standard library should only make that guarantee within a single feature release and that it may to vary across feature releases.
>
> For docutils, it may end-up being an easy fix (either with a semantic comparison or with regenerating the target files when point releases differ).  For Coverage, I don't make any presumption that reengineering the tests will be easy or fun.  Several mitigation strategies have been proposed:
>
> * alter to element creation code to create the attributes in the desired order
> * use a canonicalization tool to create output that is guarantee not to change
> * generate new baseline files when a feature release changes
> * apply Stefan's recipe for reordering attributes
> * make a semantic level comparison
>
> Will any other these work for you?

Python 3.8 is still in a very early stage of testing. We only started
to discover which projects are broken by the XML change.

IMHO the problem is wider than just unit tests written in Python.
Python can be used to produce the XML, but other languages can be used
to parse or compare the generated XML. For example, if the generated
file is stored in Git, it will be seen as modified and "git diff" will
show a lot of "irrelevant" changes.

Comparison of XML using string comparison can also be used to avoid
expensive disk/database write or reduce network bandwidth. That's an
issue if the program isn't written in Python, whereas the XML is
generated by Python.

Getting the same output on Python 3.7 and Python 3.8 is also matter
for https://reproducible-builds.org/

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From raymond.hettinger at gmail.com  Wed Mar 20 20:56:48 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Mar 2019 17:56:48 -0700
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CA+3bQGGqLJHEu6M6xrSPV2hB1FUwqTp-u1kTRNSEuWYWwoTRYA@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CA+3bQGGqLJHEu6M6xrSPV2hB1FUwqTp-u1kTRNSEuWYWwoTRYA@mail.gmail.com>
Message-ID: <C9A66D39-2F32-4A1D-9CC2-7328C0FD02BC@gmail.com>


> On Mar 20, 2019, at 5:22 PM, Victor Stinner <vstinner at redhat.com> wrote:
> 
> I don't understand why such simple solution has been rejected.

It hasn't been rejected. That is above my pay grade.  Stefan and I recommended against going down this path. However, since you're in disagreement and have marked this as a release blocker, it is now time for the steering committee to earn their pay (which is at least double what I'm making) or defer to the principal module maintainer, Stefan.

To recap reasons for not going down this path:

1) The only known use case for a "sort=True" parameter is to perpetuate the practice of byte-by-byte output comparisons guaranteed to work across feature releases.  The various XML experts in this thread have opined that isn't something we should guarantee (and sorting isn't the only aspect detail subject to change, Stefan listed others).

2) The intent of the XML modules is to implement the specification and be interoperable with other languages and other XML tools. It is not intended to be used to generate an exact binary output.  Per section 3.1 of the XML spec, "Note that the order of attribute specifications in a start-tag or empty-element tag is not significant."

3) Mitigating a test failure is a one-time problem. API expansions are forever.

4) The existing API is not small and presents a challenge for teaching. Making the API bigger will make it worse.

5) As far as I can tell, XML tools in other languages (such as Java) don't sort (and likely for good reason).  LXML is dropping its attribute sorting as well, so the standard library would become more of an outlier.


Raymond


From vstinner at redhat.com  Wed Mar 20 21:07:01 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 21 Mar 2019 02:07:01 +0100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
Message-ID: <CA+3bQGH4g7exNtvEYLUYMSuUeYhSeY4F7PjJUHKmf090dfyOkA@mail.gmail.com>

Le lun. 18 mars 2019 ? 23:41, Raymond Hettinger
<raymond.hettinger at gmail.com> a ?crit :
> The code in the current 3.8 alpha differs from 3.7 in that it removes attribute sorting and instead preserves the order the user specified when creating an element.  As far as I can tell, there is no objection to this as a feature.

By the way, what's the rationale of this backward incompatible change?

I found this short message:
"FWIW, this issue arose from an end-user problem. She had a hard
requirement to show a security clearance level as the first attribute.
We did find a work around but it was hack."
https://bugs.python.org/issue34160#msg338098

It's the first time that I hear an user asking to preserve attribute
insertion order (or did I miss a previous request?). Technically, it
was possible to implement the feature earlier using OrderedDict. So
why doing it now?

Is it really worth it to break Python backward compatibility (change
the default behavior) for everyone, if it's only needed for few users?


> 1) Revert back to the 3.7 behavior. This of course, makes all the test pass :-)  The downside is that it perpetuates the practice of bytewise equality tests and locks in all implementation quirks forever.  I don't know of anyone advocating this option, but it is the simplest thing to do.

Can't we revert Python 3.7 behavior and add a new opt-in option to
preserve the attribution insertion order (current Python 3.8 default
behavior)?

Python 3.7, sorting attributes by name, doesn't sound so silly to me.
It's one arbitrary choice, but at least the output is deterministic.
And well, Python is doing that for 20 years :-)


> 4) Fix the tests in the third-party modules (...)

I also like the option "not break the backward compatibility" to not
have to fix any project :-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From raymond.hettinger at gmail.com  Wed Mar 20 23:07:49 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Mar 2019 20:07:49 -0700
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CA+3bQGH4g7exNtvEYLUYMSuUeYhSeY4F7PjJUHKmf090dfyOkA@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CA+3bQGH4g7exNtvEYLUYMSuUeYhSeY4F7PjJUHKmf090dfyOkA@mail.gmail.com>
Message-ID: <204978ED-B6D8-45F8-BA15-F1107E481ACE@gmail.com>


> On Mar 20, 2019, at 6:07 PM, Victor Stinner <vstinner at redhat.com> wrote:
> 
> what's the rationale of this backward incompatible change?

Please refrain from abusive mischaracterizations.  It is only backwards incompatible if there was a guaranteed behavior.  Whether there was or not is what this thread is about.  

My reading of this thread was that the various experts did not want to lock in the 3.7 behavior nor did they think the purpose of the XML modules is to produce an exact binary output.  The lxml maintainer is dropping sorting (its expensive and it overrides the order specified by the user). Other XML modules don't sort. It only made sense as a way to produce a deterministic output within a feature release back when there was no other way to do it.

For my part, any agreed upon outcome in fine. I'm not willing be debased further, so I am out of this discussion. It's up to you all to do the right thing.


Raymond




From greg.ewing at canterbury.ac.nz  Thu Mar 21 00:16:20 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 Mar 2019 17:16:20 +1300
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <20190320152231.7936863b@fsol>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
 <20190319145127.0824f196@fsol> <5C916C71.5070102@canterbury.ac.nz>
 <20190320152231.7936863b@fsol>
Message-ID: <5C931014.1000008@canterbury.ac.nz>

Antoine Pitrou wrote:
> On Wed, 20 Mar 2019 11:25:53 +1300
> Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
>>So use NamedTemporaryFile(delete = False) and close it before passing
>>it to the other program.
> 
> How is it more secure than using mktemp()?

It's not, but it solves the problem someone suggested of another
program not being able to access and/or delete the file.

-- 
Greg

From solipsis at pitrou.net  Thu Mar 21 07:29:53 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Mar 2019 12:29:53 +0100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CA+3bQGH4g7exNtvEYLUYMSuUeYhSeY4F7PjJUHKmf090dfyOkA@mail.gmail.com>
Message-ID: <20190321122953.74d23952@fsol>

On Thu, 21 Mar 2019 02:07:01 +0100
Victor Stinner <vstinner at redhat.com> wrote:
> Le lun. 18 mars 2019 ? 23:41, Raymond Hettinger
> <raymond.hettinger at gmail.com> a ?crit :
> > The code in the current 3.8 alpha differs from 3.7 in that it removes attribute sorting and instead preserves the order the user specified when creating an element.  As far as I can tell, there is no objection to this as a feature.  
> 
> By the way, what's the rationale of this backward incompatible change?
> 
> I found this short message:
> "FWIW, this issue arose from an end-user problem. She had a hard
> requirement to show a security clearance level as the first attribute.
> We did find a work around but it was hack."
> https://bugs.python.org/issue34160#msg338098
> 
> It's the first time that I hear an user asking to preserve attribute
> insertion order (or did I miss a previous request?). Technically, it
> was possible to implement the feature earlier using OrderedDict. So
> why doing it now?
> 
> Is it really worth it to break Python backward compatibility (change
> the default behavior) for everyone, if it's only needed for few users?

The argument you're making is weird here.  If only "a few users" need a
deterministic ordering of XML attributes, then compatibility is broken
only for "a few users", not for "everyone".

Most users and applications should /never/ care about the order of XML
attributes.

Regards

Antoine.



From solipsis at pitrou.net  Thu Mar 21 08:09:00 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Mar 2019 13:09:00 +0100
Subject: [Python-Dev] Reproducible output (Re: Is XML serialization output
 guaranteed to be bytewise identical forever?)
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <q6q8af$3i67$1@blaine.gmane.org>
 <91f9f847-6f9f-35cb-ccc2-5838a65a9f18@nedbatchelder.com>
 <FB1A02AF-3224-40FF-AF2E-4982495F149E@gmail.com>
 <CA+3bQGFfUUWqBbXf8Vmp2yJU+agh+PLS8rgao+TsmAauy7RQzw@mail.gmail.com>
Message-ID: <20190321130900.01c05765@fsol>

On Thu, 21 Mar 2019 01:46:14 +0100
Victor Stinner <vstinner at redhat.com> wrote:
> 
> Getting the same output on Python 3.7 and Python 3.8 is also matter
> for https://reproducible-builds.org/

If you want reproducible output, you should settle on a well-known
version of Python.  You don't expect two different versions of gcc to
produce the exact same binary.  Even compression utilities (such as
gzip, xz...) can get improvements over time that change the binary
output, for example making it smaller.

Regards

Antoine.



From steve at holdenweb.com  Thu Mar 21 13:02:28 2019
From: steve at holdenweb.com (Steve Holden)
Date: Thu, 21 Mar 2019 17:02:28 +0000
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <20190321122953.74d23952@fsol>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CA+3bQGH4g7exNtvEYLUYMSuUeYhSeY4F7PjJUHKmf090dfyOkA@mail.gmail.com>
 <20190321122953.74d23952@fsol>
Message-ID: <CAMofdRDNzxi8AjS_2BGSvRMH8jwtzQadFzv2TszThzf=p0_Zvg@mail.gmail.com>

On Thu, Mar 21, 2019 at 11:33 AM Antoine Pitrou <solipsis at pitrou.net> wrote:

> [...]
>
> Most users and applications should /never/ care about the order of XML
> attributes.
>
> Regards
>
> Antoine
>

Especially as the standards specifically say that ordering has no semantic
impact.

Byte-by-byte comparison of XML is almost always inappropriate.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190321/ebdb38cb/attachment.html>

From p.f.moore at gmail.com  Thu Mar 21 13:23:55 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 21 Mar 2019 17:23:55 +0000
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CAMofdRDNzxi8AjS_2BGSvRMH8jwtzQadFzv2TszThzf=p0_Zvg@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CA+3bQGH4g7exNtvEYLUYMSuUeYhSeY4F7PjJUHKmf090dfyOkA@mail.gmail.com>
 <20190321122953.74d23952@fsol>
 <CAMofdRDNzxi8AjS_2BGSvRMH8jwtzQadFzv2TszThzf=p0_Zvg@mail.gmail.com>
Message-ID: <CACac1F8m5xxGBivvY-q-+dZtJ-YjvQ0Dto+hogkKSTUY6N=JKg@mail.gmail.com>

On Thu, 21 Mar 2019 at 17:05, Steve Holden <steve at holdenweb.com> wrote:
>
> On Thu, Mar 21, 2019 at 11:33 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>> [...]
>>
>> Most users and applications should /never/ care about the order of XML
>> attributes.
>>
>> Regards
>>
>> Antoine
>
>
> Especially as the standards specifically say that ordering has no semantic impact.
>
> Byte-by-byte comparison of XML is almost always inappropriate.

Conversely, if ordering has no semantic impact, there's no real
justification for asking for the current order to be changed.

In practice, allowing the user to control the ordering (by preserving
input order) gives users a way of handling (according to the standard)
broken consumers who ascribe semantic meaning to the attribute order.
So there's a small benefit for real-world users having to deal with
non-compliant software. But that benefit is by definition small, as
standards-compliant software won't be affected.

The cost of making the change to projects that rely on the current
output is significant, and that should be considered. But there's also
the question of setting a precedent. If we do reject this change
because of the cost to 3rd parties, are we then committing Python to
guaranteeing sorted attribute order (and worse, byte-for-byte
reproducible output) for ever - a far stronger commitment than the
standards require of us? That seems to me to be an extremely bad
precedent to set.

There's no good answer here - maybe a possible compromise would be for
us to document explicitly in 3.8 that output is only guaranteed
identical to the level the standards require (i.e., attribute order is
not guaranteed to be preserved) and then make this change in 3.9. But
in practice, that's not really any better for projects like coverage -
it just delays the point when they have to bite the bullet (and it's
not like 3.8 is imminent - there's plenty of time between now and 3.8
without adding an additional delay).

Reluctantly, I think I'd have to say that I don't think we should
reject this change simply because existing users rely on the exact
output currently being produced.

To mitigate the impact on 3rd parties, it would be very helpful if we
could add to the stdlib some form of "compare two XML documents for
semantic equality up to the level that the standards require". 3rd
party code could then use that if it's present, and fall back to
byte-equality if it's not. If we could get something like that for
3.9, but not for 3.8, then that would seem to me to be a good reason
to defer this change until 3.9 (because we don't want to have 3.8
being an exception where there's no semantic comparison function, but
the byte-equality fallback doesn't work - that's just needlessly
annoying).

Paul

From jcgoble3 at gmail.com  Thu Mar 21 13:26:00 2019
From: jcgoble3 at gmail.com (Jonathan Goble)
Date: Thu, 21 Mar 2019 13:26:00 -0400
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CAMofdRDNzxi8AjS_2BGSvRMH8jwtzQadFzv2TszThzf=p0_Zvg@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CA+3bQGH4g7exNtvEYLUYMSuUeYhSeY4F7PjJUHKmf090dfyOkA@mail.gmail.com>
 <20190321122953.74d23952@fsol>
 <CAMofdRDNzxi8AjS_2BGSvRMH8jwtzQadFzv2TszThzf=p0_Zvg@mail.gmail.com>
Message-ID: <CAK256p1sEq=PyEJCFnJ0QeQ+C81d-NjYj4RUpPemSyT0RQJW-w@mail.gmail.com>

On Thu, Mar 21, 2019, 1:05 PM Steve Holden <steve at holdenweb.com> wrote:

> On Thu, Mar 21, 2019 at 11:33 AM Antoine Pitrou <solipsis at pitrou.net>
> wrote:
>
>> [...]
>>
>> Most users and applications should /never/ care about the order of XML
>> attributes.
>>
>> Regards
>>
>> Antoine
>>
>
> Especially as the standards specifically say that ordering has no semantic
> impact.
>

When you have a lot of attributes, though, sometimes having them in a
particular defined order can make it easier to reason about and make sense
of the code when manually reviewing it.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190321/97d071d8/attachment.html>

From stefan_ml at behnel.de  Thu Mar 21 13:43:22 2019
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 21 Mar 2019 18:43:22 +0100
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CA+3bQGGqLJHEu6M6xrSPV2hB1FUwqTp-u1kTRNSEuWYWwoTRYA@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CA+3bQGGqLJHEu6M6xrSPV2hB1FUwqTp-u1kTRNSEuWYWwoTRYA@mail.gmail.com>
Message-ID: <q70ifs$2ttu$1@blaine.gmane.org>

Victor Stinner schrieb am 21.03.19 um 01:22:
> Alternatives have been proposed like a recipe to sort node attributes
> before serialization, but honestly, it's way too complex.

Hm, really? Five lines of simple and obvious Python code, that provide a
fast and completely Python-version agnostic solution to the problem that a
few users have, are "way too complex" ? That sounds a bit extreme to me.


> I don't want
> to have to copy such recipe to every project. Add a new function,
> import it, use it where XML is written into a file, etc. Taken alone,
> maybe it's acceptable. But please remember that some companies are
> still porting their large Python 2 code base to Python 3. This new
> backward incompatible gets on top of the pile of other backward
> incompatible changes between 2.7 and 3.8.
> 
> I would prefer to be able to "just add" sort=True. Don't forget that
> tests like "if sys.version >= (3, 8):"  will be needed which makes the
> overall fix more complicated.

Yes, exactly! Users would have to add that option *conditionally* to their
code somewhere. Personally, I really dislike having to say "if Python
version is X do this, otherwise, do that". I prefer a solution that just
works. There are at least four approaches that generally work across Python
releases: ignoring the ordering, using C14N, creating attributes in order,
sorting attributes before serialisation. I'd prefer if users picked one of
those, preferably the right on for their use case, rather than starting to
put version specific kludges into their code.

Stefan


From eryksun at gmail.com  Thu Mar 21 14:12:54 2019
From: eryksun at gmail.com (eryk sun)
Date: Thu, 21 Mar 2019 13:12:54 -0500
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <5C931014.1000008@canterbury.ac.nz>
References: <48ff6e54-661e-9f1b-f6de-4bc4523ac35e@wirtel.be>
 <20190319142339.76cfdc47@fsol>
 <CAPTjJmo4DuhKNOo+qXaPjeWLqPQoTLFiPGerdnrPRY48NeTrhQ@mail.gmail.com>
 <20190319145127.0824f196@fsol> <5C916C71.5070102@canterbury.ac.nz>
 <20190320152231.7936863b@fsol> <5C931014.1000008@canterbury.ac.nz>
Message-ID: <CACL+1auvMxXCjajG2a5pVg8H3UgUzJZaVvO-qMvjTq72ueRE4Q@mail.gmail.com>

On 3/20/19, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Antoine Pitrou wrote:
>
>> How is it more secure than using mktemp()?
>
> It's not, but it solves the problem someone suggested of another
> program not being able to access and/or delete the file.

NamedTemporaryFile(delete=False) is more secure than naive use of
mktemp(). The file is created exclusively (O_EXCL). Another standard
user can't overwrite it. Nor can another standard user delete it if
it's created in the default temp directory (e.g. POSIX "/tmp" has the
sticky bit set). mkstemp() is similar but lacks the convenience and
reliable resource management of a Python file wrapper.

There's still the problem of accidental name collisions with other
processes that can access the file, i.e. processes running as the same
user or, in POSIX, processes running as the super user. I saw a
suggestion in this thread to increase the length of the random
sequence from 8 characters up to 22 characters in order to make this
problem extremely improbable.

From tjreedy at udel.edu  Thu Mar 21 16:37:41 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 21 Mar 2019 16:37:41 -0400
Subject: [Python-Dev] Is XML serialization output guaranteed to be
 bytewise identical forever?
In-Reply-To: <CACac1F8m5xxGBivvY-q-+dZtJ-YjvQ0Dto+hogkKSTUY6N=JKg@mail.gmail.com>
References: <D778964A-8B2B-4C1C-9328-2AA6DDB4D622@gmail.com>
 <CA+3bQGH4g7exNtvEYLUYMSuUeYhSeY4F7PjJUHKmf090dfyOkA@mail.gmail.com>
 <20190321122953.74d23952@fsol>
 <CAMofdRDNzxi8AjS_2BGSvRMH8jwtzQadFzv2TszThzf=p0_Zvg@mail.gmail.com>
 <CACac1F8m5xxGBivvY-q-+dZtJ-YjvQ0Dto+hogkKSTUY6N=JKg@mail.gmail.com>
Message-ID: <q70smm$1sgc$1@blaine.gmane.org>

On 3/21/2019 1:23 PM, Paul Moore wrote:
> On Thu, 21 Mar 2019 at 17:05, Steve Holden <steve at holdenweb.com> wrote:

>> Especially as the standards specifically say that ordering has no semantic impact.
>>
>> Byte-by-byte comparison of XML is almost always inappropriate.
> 
> Conversely, if ordering has no semantic impact, there's no real
> justification for asking for the current order to be changed.
> 
> In practice, allowing the user to control the ordering (by preserving
> input order) gives users a way of handling (according to the standard)
> broken consumers who ascribe semantic meaning to the attribute order.

Or, as Jonathan Goble said elsewhere, use an order that makes whatever 
sense to the author and other readers.  The order of positional 
parameter names in a function definition has no semantic meaning to 
python, but it would be terrible to make them be sorted.

-- 
Terry Jan Reedy


From songofacandy at gmail.com  Thu Mar 21 22:41:35 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Fri, 22 Mar 2019 11:41:35 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
Message-ID: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>

Hi, all.

I'm thinking about removing PendingDeprecationWarning.
(previous discussion:
https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038)

It was added "not be printed by default" version of DeprecationWarning.
But DeprecationWarning is not printed by default now.

We use PendingDeprecationWarning for N-2 release, and change it to
DeprecationWarning for N-1 release.  But this workflow seems not worth
enough for now.

I want to stop using PendingDeprecationWarning for new deprecation.

More aggressively, I want to remove PendingDeprecationWarning class,
and `PendingDeprecationWarning = DeprecationWarning` for backward
compatibility.

How do you think?  May I do it in Python 3.8?

-- 
Inada Naoki  <songofacandy at gmail.com>

From greg.ewing at canterbury.ac.nz  Fri Mar 22 03:31:33 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 22 Mar 2019 20:31:33 +1300
Subject: [Python-Dev] Replacement for array.array('u')?
Message-ID: <5C948F55.4040108@canterbury.ac.nz>

A poster on comp.lang.python is asking about array.array('u').
He wants an efficient mutable collection of unicode characters
that can be initialised from a string.

According to the docs, the 'u' code is deprecated and will be
removed in 4.0, but no alternative is suggested.

Why is this being deprecated, instead of keeping it and making
it always 32 bits? It seems like useful functionality that can't
be easily obtained another way.

-- 
Greg

From vstinner at redhat.com  Fri Mar 22 03:40:38 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Fri, 22 Mar 2019 08:40:38 +0100
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
Message-ID: <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>

Hi,

I agree to make PendingDeprecationWarning an alias to
DeprecationWarning. I never liked "PendingDeprecationWarning" name,
it's way too long to type :-D

Le ven. 22 mars 2019 ? 03:45, Inada Naoki <songofacandy at gmail.com> a ?crit :
> I want to stop using PendingDeprecationWarning for new deprecation.

I'm fine with that.

> More aggressively, I want to remove PendingDeprecationWarning class,
> and `PendingDeprecationWarning = DeprecationWarning` for backward
> compatibility.

I'm not sure that I understand well. Do you want to remove the
PendingDeprecationWarning builtin symbol, or just make it an alias to
DeprecationWarning.

I'm fine with "PendingDeprecationWarning = DeprecationWarning".

IMHO your email title is misleading. You don't want to *remove*
PendingDeprecationWarning, you only want to make it an alias to
DeprecationWarning, right? In term of backward compatibility, it's
very different :-)

Victor

From vstinner at redhat.com  Fri Mar 22 03:45:03 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Fri, 22 Mar 2019 08:45:03 +0100
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <5C948F55.4040108@canterbury.ac.nz>
References: <5C948F55.4040108@canterbury.ac.nz>
Message-ID: <CA+3bQGHq9H=idLedekjs1WkKxbt3uHunc7Tdtp3d1y2ht0L-yg@mail.gmail.com>

Hi,

Internally, CPython has a _PyUnicodeWriter which is an efficient way
to create a string but appending substrings or characters.
_PyUnicodeWriter changes the internal storage format depending on
characters code points (ascii or latin1: 1 byte/character, BMP: 2 b/c,
full UCS: 4 b/c). I tried once to expose it in Python, but I wasn't
convinced by performances. The overhead of method calls was quite
significant, and I wasn't convinced by "writer += str" performance
neither. Maybe I should try again. PyPy also has such object. It
avoids the "str += str" hack in ceval.c to avoid very poor performance
(_PyUnicodeWriter also uses overallocation which can be controlled
with multiple parameters to reduce the number of realloc).

Another alternative would be have to add a "strarray" type similar to
bytes/bytearray couple.

Is is what you are looking for? Or do you really need array.array API?

Victor

Le ven. 22 mars 2019 ? 08:38, Greg Ewing <greg.ewing at canterbury.ac.nz> a ?crit :
>
> A poster on comp.lang.python is asking about array.array('u').
> He wants an efficient mutable collection of unicode characters
> that can be initialised from a string.
>
> According to the docs, the 'u' code is deprecated and will be
> removed in 4.0, but no alternative is suggested.
>
> Why is this being deprecated, instead of keeping it and making
> it always 32 bits? It seems like useful functionality that can't
> be easily obtained another way.
>
> --
> Greg
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From songofacandy at gmail.com  Fri Mar 22 03:53:50 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Fri, 22 Mar 2019 16:53:50 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
Message-ID: <CAEfz+TxCp0Nvj0a_7wdi147maX0mLtJzAJXojWWuMDz2Db+f2w@mail.gmail.com>

On Fri, Mar 22, 2019 at 4:40 PM Victor Stinner <vstinner at redhat.com> wrote:
>
>
> > More aggressively, I want to remove PendingDeprecationWarning class,
> > and `PendingDeprecationWarning = DeprecationWarning` for backward
> > compatibility.
>
> I'm not sure that I understand well. Do you want to remove the
> PendingDeprecationWarning builtin symbol, or just make it an alias to
> DeprecationWarning.
>
> I'm fine with "PendingDeprecationWarning = DeprecationWarning".
>

I meant an alias in builtin.

> IMHO your email title is misleading. You don't want to *remove*
> PendingDeprecationWarning, you only want to make it an alias to
> DeprecationWarning, right? In term of backward compatibility, it's
> very different :-)
>
> Victor

Yes.  It will be removed at some point, but not in near future.

But when when backward compatibility can be kept by alias,
we can be very lazy about removing it.

We have `socket.error` for long time.

Regards,
-- 
Inada Naoki  <songofacandy at gmail.com>

From vstinner at redhat.com  Fri Mar 22 04:06:00 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Fri, 22 Mar 2019 09:06:00 +0100
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+TxCp0Nvj0a_7wdi147maX0mLtJzAJXojWWuMDz2Db+f2w@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAEfz+TxCp0Nvj0a_7wdi147maX0mLtJzAJXojWWuMDz2Db+f2w@mail.gmail.com>
Message-ID: <CA+3bQGE0az9JHEHihRq-6CXqRW1zF8ZWNikxJoNzYnV6qPTgvw@mail.gmail.com>

Le ven. 22 mars 2019 ? 08:54, Inada Naoki <songofacandy at gmail.com> a ?crit :
> Yes.  It will be removed at some point, but not in near future.
>
> But when when backward compatibility can be kept by alias,
> we can be very lazy about removing it.

Honestly, I don't see the point of removing PendingDeprecationWarning
anytime soon. Practicality beats purity. Python is no longer about the
purity of the language (we tried in Python 3.0, not sure that I want
to want to do that again ;-)), but making it convenient to use for
most people.

Removing PendingDeprecationWarning doesn't bring any value to anyone.
"PendingDeprecationWarning = DeprecationWarning" costs nothing in term
of maintenance.

If you care, I would suggest to invest time in static analyzers like
pycodestyle, pylint, pyflakes, etc. Emit a warning with a low
priority, and let users make their own choices.

> We have `socket.error` for long time.

And it's perfectly fine, no?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From songofacandy at gmail.com  Fri Mar 22 04:16:08 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Fri, 22 Mar 2019 17:16:08 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CA+3bQGE0az9JHEHihRq-6CXqRW1zF8ZWNikxJoNzYnV6qPTgvw@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAEfz+TxCp0Nvj0a_7wdi147maX0mLtJzAJXojWWuMDz2Db+f2w@mail.gmail.com>
 <CA+3bQGE0az9JHEHihRq-6CXqRW1zF8ZWNikxJoNzYnV6qPTgvw@mail.gmail.com>
Message-ID: <CAEfz+Tz+gtGUOKdp4G+cA32fwdpw6PewrUeFBU04CUO4=CMS2A@mail.gmail.com>

On Fri, Mar 22, 2019 at 5:06 PM Victor Stinner <vstinner at redhat.com> wrote:
>
> Le ven. 22 mars 2019 ? 08:54, Inada Naoki <songofacandy at gmail.com> a ?crit :
> > Yes.  It will be removed at some point, but not in near future.
> >
> > But when when backward compatibility can be kept by alias,
> > we can be very lazy about removing it.
>
> Honestly, I don't see the point of removing PendingDeprecationWarning
> anytime soon. Practicality beats purity.

I totally agree.

> Removing PendingDeprecationWarning doesn't bring any value to anyone.
> "PendingDeprecationWarning = DeprecationWarning" costs nothing in term
> of maintenance.
>
> If you care, I would suggest to invest time in static analyzers like
> pycodestyle, pylint, pyflakes, etc. Emit a warning with a low
> priority, and let users make their own choices.

Agree too.

>
> > We have `socket.error` for long time.
>
> And it's perfectly fine, no?
>

Yes.  Waiting 10+ years to remove aliases is fine.


-- 
Inada Naoki  <songofacandy at gmail.com>

From vstinner at redhat.com  Fri Mar 22 04:41:35 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Fri, 22 Mar 2019 09:41:35 +0100
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+Tz+gtGUOKdp4G+cA32fwdpw6PewrUeFBU04CUO4=CMS2A@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAEfz+TxCp0Nvj0a_7wdi147maX0mLtJzAJXojWWuMDz2Db+f2w@mail.gmail.com>
 <CA+3bQGE0az9JHEHihRq-6CXqRW1zF8ZWNikxJoNzYnV6qPTgvw@mail.gmail.com>
 <CAEfz+Tz+gtGUOKdp4G+cA32fwdpw6PewrUeFBU04CUO4=CMS2A@mail.gmail.com>
Message-ID: <CA+3bQGGTtLPcSaHZbHu5g-f+uPm5METL9Gdhs=yS2+CezY_Xpg@mail.gmail.com>

Le ven. 22 mars 2019 ? 09:16, Inada Naoki <songofacandy at gmail.com> a ?crit :
> > > We have `socket.error` for long time.
> >
> > And it's perfectly fine, no?
> >
>
> Yes.  Waiting 10+ years to remove aliases is fine.

Sure.

Let me elaborate my point of view on deprecation, since we are
discussing it here (and I know that my opinion is not fully shared by
all core devs, according to the bpo-35283 discussion, or maybe I'm
wrong?) :-)

Methods of threading.Thread changed their names to comply to PEP 8
coding style in Python 3: isAlive() has been renamed to is_alive().
The Threading.isAlive() method still exists in Python 3.8 and I think
that it's ok. Removing it immediately would go against the best
practice of writing a single code base working on Python 2 and
Python3... I would be very annoyed to have to replace a simple
"thread.isAlive()" call with something like
"six.threading_is_alive(thread)" in my code, just because someone
considered that the alias had to go away from the stdlib.

Last December, Serhiy started to talk about removing isAlive(): "It is
not even documented in Python 3, and can be removed in future."
https://bugs.python.org/issue35283#msg330242

Antoine Pitrou suggested that "If it's not already deprecated, I'd say
deprecate it first." And it has been done.

I'm fine with deprecating things, since it doesn't prevent an
application to be used. Use python3 -Wd (or python3 -X dev) to see
these warnings if you want to be pedantic, but honestly, keeping the
alias doesn't hurt anyone. Again, I mostly care about the maintenance
cost: isAlive() is not documented, it's just 8 lines in threading.py
and 2 lines in test_threading.py, and it's no like these lines require
a lot of maintenance.

IMHO the main metric should be to compare to cost to maintain such
alias vs the pain affecting *all* Python users if we remove it.

Right now, the maintenance cost is close to zero, whereas removing the
alias would annoy a lot of people who will suddenly no longer be able
to use their legacy code (written for Python 2 long time ago, but only
ported to Python 3 recently). Getting a hard AttributeError exception
is different than getting a silent DeprecationWarning :-)

--

Wait, I'm not against removing things from Python. Don't tell anyone,
but I *love* removing code, that's my greatest pleasure! IMHO Python
is way too big and is expensive to maintain. But we should carefully
discuss *each* feature removal, and plan properly a slow deprecation
period to make sure that users had enough time to upgrade (and maybe
extend it if needed).

For Threading.isAlive(), IMHO the fact that Python 2 is still alive is
simply a strong blocker issue. Let's discuss that that Python 2 usage
will be closer to 0,1% than 50% :-) (Sorry, I don't think that it's
really useful to discuss if Python 2 usage is more around 10% or 70%,
that's not really the point: I made up "50%" :-))

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From songofacandy at gmail.com  Fri Mar 22 05:11:20 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Fri, 22 Mar 2019 18:11:20 +0900
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <5C948F55.4040108@canterbury.ac.nz>
References: <5C948F55.4040108@canterbury.ac.nz>
Message-ID: <CAEfz+TzWZ-DdRQhs2EV2sUbNbBysDDwsRWZ0wtseCMdSYLMCdg@mail.gmail.com>

On Fri, Mar 22, 2019 at 4:38 PM Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> A poster on comp.lang.python is asking about array.array('u').
> He wants an efficient mutable collection of unicode characters
> that can be initialised from a string.
>
> According to the docs, the 'u' code is deprecated and will be
> removed in 4.0, but no alternative is suggested.
>
> Why is this being deprecated, instead of keeping it and making
> it always 32 bits? It seems like useful functionality that can't
> be easily obtained another way.
>

I think it's because there are not much use cases found
when implementing PEP 393.

If there are use cases enough to keep it in stdlib, I'm OK
about un-deprecate it and make it always 32bit (int32_t).

-- 
Inada Naoki  <songofacandy at gmail.com>

From steve at pearwood.info  Fri Mar 22 05:24:23 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 22 Mar 2019 20:24:23 +1100
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <5C948F55.4040108@canterbury.ac.nz>
References: <5C948F55.4040108@canterbury.ac.nz>
Message-ID: <20190322092423.GU12502@ando.pearwood.info>

On Fri, Mar 22, 2019 at 08:31:33PM +1300, Greg Ewing wrote:
> A poster on comp.lang.python is asking about array.array('u').
> He wants an efficient mutable collection of unicode characters
> that can be initialised from a string.
> 
> According to the docs, the 'u' code is deprecated and will be
> removed in 4.0, but no alternative is suggested.
> 
> Why is this being deprecated, instead of keeping it and making
> it always 32 bits? It seems like useful functionality that can't
> be easily obtained another way.

I can't answer any of those questions, but perhaps the poster can do 
this instead:

py> a = array('L', '????? ?????'.encode('utf-32be'))
py> a
array('L', [220266496, 807469056, 3791650816, 1963196416, 4278190080, 
536870912, 4194500608, 3036872704, 3288530944, 2969763840, 1107361792])

Getting the string out again is no harder:

py> bytes(a).decode('utf-32be')
'????? ?????'

But having said that, it would be nice to have an array code which 
treated the values as single UTF-32 characters:

array('?', ['?', '?', '?', '?', '?', ' ', '?', '?', '?', '?', '?'])

if for no other reason than it looks nicer than a bunch of 32 bit ints.


-- 
Steven

From steve at holdenweb.com  Fri Mar 22 05:30:07 2019
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 22 Mar 2019 09:30:07 +0000
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CA+3bQGGTtLPcSaHZbHu5g-f+uPm5METL9Gdhs=yS2+CezY_Xpg@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAEfz+TxCp0Nvj0a_7wdi147maX0mLtJzAJXojWWuMDz2Db+f2w@mail.gmail.com>
 <CA+3bQGE0az9JHEHihRq-6CXqRW1zF8ZWNikxJoNzYnV6qPTgvw@mail.gmail.com>
 <CAEfz+Tz+gtGUOKdp4G+cA32fwdpw6PewrUeFBU04CUO4=CMS2A@mail.gmail.com>
 <CA+3bQGGTtLPcSaHZbHu5g-f+uPm5METL9Gdhs=yS2+CezY_Xpg@mail.gmail.com>
Message-ID: <CAMofdRDDjDDtjn12BMKo2vUBrfbLOphEZYOZ3e0HwC3S05XVSQ@mail.gmail.com>

On Fri, Mar 22, 2019 at 8:43 AM Victor Stinner <vstinner at redhat.com> wrote:

> Le ven. 22 mars 2019 ? 09:16, Inada Naoki <songofacandy at gmail.com> a
> ?crit :
> > > > We have `socket.error` for long time.
> > >
> > > And it's perfectly fine, no?
> > >
> >
> > Yes.  Waiting 10+ years to remove aliases is fine.
>
> [...]

>
> Right now, the maintenance cost is close to zero, whereas removing the
> alias would annoy a lot of people who will suddenly no longer be able
> to use their legacy code (written for Python 2 long time ago, but only
> ported to Python 3 recently). Getting a hard AttributeError exception
> is different than getting a silent DeprecationWarning :-)
>

Quite. Why expend the effort on a development that will cause unnecessary
pain?

Generally speaking, end-users will live their lives in blissful ignorance
of [Pending]DeprecationWarning and should be allowed to do so. When a
developer wants to migrate them to more recent versions of Python there's a
chance that warnings will be enabled and examined, but it's by no means
certain.

I suspect the real value of the warnings is so that the dev team can shrug
their shoulders when someone complains a feature has gone missing after ten
years in deprecated status. This is the price of having them normally
silent, which is certainly worth any trouble it causes by refusing to
hassle innocent non-developers with warnings they can do nothing about.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190322/040e54e4/attachment.html>

From storchaka at gmail.com  Fri Mar 22 06:33:04 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 22 Mar 2019 12:33:04 +0200
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
Message-ID: <q72dl3$mbp$1@blaine.gmane.org>

22.03.19 04:41, Inada Naoki ????:
> I'm thinking about removing PendingDeprecationWarning.
> (previous discussion:
> https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038)
> 
> It was added "not be printed by default" version of DeprecationWarning.
> But DeprecationWarning is not printed by default now.
> 
> We use PendingDeprecationWarning for N-2 release, and change it to
> DeprecationWarning for N-1 release.  But this workflow seems not worth
> enough for now.
> 
> I want to stop using PendingDeprecationWarning for new deprecation.
> 
> More aggressively, I want to remove PendingDeprecationWarning class,
> and `PendingDeprecationWarning = DeprecationWarning` for backward
> compatibility.
> 
> How do you think?  May I do it in Python 3.8?

What is wrong with PendingDeprecationWarning? What problem do you want 
to solve at the cost of removing this feature?

Now, when DeprecationWarning is displayed by default in the interactive 
session, in __main__ and in development runtime mode (and this list can 
be extended), PendingDeprecationWarning is useful again. Even if the 
interpreter itself would not use it, it is used in third-party projects.


From solipsis at pitrou.net  Fri Mar 22 06:39:05 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 22 Mar 2019 11:39:05 +0100
Subject: [Python-Dev] Replacement for array.array('u')?
References: <5C948F55.4040108@canterbury.ac.nz>
Message-ID: <20190322113905.3fefed62@fsol>

On Fri, 22 Mar 2019 20:31:33 +1300
Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> A poster on comp.lang.python is asking about array.array('u').
> He wants an efficient mutable collection of unicode characters
> that can be initialised from a string.

TBH, I think anyone trying to use array.array should be directed to
Numpy these days.  The only reason for array.array being here is that
it predates Numpy.  Otherwise we'd never have added it.

Regards

Antoine.



From songofacandy at gmail.com  Fri Mar 22 06:48:21 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Fri, 22 Mar 2019 19:48:21 +0900
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <CAEfz+TzWZ-DdRQhs2EV2sUbNbBysDDwsRWZ0wtseCMdSYLMCdg@mail.gmail.com>
References: <5C948F55.4040108@canterbury.ac.nz>
 <CAEfz+TzWZ-DdRQhs2EV2sUbNbBysDDwsRWZ0wtseCMdSYLMCdg@mail.gmail.com>
Message-ID: <CAEfz+TwNukXY7mE=j42sg4JcBPc1baY10xGWFq7mO0bDOdUE+g@mail.gmail.com>

FYI, I have created issue on bugs.python.org about adding deprecation warning
for array('u').
https://bugs.python.org/issue36299

I created PR to change Py_UNICODE to Py_UCS4, instead of deprecate it.
https://github.com/python/cpython/pull/12497

Then, I found same change had made and reverted in the past.
https://github.com/python/cpython/commit/62bb394729a167a46d950954c4aed5f3ba7b8a69

The issue for the revert is this.
https://bugs.python.org/issue13072

-- 
Inada Naoki  <songofacandy at gmail.com>

From J.Demeyer at UGent.be  Fri Mar 22 06:51:38 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 22 Mar 2019 11:51:38 +0100
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <q72dl3$mbp$1@blaine.gmane.org>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <q72dl3$mbp$1@blaine.gmane.org>
Message-ID: <5C94BE3A.8010304@UGent.be>

On 2019-03-22 11:33, Serhiy Storchaka wrote:
> What is wrong with PendingDeprecationWarning?

It serves the same purpose as DeprecationWarning: it indicates that a 
feature is planned to be removed in the future. There is no point in 
having two warnings with exactly the same meaning.

> What problem do you want
> to solve at the cost of removing this feature?

1. Typically, a PendingDeprecationWarning is meant to be promoted to a 
DeprecationWarning in some future release. It takes a minor effort to do 
that and it may be forgotten. It's just simpler to use 
DeprecationWarning from the start.

2. Whenever somebody wants to deprecate something, that person has to 
decide between the two. That's just more complicated than it needs to be.

And I can easily ask the converse question: what problem do you want to 
solve by including that feature?

From songofacandy at gmail.com  Fri Mar 22 07:23:32 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Fri, 22 Mar 2019 20:23:32 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <q72dl3$mbp$1@blaine.gmane.org>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <q72dl3$mbp$1@blaine.gmane.org>
Message-ID: <CAEfz+TwkxoQB81ww37GE3USqS6TsUBz+B9Xo5Nsfs_Rju_YtXA@mail.gmail.com>

On Fri, Mar 22, 2019 at 7:36 PM Serhiy Storchaka <storchaka at gmail.com> wrote:
>
> >
> > How do you think?  May I do it in Python 3.8?
>
> What is wrong with PendingDeprecationWarning? What problem do you want
> to solve at the cost of removing this feature?
>

The main problem is complexity.  In other words, learning cost.

All Python library developer would like to know how to deprecate something.

They will understand "deprecated" means "will be removed in the future
version" easily.

Then, "will be deprecated" [1] is easy to understand?  "will be (will
be removed)"
is very curious.  To understand what PendingDeprecationWarning is for, they
need to learn history which is not documented in clearly.

[1]: https://docs.python.org/3/library/exceptions.html#PendingDeprecationWarning

If we deprecate PendingDeprecationWarning, people don't have to understand
what it is for.


> Now, when DeprecationWarning is displayed by default in the interactive
> session, in __main__ and in development runtime mode (and this list can
> be extended), PendingDeprecationWarning is useful again. Even if the
> interpreter itself would not use it, it is used in third-party projects.
>

This benefits seems too small compared to the learning cost.

-- 
Inada Naoki  <songofacandy at gmail.com>

From storchaka at gmail.com  Fri Mar 22 07:27:08 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 22 Mar 2019 13:27:08 +0200
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <5C948F55.4040108@canterbury.ac.nz>
References: <5C948F55.4040108@canterbury.ac.nz>
Message-ID: <q72gqf$81gc$1@blaine.gmane.org>

22.03.19 09:31, Greg Ewing ????:
> A poster on comp.lang.python is asking about array.array('u').
> He wants an efficient mutable collection of unicode characters
> that can be initialised from a string.
> 
> According to the docs, the 'u' code is deprecated and will be
> removed in 4.0, but no alternative is suggested.
> 
> Why is this being deprecated, instead of keeping it and making
> it always 32 bits? It seems like useful functionality that can't
> be easily obtained another way.

Making it always 32 bits would be compatibility breaking change. 
Currently array('u') represents the wchar_t string, and many API on 
Windows require it.

But we can add a new code, e.g. 'U', for UCS4.


From solipsis at pitrou.net  Fri Mar 22 07:33:01 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 22 Mar 2019 12:33:01 +0100
Subject: [Python-Dev] Replacement for array.array('u')?
References: <5C948F55.4040108@canterbury.ac.nz>
 <q72gqf$81gc$1@blaine.gmane.org>
Message-ID: <20190322123301.7489ae78@fsol>

On Fri, 22 Mar 2019 13:27:08 +0200
Serhiy Storchaka <storchaka at gmail.com> wrote:

> 22.03.19 09:31, Greg Ewing ????:
> > A poster on comp.lang.python is asking about array.array('u').
> > He wants an efficient mutable collection of unicode characters
> > that can be initialised from a string.
> > 
> > According to the docs, the 'u' code is deprecated and will be
> > removed in 4.0, but no alternative is suggested.
> > 
> > Why is this being deprecated, instead of keeping it and making
> > it always 32 bits? It seems like useful functionality that can't
> > be easily obtained another way.  
> 
> Making it always 32 bits would be compatibility breaking change. 
> Currently array('u') represents the wchar_t string, and many API on 
> Windows require it.

The question is: why would you use a array.array() with a Windows C API?

Regards

Antoine.



From storchaka at gmail.com  Fri Mar 22 07:31:48 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 22 Mar 2019 13:31:48 +0200
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <CA+3bQGHq9H=idLedekjs1WkKxbt3uHunc7Tdtp3d1y2ht0L-yg@mail.gmail.com>
References: <5C948F55.4040108@canterbury.ac.nz>
 <CA+3bQGHq9H=idLedekjs1WkKxbt3uHunc7Tdtp3d1y2ht0L-yg@mail.gmail.com>
Message-ID: <q72h38$g55$1@blaine.gmane.org>

22.03.19 09:45, Victor Stinner ????:
> Internally, CPython has a _PyUnicodeWriter which is an efficient way
> to create a string but appending substrings or characters.
> _PyUnicodeWriter changes the internal storage format depending on
> characters code points (ascii or latin1: 1 byte/character, BMP: 2 b/c,
> full UCS: 4 b/c). I tried once to expose it in Python, but I wasn't
> convinced by performances. The overhead of method calls was quite
> significant, and I wasn't convinced by "writer += str" performance
> neither. Maybe I should try again. PyPy also has such object. It
> avoids the "str += str" hack in ceval.c to avoid very poor performance
> (_PyUnicodeWriter also uses overallocation which can be controlled
> with multiple parameters to reduce the number of realloc).
> 
> Another alternative would be have to add a "strarray" type similar to
> bytes/bytearray couple.

Another alternative of mutable string buffer and string builder is 
io.StringIO.


From storchaka at gmail.com  Fri Mar 22 07:49:19 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 22 Mar 2019 13:49:19 +0200
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <5C94BE3A.8010304@UGent.be>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <q72dl3$mbp$1@blaine.gmane.org> <5C94BE3A.8010304@UGent.be>
Message-ID: <q72i43$32pg$1@blaine.gmane.org>

22.03.19 12:51, Jeroen Demeyer ????:
> On 2019-03-22 11:33, Serhiy Storchaka wrote:
>> What is wrong with PendingDeprecationWarning?
> 
> It serves the same purpose as DeprecationWarning: it indicates that a 
> feature is planned to be removed in the future. There is no point in 
> having two warnings with exactly the same meaning.

But does not flood the end user with warnings, so you can continue to 
use deprecated features for a while. This makes the transition more smooth.

>> What problem do you want
>> to solve at the cost of removing this feature?
> 
> 1. Typically, a PendingDeprecationWarning is meant to be promoted to a 
> DeprecationWarning in some future release. It takes a minor effort to do 
> that and it may be forgotten. It's just simpler to use 
> DeprecationWarning from the start.

What is wrong if it be forgotten?

Simpler does not mean better.

> 2. Whenever somebody wants to deprecate something, that person has to 
> decide between the two. That's just more complicated than it needs to be.

If he want to provide a more smooth transition way, he should start with 
a PendingDeprecationWarning. Otherwise he can use a DeprecationWarning.

> And I can easily ask the converse question: what problem do you want to 
> solve by including that feature?

It allowed to mark a feature deprecated for developers without harming 
the end users.


From stefan_ml at behnel.de  Fri Mar 22 07:51:49 2019
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 22 Mar 2019 12:51:49 +0100
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <20190322113905.3fefed62@fsol>
References: <5C948F55.4040108@canterbury.ac.nz> <20190322113905.3fefed62@fsol>
Message-ID: <q72i8l$3ded$1@blaine.gmane.org>

Antoine Pitrou schrieb am 22.03.19 um 11:39:
> On Fri, 22 Mar 2019 20:31:33 +1300 Greg Ewing wrote:
>> A poster on comp.lang.python is asking about array.array('u').
>> He wants an efficient mutable collection of unicode characters
>> that can be initialised from a string.
> 
> TBH, I think anyone trying to use array.array should be directed to
> Numpy these days.  The only reason for array.array being here is that
> it predates Numpy.  Otherwise we'd never have added it.

Well, maybe it wouldn't get *added* these days anymore, with pip+PyPI
nicely in place. But being there already, it makes for a nice and efficient
"batteries included" list replacement for simple data that would otherwise
waste a lot of object memory.

Stefan


From songofacandy at gmail.com  Fri Mar 22 07:52:58 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Fri, 22 Mar 2019 20:52:58 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <q72i43$32pg$1@blaine.gmane.org>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <q72dl3$mbp$1@blaine.gmane.org> <5C94BE3A.8010304@UGent.be>
 <q72i43$32pg$1@blaine.gmane.org>
Message-ID: <CAEfz+Tx1hMwUkL=7E6OQAaJ9i-scpSu5tyg7FG0i8Zu-YQzjJg@mail.gmail.com>

On Fri, Mar 22, 2019 at 8:49 PM Serhiy Storchaka <storchaka at gmail.com> wrote:
>
> > And I can easily ask the converse question: what problem do you want to
> > solve by including that feature?
>
> It allowed to mark a feature deprecated for developers without harming
> the end users.
>

Both of DeprecationWarning and PendingDeprecationWarning are not for end users.
Only FutureWarning is for end users now.

-- 
Inada Naoki  <songofacandy at gmail.com>

From storchaka at gmail.com  Fri Mar 22 08:01:54 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 22 Mar 2019 14:01:54 +0200
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+TwkxoQB81ww37GE3USqS6TsUBz+B9Xo5Nsfs_Rju_YtXA@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <q72dl3$mbp$1@blaine.gmane.org>
 <CAEfz+TwkxoQB81ww37GE3USqS6TsUBz+B9Xo5Nsfs_Rju_YtXA@mail.gmail.com>
Message-ID: <q72irl$4t79$1@blaine.gmane.org>

22.03.19 13:23, Inada Naoki ????:
> On Fri, Mar 22, 2019 at 7:36 PM Serhiy Storchaka <storchaka at gmail.com> wrote:
>> What is wrong with PendingDeprecationWarning? What problem do you want
>> to solve at the cost of removing this feature?
>>
> 
> The main problem is complexity.  In other words, learning cost.

Do you have evidences that many people have troubles with learning 
PendingDeprecationWarning?

> All Python library developer would like to know how to deprecate something.
> 
> They will understand "deprecated" means "will be removed in the future
> version" easily.
> 
> Then, "will be deprecated" [1] is easy to understand?  "will be (will
> be removed)"
> is very curious.  To understand what PendingDeprecationWarning is for, they
> need to learn history which is not documented in clearly.
> 
> [1]: https://docs.python.org/3/library/exceptions.html#PendingDeprecationWarning

Perhaps the better solution of this is to improve the documentation.

PendingDeprecationWarning means that you still can use the deprecated 
feature.

When we deprecate some feature, we should provide an alternate way to 
solve the same problem. And it is better if that way is available in few 
previous Python versions. So the developer which need to support several 
Python versions could just switch to a modern way. This is how we do, or 
at least must to do, to be polite with Python programmers.

PendingDeprecationWarning just give us time to add an alternate way if 
it is not available yet, and give Python programmers time to adapt their 
code.

> If we deprecate PendingDeprecationWarning, people don't have to understand
> what it is for.
> 
> 
>> Now, when DeprecationWarning is displayed by default in the interactive
>> session, in __main__ and in development runtime mode (and this list can
>> be extended), PendingDeprecationWarning is useful again. Even if the
>> interpreter itself would not use it, it is used in third-party projects.
>>
> 
> This benefits seems too small compared to the learning cost.

I do not think so.


From solipsis at pitrou.net  Fri Mar 22 08:50:04 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 22 Mar 2019 13:50:04 +0100
Subject: [Python-Dev] Replacement for array.array('u')?
References: <5C948F55.4040108@canterbury.ac.nz> <20190322113905.3fefed62@fsol>
 <q72i8l$3ded$1@blaine.gmane.org>
Message-ID: <20190322135004.2288dbd9@fsol>

On Fri, 22 Mar 2019 12:51:49 +0100
Stefan Behnel <stefan_ml at behnel.de> wrote:
> Antoine Pitrou schrieb am 22.03.19 um 11:39:
> > On Fri, 22 Mar 2019 20:31:33 +1300 Greg Ewing wrote:  
> >> A poster on comp.lang.python is asking about array.array('u').
> >> He wants an efficient mutable collection of unicode characters
> >> that can be initialised from a string.  
> > 
> > TBH, I think anyone trying to use array.array should be directed to
> > Numpy these days.  The only reason for array.array being here is that
> > it predates Numpy.  Otherwise we'd never have added it.  
> 
> Well, maybe it wouldn't get *added* these days anymore, with pip+PyPI
> nicely in place. But being there already, it makes for a nice and efficient
> "batteries included" list replacement for simple data that would otherwise
> waste a lot of object memory.

It's not really "batteries included".  array.array() supports almost no
useful operation. It's a bare-bones container for which you have to
implement every useful feature by yourself.

(yes, you can use generic mutable sequence algorithms such
as heapq or random.shuffle; how often do you need to heapify or shuffle
an array of unicode codepoints?)

Also, when using a unicode array, there's no substantial win of memory
compared to a single str object.  You may be losing some actually,
because of the flexible str representation.

Regards

Antoine.



From storchaka at gmail.com  Fri Mar 22 10:11:45 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 22 Mar 2019 16:11:45 +0200
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <20190322123301.7489ae78@fsol>
References: <5C948F55.4040108@canterbury.ac.nz>
 <q72gqf$81gc$1@blaine.gmane.org> <20190322123301.7489ae78@fsol>
Message-ID: <q72qf4$6nd4$1@blaine.gmane.org>

22.03.19 13:33, Antoine Pitrou ????:
> On Fri, 22 Mar 2019 13:27:08 +0200
> Serhiy Storchaka <storchaka at gmail.com> wrote:
>> Making it always 32 bits would be compatibility breaking change.
>> Currently array('u') represents the wchar_t string, and many API on
>> Windows require it.
> 
> The question is: why would you use a array.array() with a Windows C API?

I do not. But maybe it is used. And changing the width of the 'u' code 
would break such use case.


From solipsis at pitrou.net  Fri Mar 22 10:49:14 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 22 Mar 2019 15:49:14 +0100
Subject: [Python-Dev] Replacement for array.array('u')?
References: <5C948F55.4040108@canterbury.ac.nz>
 <q72gqf$81gc$1@blaine.gmane.org> <20190322123301.7489ae78@fsol>
 <q72qf4$6nd4$1@blaine.gmane.org>
Message-ID: <20190322154914.08dd503b@fsol>

On Fri, 22 Mar 2019 16:11:45 +0200
Serhiy Storchaka <storchaka at gmail.com> wrote:
> 22.03.19 13:33, Antoine Pitrou ????:
> > On Fri, 22 Mar 2019 13:27:08 +0200
> > Serhiy Storchaka <storchaka at gmail.com> wrote:  
> >> Making it always 32 bits would be compatibility breaking change.
> >> Currently array('u') represents the wchar_t string, and many API on
> >> Windows require it.  
> > 
> > The question is: why would you use a array.array() with a Windows C API?  
> 
> I do not. But maybe it is used. And changing the width of the 'u' code 
> would break such use case.

Right.  But changing the width is not what I had in mind.  Just keep it
identical until we finally decide to remove it.

Regards

Antoine.



From vstinner at redhat.com  Fri Mar 22 12:08:08 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Fri, 22 Mar 2019 17:08:08 +0100
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <q72irl$4t79$1@blaine.gmane.org>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <q72dl3$mbp$1@blaine.gmane.org>
 <CAEfz+TwkxoQB81ww37GE3USqS6TsUBz+B9Xo5Nsfs_Rju_YtXA@mail.gmail.com>
 <q72irl$4t79$1@blaine.gmane.org>
Message-ID: <CA+3bQGFKccWMD=y=3koi8FMXh_1FY-DDmmzz8NBmjn-1EAMtfw@mail.gmail.com>

Le ven. 22 mars 2019 ? 13:05, Serhiy Storchaka <storchaka at gmail.com> a ?crit :
> > The main problem is complexity.  In other words, learning cost.
>
> Do you have evidences that many people have troubles with learning
> PendingDeprecationWarning?

I have no idea when I should use PendingDeprecationWarning rather than
DeprecationWarning. For example, let's say that we deprecate a feature
in Python N with an open question of remove it from Python N+1 or N+2.
Should I started with PendingDeprecationWarning?

The following discussion mentioned that many deprecated features are
not removed just because everybody forgot to remove it:
https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038

And I consider that it's perfectly fine to not have a strict plan to
remove a feature as part of a deprecation. The usage of a deprecated
feature can evolve. There are a few exceptions of deprecated features
which were only modified to remove the deprecation, rather than
removing the feature, because we decided that the fetaure must stay.
Python language is living, its usage is changing frequently.

About showing warnings: I never ever used a specific filter to only
hide PendingDeprecationWarning but show DeprecationWarning. When I
want to fix warnings, I want to fix all of them, pending or not. I
would like to prepare a project for all future Python versions, not
only fix DeprecationWarning.

"python3 -X dev", "python3 -Wd" and python3 compiled in debug modes
all show DeprecationWarning *and* PendingDeprecationWarning. The
granularity no longer matters.

It was useful to have PendingDeprecationWarning when it was hidden by
default, whereas DeprecationWarning warnings were displayed by
default. But now both are hidden by default.

Well, maybe read the whole previous discussion for the full rationale:
https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038



> Perhaps the better solution of this is to improve the documentation.
>
> PendingDeprecationWarning means that you still can use the deprecated
> feature.

Honestly, I'm not really excited by using a feature that is tagged to
be deprecated in the future... IMHO the difference between
PendingDeprecationWarning and DeprecationWarning is too subtle to be
useful in practice.


> PendingDeprecationWarning just give us time to add an alternate way if
> it is not available yet, and give Python programmers time to adapt their
> code.

Do you have some examples?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From steve.dower at python.org  Fri Mar 22 12:09:40 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 22 Mar 2019 09:09:40 -0700
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <20190322123301.7489ae78@fsol>
References: <5C948F55.4040108@canterbury.ac.nz>
 <q72gqf$81gc$1@blaine.gmane.org> <20190322123301.7489ae78@fsol>
Message-ID: <543c3748-42d9-44bb-d18f-fe8e3c134584@python.org>

On 22Mar2019 0433, Antoine Pitrou wrote:
> The question is: why would you use a array.array() with a Windows C API?

I started replying to this with a whole lot of examples, and eventually 
convinced myself that you wouldn't (or shouldn't).

That said, I see value in having a type for array/struct/memoryview that 
"is the same size as wchar_t", since that will avoid people having to 
guess (the docs for array in particular are deliberately vague about the 
actual size of the various types).

This is not the same as "UCS-4" - that's a very Linux-centric point of 
view. Decoupling it from Py_UNICODE is fine though, since that type will 
have no meaning eventually. But the PyUnicode_*WideChar APIs are not 
going away, which means that wchar_t still exists and has to have a 
known size at compile time.

Cheers,
Steve

From brett at python.org  Fri Mar 22 13:25:46 2019
From: brett at python.org (Brett Cannon)
Date: Fri, 22 Mar 2019 10:25:46 -0700
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
Message-ID: <CAP1=2W6uE+YXTg8j4o+3z8osipdOAd+rkCZx2QZ-hKmM5A4tDw@mail.gmail.com>

On Fri, Mar 22, 2019 at 12:41 AM Victor Stinner <vstinner at redhat.com> wrote:

> Hi,
>
> I agree to make PendingDeprecationWarning an alias to
> DeprecationWarning. I never liked "PendingDeprecationWarning" name,
> it's way too long to type :-D
>
> Le ven. 22 mars 2019 ? 03:45, Inada Naoki <songofacandy at gmail.com> a
> ?crit :
> > I want to stop using PendingDeprecationWarning for new deprecation.
>
> I'm fine with that.
>
> > More aggressively, I want to remove PendingDeprecationWarning class,
> > and `PendingDeprecationWarning = DeprecationWarning` for backward
> > compatibility.
>
> I'm not sure that I understand well. Do you want to remove the
> PendingDeprecationWarning builtin symbol, or just make it an alias to
> DeprecationWarning.
>
> I'm fine with "PendingDeprecationWarning = DeprecationWarning".
>

We can't do that as it will break code. Think of code which is having
warnings raise exceptions and that are purposefully catching
PendingDeprecationWarning but not DeprecationWarning; this change would
break that. These classes are part of the public API of the warnings module
and so we shouldn't change semantics like that for people who have a
specific use for those two different classes regardless of how the stdlib
may choose to use them.


>
> IMHO your email title is misleading. You don't want to *remove*
> PendingDeprecationWarning, you only want to make it an alias to
> DeprecationWarning, right? In term of backward compatibility, it's
> very different :-)
>

If you want to remove PendingDeprecationWarning that's a discussion we can
obviously have (which I disagree with as shown in the discuss.python.org
discussion), but I think aliasing is a non-starter.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190322/cfb3d616/attachment.html>

From songofacandy at gmail.com  Fri Mar 22 13:37:11 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Sat, 23 Mar 2019 02:37:11 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAP1=2W6uE+YXTg8j4o+3z8osipdOAd+rkCZx2QZ-hKmM5A4tDw@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAP1=2W6uE+YXTg8j4o+3z8osipdOAd+rkCZx2QZ-hKmM5A4tDw@mail.gmail.com>
Message-ID: <CAEfz+Tzqnrx3Jfcf8hCia4DBCV4hVWxpHhEW8-f8LsfYps1uyQ@mail.gmail.com>

On Sat, Mar 23, 2019 at 2:26 AM Brett Cannon <brett at python.org> wrote:
>
>
> We can't do that as it will break code. Think of code which is having warnings raise exceptions and that are purposefully catching PendingDeprecationWarning but not DeprecationWarning; this change would break that. These classes are part of the public API of the warnings module and so we shouldn't change semantics like that for people who have a specific use for those two different classes regardless of how the stdlib may choose to use them.
>

Didn't we already do it?  socket.error was independent error class.
It became alias of OSError from Python 3.3.

There might be some small troubles.  But it was small enough for
Python minor versions, I think.


-- 
Inada Naoki  <songofacandy at gmail.com>

From brett at python.org  Fri Mar 22 14:01:44 2019
From: brett at python.org (Brett Cannon)
Date: Fri, 22 Mar 2019 11:01:44 -0700
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+Tzqnrx3Jfcf8hCia4DBCV4hVWxpHhEW8-f8LsfYps1uyQ@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAP1=2W6uE+YXTg8j4o+3z8osipdOAd+rkCZx2QZ-hKmM5A4tDw@mail.gmail.com>
 <CAEfz+Tzqnrx3Jfcf8hCia4DBCV4hVWxpHhEW8-f8LsfYps1uyQ@mail.gmail.com>
Message-ID: <CAP1=2W4+WckF3FmM1RjX7DtbU1HN5V=WqnJawXCeV+evus0Wfw@mail.gmail.com>

On Fri, Mar 22, 2019 at 10:37 AM Inada Naoki <songofacandy at gmail.com> wrote:

> On Sat, Mar 23, 2019 at 2:26 AM Brett Cannon <brett at python.org> wrote:
> >
> >
> > We can't do that as it will break code. Think of code which is having
> warnings raise exceptions and that are purposefully catching
> PendingDeprecationWarning but not DeprecationWarning; this change would
> break that. These classes are part of the public API of the warnings module
> and so we shouldn't change semantics like that for people who have a
> specific use for those two different classes regardless of how the stdlib
> may choose to use them.
> >
>
> Didn't we already do it?  socket.error was independent error class.
> It became alias of OSError from Python 3.3.
>

I would argue that makes more sense semantically than what you're proposing
here as socket.error is a type of OSError while you're suddenly making
PendingDeprecationWarning a "stronger" exception than it was.


>
> There might be some small troubles.  But it was small enough for
> Python minor versions, I think.
>

 I don't think it's worth the cost to users. We can just choose to stop
using it in the stdlib and not use PendingDeprecationWarning. And if people
want to force others to define their own PendingDeprecationWarning by
deprecating that's fine, but the aliasing where it could cause unintended
exception swallowing for something related to breaking changes seems
unnecessarily risky to me simply because we don't want to ask users to
update their code in a backwards-compatible fashion.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190322/e0c2af34/attachment.html>

From status at bugs.python.org  Fri Mar 22 14:08:13 2019
From: status at bugs.python.org (Python tracker)
Date: Fri, 22 Mar 2019 18:08:13 +0000 (UTC)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190322180813.555CF52BEA9@bugs.ams1.psf.io>


ACTIVITY SUMMARY (2019-03-15 - 2019-03-22)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    7051 ( +3)
  closed 41100 (+91)
  total  48151 (+94)

Open issues with patches: 2817 


Issues opened (55)
==================

#35121: Cookie domain check returns incorrect results
https://bugs.python.org/issue35121  reopened by xtreak

#36309: Remove tempfile.mktemp()
https://bugs.python.org/issue36309  opened by John Hagen

#36310: pygettext3.7 Does Not Recognize gettext Calls Within fstrings
https://bugs.python.org/issue36310  opened by Allie Fitter

#36311: Flaw in Windows code page decoder for large input
https://bugs.python.org/issue36311  opened by serhiy.storchaka

#36313: error: [Errno 13] Permission denied: '/usr/local/lib/python3.7
https://bugs.python.org/issue36313  opened by bunhin

#36315: Unable to install Python 3.7.2
https://bugs.python.org/issue36315  opened by ecesujoy at gmail.com

#36318: Adding support for setting the "disabled" attribute of loggers
https://bugs.python.org/issue36318  opened by maggyero

#36319: Erro 0xC0000374 on windows 10
https://bugs.python.org/issue36319  opened by heckad

#36322: Argument typo in dbm.ndbm.open
https://bugs.python.org/issue36322  opened by rougeth

#36323: IDLE: always display full grep path
https://bugs.python.org/issue36323  opened by terry.reedy

#36326: Build-out help() to read __slots__ with an optional data dicti
https://bugs.python.org/issue36326  opened by rhettinger

#36329: use the right python "make -C Doc/ serve"
https://bugs.python.org/issue36329  opened by matrixise

#36330: Warning about a potential dead code in timemodule and Clang
https://bugs.python.org/issue36330  opened by matrixise

#36335: Factor out / add bdb.Bdb.is_skipped_frame
https://bugs.python.org/issue36335  opened by blueyed

#36338: urlparse of urllib returns wrong hostname
https://bugs.python.org/issue36338  opened by Xianbo Wang

#36339: test_ttk_guionly: test_traversal() fails randomly on AMD64 Win
https://bugs.python.org/issue36339  opened by vstinner

#36340: 3.7.3rc1 Install Certificates fails on macOS
https://bugs.python.org/issue36340  opened by Dima.Tisnek

#36341: bind() on AF_UNIX socket may fail in tests run as non-root
https://bugs.python.org/issue36341  opened by xdegaye

#36342: test_venv failure when executed by test_multiprocessing and th
https://bugs.python.org/issue36342  opened by xdegaye

#36345: Deprecate Tools/scripts/serve.py in favour of python -m http.s
https://bugs.python.org/issue36345  opened by matrixise

#36346: Prepare for removing the legacy Unicode C API
https://bugs.python.org/issue36346  opened by serhiy.storchaka

#36347: Renaming the constants for the .flags of PyMemberDef
https://bugs.python.org/issue36347  opened by matrixise

#36348: test_imaplib.RemoteIMAP_STARTTLSTest.test_logout() fails rando
https://bugs.python.org/issue36348  opened by vstinner

#36350: inspect.Signature.parameters and inspect.BoundArguments.argume
https://bugs.python.org/issue36350  opened by remi.lapeyre

#36351: the ipv6type variable in configure.ac may be set incorrectly w
https://bugs.python.org/issue36351  opened by xdegaye

#36353: rpath incorrectly handled on OSX by build_ext
https://bugs.python.org/issue36353  opened by Toon Verstraelen

#36354: Use CreateProcessW for Python 2.7 on Windows.
https://bugs.python.org/issue36354  opened by Ray Donnelly

#36355: Remove documentation and internal use of the *RESTRICTED const
https://bugs.python.org/issue36355  opened by josh.r

#36356: Failure to build with address sanitizer
https://bugs.python.org/issue36356  opened by btharper

#36359: contextvars documentation unclear on thread-locality
https://bugs.python.org/issue36359  opened by Hameer Abbasi2

#36361: generate correct pyconfig.h when cross-compiling
https://bugs.python.org/issue36361  opened by xdegaye

#36364: errors in multiprocessing.shared_memory examples
https://bugs.python.org/issue36364  opened by pierreglaser

#36366: Patcher stop method should be idempotent
https://bugs.python.org/issue36366  opened by sfreilich

#36368: server process of shared_memory shuts down if KeyboardInterrup
https://bugs.python.org/issue36368  opened by pierreglaser

#36369: test_weakref super slow on RPi Zero
https://bugs.python.org/issue36369  opened by DNSGeek

#36370: Check for PyErr_Occurred() after PyImport_GetModule().
https://bugs.python.org/issue36370  opened by asmeurer

#36372: Flawed handling of URL redirect
https://bugs.python.org/issue36372  opened by bugburger

#36373: asyncio.gather: no docs for deprecated loop parameter
https://bugs.python.org/issue36373  opened by dtrauma

#36375: PEP 499 implementation: "python -m foo" binds the main module 
https://bugs.python.org/issue36375  opened by cameron

#36377: Python 'datastructures.html' docs page needs improvement becau
https://bugs.python.org/issue36377  opened by shoark7

#36378: Add support to load from paths to json.load
https://bugs.python.org/issue36378  opened by DaniFojo

#36379: nb_inplace_pow is always called with an invalid argument
https://bugs.python.org/issue36379  opened by Zuzu_Typ

#36381: Deprecate "#" argument format without PY_SSIZE_T_CLEAN
https://bugs.python.org/issue36381  opened by inada.naoki

#36383: In Windows 10 virtual environments distutils.sysconfig.get_pyt
https://bugs.python.org/issue36383  opened by Mark Campanelli

#36384: ipaddress Should not reject IPv4 addresses with leading zeroes
https://bugs.python.org/issue36384  opened by Joel Croteau

#36387: Refactor getenvironment() in _winapi.c
https://bugs.python.org/issue36387  opened by serhiy.storchaka

#36388: pdb: do_debug installs sys.settrace handler when used inside p
https://bugs.python.org/issue36388  opened by blueyed

#36389: Add gc.enable_object_debugger(): detect corrupted Python objec
https://bugs.python.org/issue36389  opened by vstinner

#36390: IDLE: Refactor formatting methods from editor
https://bugs.python.org/issue36390  opened by cheryl.sabella

#36392: IPv4Interface Object has no attributte prefixlen
https://bugs.python.org/issue36392  opened by Eddgar Rojas

#36394: test_multiprocessing_spawn fails on Windows7 3.x buildbot
https://bugs.python.org/issue36394  opened by pablogsal

#36395: Add deferred single-threaded/fake executor to concurrent.futur
https://bugs.python.org/issue36395  opened by Brian McCutchon

#36396: Remove fgBg param of idlelib.config.GetHighlight()
https://bugs.python.org/issue36396  opened by terry.reedy

#36398: A possible crash in structseq.c's structseq_repr()
https://bugs.python.org/issue36398  opened by ZackerySpytz

#36399: multiprocessing: Queue does not work in virtualenv but works f
https://bugs.python.org/issue36399  opened by Bernt.R??skar.Brenna



Most recent 15 issues with no replies (15)
==========================================

#36395: Add deferred single-threaded/fake executor to concurrent.futur
https://bugs.python.org/issue36395

#36394: test_multiprocessing_spawn fails on Windows7 3.x buildbot
https://bugs.python.org/issue36394

#36392: IPv4Interface Object has no attributte prefixlen
https://bugs.python.org/issue36392

#36388: pdb: do_debug installs sys.settrace handler when used inside p
https://bugs.python.org/issue36388

#36387: Refactor getenvironment() in _winapi.c
https://bugs.python.org/issue36387

#36384: ipaddress Should not reject IPv4 addresses with leading zeroes
https://bugs.python.org/issue36384

#36383: In Windows 10 virtual environments distutils.sysconfig.get_pyt
https://bugs.python.org/issue36383

#36369: test_weakref super slow on RPi Zero
https://bugs.python.org/issue36369

#36364: errors in multiprocessing.shared_memory examples
https://bugs.python.org/issue36364

#36361: generate correct pyconfig.h when cross-compiling
https://bugs.python.org/issue36361

#36359: contextvars documentation unclear on thread-locality
https://bugs.python.org/issue36359

#36353: rpath incorrectly handled on OSX by build_ext
https://bugs.python.org/issue36353

#36351: the ipv6type variable in configure.ac may be set incorrectly w
https://bugs.python.org/issue36351

#36350: inspect.Signature.parameters and inspect.BoundArguments.argume
https://bugs.python.org/issue36350

#36341: bind() on AF_UNIX socket may fail in tests run as non-root
https://bugs.python.org/issue36341



Most recent 15 issues waiting for review (15)
=============================================

#36398: A possible crash in structseq.c's structseq_repr()
https://bugs.python.org/issue36398

#36396: Remove fgBg param of idlelib.config.GetHighlight()
https://bugs.python.org/issue36396

#36394: test_multiprocessing_spawn fails on Windows7 3.x buildbot
https://bugs.python.org/issue36394

#36390: IDLE: Refactor formatting methods from editor
https://bugs.python.org/issue36390

#36389: Add gc.enable_object_debugger(): detect corrupted Python objec
https://bugs.python.org/issue36389

#36388: pdb: do_debug installs sys.settrace handler when used inside p
https://bugs.python.org/issue36388

#36387: Refactor getenvironment() in _winapi.c
https://bugs.python.org/issue36387

#36381: Deprecate "#" argument format without PY_SSIZE_T_CLEAN
https://bugs.python.org/issue36381

#36378: Add support to load from paths to json.load
https://bugs.python.org/issue36378

#36377: Python 'datastructures.html' docs page needs improvement becau
https://bugs.python.org/issue36377

#36375: PEP 499 implementation: "python -m foo" binds the main module 
https://bugs.python.org/issue36375

#36368: server process of shared_memory shuts down if KeyboardInterrup
https://bugs.python.org/issue36368

#36366: Patcher stop method should be idempotent
https://bugs.python.org/issue36366

#36364: errors in multiprocessing.shared_memory examples
https://bugs.python.org/issue36364

#36361: generate correct pyconfig.h when cross-compiling
https://bugs.python.org/issue36361



Top 10 most discussed issues (10)
=================================

#34160: ElementTree not preserving attribute order
https://bugs.python.org/issue34160  23 msgs

#36236: Python crash on macOS when CWD is invalid
https://bugs.python.org/issue36236  13 msgs

#36356: Failure to build with address sanitizer
https://bugs.python.org/issue36356  11 msgs

#36346: Prepare for removing the legacy Unicode C API
https://bugs.python.org/issue36346  10 msgs

#36370: Check for PyErr_Occurred() after PyImport_GetModule().
https://bugs.python.org/issue36370   9 msgs

#36299: array: Deprecate 'u' type in array module
https://bugs.python.org/issue36299   8 msgs

#35066: Inconsistency between dangling '%' handling in time.strftime()
https://bugs.python.org/issue35066   6 msgs

#36342: test_venv failure when executed by test_multiprocessing and th
https://bugs.python.org/issue36342   6 msgs

#36389: Add gc.enable_object_debugger(): detect corrupted Python objec
https://bugs.python.org/issue36389   6 msgs

#23984: Documentation error: Descriptors
https://bugs.python.org/issue23984   5 msgs



Issues closed (83)
==================

#4071: ntpath.abspath fails for long str paths
https://bugs.python.org/issue4071  closed by eryksun

#7936: sys.argv contains only scriptname
https://bugs.python.org/issue7936  closed by terry.reedy

#8677: Modules needing PY_SSIZE_T_CLEAN
https://bugs.python.org/issue8677  closed by inada.naoki

#10641: kill_python sometimes fails to kill processes on buildbots
https://bugs.python.org/issue10641  closed by zach.ware

#12616: zip fixer fails on zip()[:-1]
https://bugs.python.org/issue12616  closed by xtreak

#12622: failfast argument to TextTestRunner not documented
https://bugs.python.org/issue12622  closed by cheryl.sabella

#13668: mute ImportError in __del__ of _threading_local module
https://bugs.python.org/issue13668  closed by xtreak

#18016: subprocess should open stdin in mode w+b on windows
https://bugs.python.org/issue18016  closed by cheryl.sabella

#18249: Incorrect and incomplete help docs for close() method
https://bugs.python.org/issue18249  closed by adelfino

#19415: test_gdb fails when using --without-doc-strings on Fedora 19
https://bugs.python.org/issue19415  closed by matrixise

#21960: Better path handling in Idle find in files
https://bugs.python.org/issue21960  closed by cheryl.sabella

#22587: os.path.abspath(None) behavior is inconsistent between platfor
https://bugs.python.org/issue22587  closed by SilentGhost

#23216: IDLE grep/find/replace source code needs docstrings
https://bugs.python.org/issue23216  closed by cheryl.sabella

#26103: Contradiction in definition of "data descriptor" between (dott
https://bugs.python.org/issue26103  closed by Aaron Hall

#26517: Crash in installer
https://bugs.python.org/issue26517  closed by cheryl.sabella

#30903: IPv4Network's hostmask attribute doesn't returns string value 
https://bugs.python.org/issue30903  closed by cheryl.sabella

#31470: Py_Initialize documentation wrong
https://bugs.python.org/issue31470  closed by cheryl.sabella

#31495: Wrong offset with IndentationError ("expected an indented bloc
https://bugs.python.org/issue31495  closed by cheryl.sabella

#32142: heapq.heappop - documentation misleading or doesn't work
https://bugs.python.org/issue32142  closed by cheryl.sabella

#32147: improve performance of binascii.unhexlify() by using conversio
https://bugs.python.org/issue32147  closed by serhiy.storchaka

#33635: OSError when using pathlib.Path.rglob() to list device files
https://bugs.python.org/issue33635  closed by cheryl.sabella

#34461: Availability of parsers in etree initializer
https://bugs.python.org/issue34461  closed by ned.deily

#34965: Python on Docker container using flask is going down after som
https://bugs.python.org/issue34965  closed by cheryl.sabella

#35284: Incomplete error handling in Python/compile.c:compiler_call()
https://bugs.python.org/issue35284  closed by serhiy.storchaka

#35388: _PyRuntime_Initialize() called after Py_Finalize() does nothin
https://bugs.python.org/issue35388  closed by vstinner

#35493: multiprocessing.Pool._worker_handler(): use SIGCHLD to be noti
https://bugs.python.org/issue35493  closed by pablogsal

#35715: ProcessPool workers hold onto return value of last task in mem
https://bugs.python.org/issue35715  closed by pablogsal

#35752: test_buffer fails on ppc64le: memoryview pack_single() is misc
https://bugs.python.org/issue35752  closed by vstinner

#35978: test_venv fails in Travis with GCC
https://bugs.python.org/issue35978  closed by steve.dower

#36124: Provide convenient C API for storing per-interpreter state
https://bugs.python.org/issue36124  closed by eric.snow

#36195: initializer is not a valid param in ThreadPoolExecutor
https://bugs.python.org/issue36195  closed by ned.deily

#36211: show full url when execute "make -C Doc/ serve"
https://bugs.python.org/issue36211  closed by mdk

#36215: Should AppVeyor run compile Python in debug mode?
https://bugs.python.org/issue36215  closed by vstinner

#36233: xml ElementTree quotation marks of xml version string
https://bugs.python.org/issue36233  closed by serhiy.storchaka

#36235: distutils.sysconfig.customize_compiler() overrides CFLAGS var 
https://bugs.python.org/issue36235  closed by vstinner

#36256: parser module fails on legal input
https://bugs.python.org/issue36256  closed by pablogsal

#36268: Change default tar format to modern POSIX 2001 (pax) for bette
https://bugs.python.org/issue36268  closed by serhiy.storchaka

#36272: Recursive logging crashes Interpreter in Python 3
https://bugs.python.org/issue36272  closed by vinay.sajip

#36276: Python urllib CRLF injection vulnerability
https://bugs.python.org/issue36276  closed by orsenthil

#36292: Coverity scan: Resource leaks in longobject.c
https://bugs.python.org/issue36292  closed by vstinner

#36297: Remove unicode_internal codec
https://bugs.python.org/issue36297  closed by inada.naoki

#36300: eval of comprehension cannot access local variables
https://bugs.python.org/issue36300  closed by terry.reedy

#36307: Upgrade Travis CI config to Xenial from near-EoL Trusty and re
https://bugs.python.org/issue36307  closed by inada.naoki

#36308: Fix warning in _PyPathConfig_ComputeArgv0
https://bugs.python.org/issue36308  closed by matrixise

#36312: Invalid flag for some code page decoders
https://bugs.python.org/issue36312  closed by serhiy.storchaka

#36314: Pivot_Table Docstring Error
https://bugs.python.org/issue36314  closed by SilentGhost

#36316: Provide SHA256 checksums for installers
https://bugs.python.org/issue36316  closed by benjamin.peterson

#36317: Typo in documentation of _PyObject_FastCallDict
https://bugs.python.org/issue36317  closed by vstinner

#36320: typing.NamedTuple to switch from OrderedDict to regular dict
https://bugs.python.org/issue36320  closed by rhettinger

#36321: Fix misspelled attribute name in namedtuple()
https://bugs.python.org/issue36321  closed by rhettinger

#36324: Inverse cumulative normal distribution function
https://bugs.python.org/issue36324  closed by rhettinger

#36325: Build-out help() to support a class level data dictionary
https://bugs.python.org/issue36325  closed by rhettinger

#36327: Remove EOLed Py34 from "Status of Python branches"
https://bugs.python.org/issue36327  closed by xtreak

#36328: tstate may be used uninitialized in Py_NewInterpreter
https://bugs.python.org/issue36328  closed by vstinner

#36331: check_output is returning non-zero exit status 1
https://bugs.python.org/issue36331  closed by ned.deily

#36332: compile() error on AST object with assignment expression
https://bugs.python.org/issue36332  closed by pablogsal

#36333: memory leaks detected with valgrind for python -V
https://bugs.python.org/issue36333  closed by vstinner

#36334: pathlib.Path  unexpected (wrong) behaviour when combining path
https://bugs.python.org/issue36334  closed by eryksun

#36336: test_httplib, test_multiprocessing_forkserver leaks on x86 Gen
https://bugs.python.org/issue36336  closed by vstinner

#36337: Use socket.sendall()/send() send data larger than 2GB will be 
https://bugs.python.org/issue36337  closed by vstinner

#36343: Certificate added to Win Store not available
https://bugs.python.org/issue36343  closed by steve.dower

#36344: install_certificates.command too complicated,  copy from pip's
https://bugs.python.org/issue36344  closed by ned.deily

#36349: Including parentheses in a regular expression pattern enclosed
https://bugs.python.org/issue36349  closed by matrixise

#36352: Modules/getpath.c should not use hardcoded buffer sizes (MAXPA
https://bugs.python.org/issue36352  closed by vstinner

#36357: Build 32bit Python on Windows with SSE2 instruction set
https://bugs.python.org/issue36357  closed by Ma Lin

#36358: bool type does not support True or False as command line argv
https://bugs.python.org/issue36358  closed by serhiy.storchaka

#36360: undef HAVE_STROPTS_H in pyconfig.h.in is ignored
https://bugs.python.org/issue36360  closed by Shady Atef

#36362: Detected unused variables in import.c and HAVE_DYNAMIC_LOADING
https://bugs.python.org/issue36362  closed by brett.cannon

#36363: Wrong type when missname dataclass attribute with existing var
https://bugs.python.org/issue36363  closed by gvanrossum

#36365: Objects/structseq.c: warning: 'strncpy' specified bound depend
https://bugs.python.org/issue36365  closed by vstinner

#36367: tokenizer.c memory leak in case of realloc failure
https://bugs.python.org/issue36367  closed by pablogsal

#36371: FancyGetopt.generate_help crashes with TypeError
https://bugs.python.org/issue36371  closed by SilentGhost

#36374: A possible null pointer dereference in compile.c's merge_const
https://bugs.python.org/issue36374  closed by inada.naoki

#36376: Wrong position of SyntaxError in IDLE
https://bugs.python.org/issue36376  closed by terry.reedy

#36380: collections.Counter in-place operators are unexpectedly slow
https://bugs.python.org/issue36380  closed by rhettinger

#36382: socket.getfqdn() returns domain "mshome.net"
https://bugs.python.org/issue36382  closed by markuskramerIgitt

#36385: Add ``elif`` sentence on to avoid multiple ``if``
https://bugs.python.org/issue36385  closed by rhettinger

#36386: segfault on PyUnicode_DecodeFSDefaultAndSize for uninitialized
https://bugs.python.org/issue36386  closed by vstinner

#36391: XSS in bugs.python.org 404 error page
https://bugs.python.org/issue36391  closed by SilentGhost

#36393: python user define function is replacing variable value
https://bugs.python.org/issue36393  closed by zach.ware

#36397: re.split() incorrectly splitting on zero-width pattern
https://bugs.python.org/issue36397  closed by mrabarnett

#36400: Add activate script to venv root folder
https://bugs.python.org/issue36400  closed by brett.cannon

#969718: BASECFLAGS are not passed to module build line
https://bugs.python.org/issue969718  closed by vstinner

From steve.dower at python.org  Fri Mar 22 14:10:38 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 22 Mar 2019 11:10:38 -0700
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAP1=2W4+WckF3FmM1RjX7DtbU1HN5V=WqnJawXCeV+evus0Wfw@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAP1=2W6uE+YXTg8j4o+3z8osipdOAd+rkCZx2QZ-hKmM5A4tDw@mail.gmail.com>
 <CAEfz+Tzqnrx3Jfcf8hCia4DBCV4hVWxpHhEW8-f8LsfYps1uyQ@mail.gmail.com>
 <CAP1=2W4+WckF3FmM1RjX7DtbU1HN5V=WqnJawXCeV+evus0Wfw@mail.gmail.com>
Message-ID: <9ded6786-7e84-c8cc-e841-94cfcd59b687@python.org>

On 22Mar2019 1101, Brett Cannon wrote:
> On Fri, Mar 22, 2019 at 10:37 AM Inada Naoki <songofacandy at gmail.com 
> <mailto:songofacandy at gmail.com>> wrote:
>     There might be some small troubles.? But it was small enough for
>     Python minor versions, I think.
> 
> 
>  ?I don't think it's worth the cost to users. We can just choose to stop 
> using it in the stdlib and not use PendingDeprecationWarning. And if 
> people want to force others to define their own 
> PendingDeprecationWarning by deprecating that's fine, but the aliasing 
> where it could cause unintended exception swallowing for something 
> related to breaking changes seems unnecessarily risky to me simply 
> because we don't want to ask users to update their code in a 
> backwards-compatible fashion.

What does it mean to put DeprecationWarning on PendingDeprecationWarning 
and then alias PendingDeprecationWarning to DeprecationWarning?

"If the implementation is hard to explain, it's a bad idea." :)

(FWIW, agree with Brett. We can simply stop using it ourselves without 
breaking anyone. Of all the things in the stdlib that are hard to 
maintain, this is nowhere near the top of the list.)

Cheers,
Steve

From guido at python.org  Fri Mar 22 14:23:55 2019
From: guido at python.org (Guido van Rossum)
Date: Fri, 22 Mar 2019 11:23:55 -0700
Subject: [Python-Dev] Invitation to the PyCon typing summit (Thu, May 2nd,
 1-5pm, at PyCon in Cleveland)
Message-ID: <CAP7+vJL0_Y2gfs3C2-u80ZK2JCfX-Gx=a_2G2kTD9OG3-oqn5A@mail.gmail.com>

The typing summit is primarily a place for developers of type checkers to
collaborate, but we are also inviting (potential) users of type checkers.
For example, there are plans to extend the standard Python type system with
features intended to support numpy, Pandas, tensorflow and similar
libraries, and we will discuss these at the summit. Therefore developers
and power-users of such frameworks are especially welcome at the summit.

With Ewa's and Dropbox's help I've arranged a room at PyCon.


*When: Thursday May 2nd, 1-5 pm (i.e. the day between the Language Summit
and the conference proper)*
*Where: Room 6 at PyCon in Cleveland*

If you're planning to attend, please fill out this form:

https://goo.gl/forms/rG9dVTBbgyBgDK8H2

*If you already emailed me about attending, please still fill out the form!*

(Also if you already registered in response to my invitation on typing-sig,
I've got your registration info.)


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190322/266bfee5/attachment.html>

From greg.ewing at canterbury.ac.nz  Fri Mar 22 20:01:17 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 23 Mar 2019 13:01:17 +1300
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <q72gqf$81gc$1@blaine.gmane.org>
References: <5C948F55.4040108@canterbury.ac.nz>
 <q72gqf$81gc$1@blaine.gmane.org>
Message-ID: <5C95774D.9050907@canterbury.ac.nz>

Serhiy Storchaka wrote:
> Making it always 32 bits would be compatibility breaking change. 
> Currently array('u') represents the wchar_t string, and many API on 
> Windows require it.

Ah, I see. It would be helpful if the array module docs made that clear.
At one point the 3.x docs said that it depended on whether you had a wide
or narrow unicode Python build, which confused me, because I thought that 
distinction had gone away in Python 3.

> But we can add a new code, e.g. 'U', for UCS4.

+1.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Mar 22 20:19:01 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 23 Mar 2019 13:19:01 +1300
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <20190322113905.3fefed62@fsol>
References: <5C948F55.4040108@canterbury.ac.nz> <20190322113905.3fefed62@fsol>
Message-ID: <5C957B75.7000402@canterbury.ac.nz>

Antoine Pitrou wrote:
> TBH, I think anyone trying to use array.array should be directed to
> Numpy these days.  The only reason for array.array being here is that
> it predates Numpy.

Numpy is a huge dependency to pull in when you don't need all the
heavyweight array machinery.

Also, numpy arrays behave very differently from Python sequences
in many ways. For example, they don't have a flexible size, and
you can't concatenate them using +.

Arrays from the array module follow the sequence protocol much
more closely, and that makes them valuable, IMO.

-- 
Greg

From songofacandy at gmail.com  Fri Mar 22 21:18:57 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Sat, 23 Mar 2019 10:18:57 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAP1=2W4+WckF3FmM1RjX7DtbU1HN5V=WqnJawXCeV+evus0Wfw@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAP1=2W6uE+YXTg8j4o+3z8osipdOAd+rkCZx2QZ-hKmM5A4tDw@mail.gmail.com>
 <CAEfz+Tzqnrx3Jfcf8hCia4DBCV4hVWxpHhEW8-f8LsfYps1uyQ@mail.gmail.com>
 <CAP1=2W4+WckF3FmM1RjX7DtbU1HN5V=WqnJawXCeV+evus0Wfw@mail.gmail.com>
Message-ID: <CAEfz+TyUGnaABJGph0p9bAdob_Diq5bibPNpv4uGk0=6P79rqA@mail.gmail.com>

On Sat, Mar 23, 2019 at 3:02 AM Brett Cannon <brett at python.org> wrote:
>
>>
>> There might be some small troubles.  But it was small enough for
>> Python minor versions, I think.
>
>
>  I don't think it's worth the cost to users. We can just choose to stop using it in the stdlib and not use PendingDeprecationWarning. And if people want to force others to define their own PendingDeprecationWarning by deprecating that's fine, but the aliasing where it could cause unintended exception swallowing for something related to breaking changes seems unnecessarily risky to me simply because we don't want to ask users to update their code in a backwards-compatible fashion.

I still can't believe there are real world usage of PendingDeprecationWarning
other than warnings.warn() and assertRaises().

But I'm OK to not removing actual class.

Stop using it in stdlib reduces maintenance cost like this:
https://github.com/python/cpython/pull/12494/files

And deprecation in document reduces learning cost.
People can skip reading and understanding document of PendingDeprecatedWarning.

Keeping PendingDeprecationWarning class for several years is very
low cost compared to these cost.

Regards,
-- 
Inada Naoki  <songofacandy at gmail.com>

From songofacandy at gmail.com  Fri Mar 22 22:10:00 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Sat, 23 Mar 2019 11:10:00 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+TyUGnaABJGph0p9bAdob_Diq5bibPNpv4uGk0=6P79rqA@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAP1=2W6uE+YXTg8j4o+3z8osipdOAd+rkCZx2QZ-hKmM5A4tDw@mail.gmail.com>
 <CAEfz+Tzqnrx3Jfcf8hCia4DBCV4hVWxpHhEW8-f8LsfYps1uyQ@mail.gmail.com>
 <CAP1=2W4+WckF3FmM1RjX7DtbU1HN5V=WqnJawXCeV+evus0Wfw@mail.gmail.com>
 <CAEfz+TyUGnaABJGph0p9bAdob_Diq5bibPNpv4uGk0=6P79rqA@mail.gmail.com>
Message-ID: <CAEfz+Twd9Tw6yz7hkD28PxzTT02O=+pKUj_9oiTjyWif2QMSMA@mail.gmail.com>

I created the PR to deprecate PendingDeprecationWarning only in document.
https://github.com/python/cpython/pull/12505

On Sat, Mar 23, 2019 at 10:18 AM Inada Naoki <songofacandy at gmail.com> wrote:
>
> On Sat, Mar 23, 2019 at 3:02 AM Brett Cannon <brett at python.org> wrote:
> >
> >>
> >> There might be some small troubles.  But it was small enough for
> >> Python minor versions, I think.
> >
> >
> >  I don't think it's worth the cost to users. We can just choose to stop using it in the stdlib and not use PendingDeprecationWarning. And if people want to force others to define their own PendingDeprecationWarning by deprecating that's fine, but the aliasing where it could cause unintended exception swallowing for something related to breaking changes seems unnecessarily risky to me simply because we don't want to ask users to update their code in a backwards-compatible fashion.
>
> I still can't believe there are real world usage of PendingDeprecationWarning
> other than warnings.warn() and assertRaises().
>
> But I'm OK to not removing actual class.
>
> Stop using it in stdlib reduces maintenance cost like this:
> https://github.com/python/cpython/pull/12494/files
>
> And deprecation in document reduces learning cost.
> People can skip reading and understanding document of PendingDeprecatedWarning.
>
> Keeping PendingDeprecationWarning class for several years is very
> low cost compared to these cost.
>
> Regards,
> --
> Inada Naoki  <songofacandy at gmail.com>



-- 
Inada Naoki  <songofacandy at gmail.com>

From brett at python.org  Sat Mar 23 19:07:29 2019
From: brett at python.org (Brett Cannon)
Date: Sat, 23 Mar 2019 16:07:29 -0700
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+TyUGnaABJGph0p9bAdob_Diq5bibPNpv4uGk0=6P79rqA@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAP1=2W6uE+YXTg8j4o+3z8osipdOAd+rkCZx2QZ-hKmM5A4tDw@mail.gmail.com>
 <CAEfz+Tzqnrx3Jfcf8hCia4DBCV4hVWxpHhEW8-f8LsfYps1uyQ@mail.gmail.com>
 <CAP1=2W4+WckF3FmM1RjX7DtbU1HN5V=WqnJawXCeV+evus0Wfw@mail.gmail.com>
 <CAEfz+TyUGnaABJGph0p9bAdob_Diq5bibPNpv4uGk0=6P79rqA@mail.gmail.com>
Message-ID: <CAP1=2W5V7Oj2L-LswfikZqbHtP3DMWdWaTf=E-X-uUjMKx0+PA@mail.gmail.com>

On Fri, Mar 22, 2019 at 6:19 PM Inada Naoki <songofacandy at gmail.com> wrote:

> On Sat, Mar 23, 2019 at 3:02 AM Brett Cannon <brett at python.org> wrote:
> >
> >>
> >> There might be some small troubles.  But it was small enough for
> >> Python minor versions, I think.
> >
> >
> >  I don't think it's worth the cost to users. We can just choose to stop
> using it in the stdlib and not use PendingDeprecationWarning. And if people
> want to force others to define their own PendingDeprecationWarning by
> deprecating that's fine, but the aliasing where it could cause unintended
> exception swallowing for something related to breaking changes seems
> unnecessarily risky to me simply because we don't want to ask users to
> update their code in a backwards-compatible fashion.
>
> I still can't believe there are real world usage of
> PendingDeprecationWarning
> other than warnings.warn() and assertRaises().
>

I've made the same mistake of assuming something that made no sense to me
wouldn't make sense to anyone else and I have been proven wrong nearly
every time. ;)

-Brett


>
> But I'm OK to not removing actual class.
>
> Stop using it in stdlib reduces maintenance cost like this:
> https://github.com/python/cpython/pull/12494/files
>
> And deprecation in document reduces learning cost.
> People can skip reading and understanding document of
> PendingDeprecatedWarning.
>
> Keeping PendingDeprecationWarning class for several years is very
> low cost compared to these cost.
>
> Regards,
> --
> Inada Naoki  <songofacandy at gmail.com>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190323/33459885/attachment.html>

From cs at cskk.id.au  Sat Mar 23 19:59:35 2019
From: cs at cskk.id.au (Cameron Simpson)
Date: Sun, 24 Mar 2019 10:59:35 +1100
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <5C9229B0.20501@UGent.be>
References: <5C9229B0.20501@UGent.be>
Message-ID: <20190323235935.GA80995@cskk.homeip.net>

On 20Mar2019 12:53, Jeroen Demeyer <J.Demeyer at UGent.be> wrote:
>On 2019-03-20 12:45, Victor Stinner wrote:
>>You can watch the /tmp directory using inotify and "discover"
>>immediately the "secret" filename, it doesn't depend on the amount of
>>entropy used to generate the filename.
>
>That's not the problem. The security issue here is guessing the 
>filename *before* it's created and putting a different file or symlink 
>in place.
>
>So I actually do think that mktemp() could be made secure by using a 
>longer name generated by a secure random generator.

I know it is days later, but to add a little nuance: the security issue 
is guessing the filename before it is _used_. Consider:

  path = tempfile.mktemp()
  with open(path, "w"):
    write some secret stuff ...
  call_other_function(path)

If an attacker gets in _after_ the open (which creates the file) by 
using something like inotify to _observe_ the pathname instead of 
guessing and supplants the file then, call_other_function is then 
subverted.

Also, the common examples are attackers who are not the user making the 
tempfile, in which case the _default_ mktemp is sort of secure with the 
above because it gets made in /tmp which on a modern POSIX system 
prevents _other_ uses from removing/renaming a file. (And Eryk I think 
described the Windows situation which is similarly protected).

However, mktemp somewhere else is not so protected.

And the attacker might be malware running as the orignal user (yes the 
game may already be overin that case for other reasons).

However, I wanted to make the point that the security issue isn't around 
creation but use - trusting the mktemp pathname to be the same state as 
it was earlier.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From eryksun at gmail.com  Sat Mar 23 21:13:26 2019
From: eryksun at gmail.com (eryk sun)
Date: Sat, 23 Mar 2019 20:13:26 -0500
Subject: [Python-Dev] Remove tempfile.mktemp()
In-Reply-To: <20190323235935.GA80995@cskk.homeip.net>
References: <5C9229B0.20501@UGent.be> <20190323235935.GA80995@cskk.homeip.net>
Message-ID: <CACL+1atNQXcBaGMF-4cS_3Bc0MVazC0_00fLks0hhNoR0gKQTg@mail.gmail.com>

On 3/23/19, Cameron Simpson <cs at cskk.id.au> wrote:
>
> Also, the common examples are attackers who are not the user making the
> tempfile, in which case the _default_ mktemp is sort of secure with the
> above because it gets made in /tmp which on a modern POSIX system
> prevents _other_ uses from removing/renaming a file. (And Eryk I think
> described the Windows situation which is similarly protected).

Using NamedTemporaryFile(delete=False) or mkstemp() ensures that the
file is created and opened securely. in contrast, the filename from
mktemp() might be used naively in POSIX, such as open(path, "w"). This
file might grant read access to everyone depending on the file-mode
creation mask (umask). Also, since it neglects to use exclusive mode
("x"), it might open an existing file that grants read-write
permission to the world, or maybe it's a symlink.

By default, even naive use of the mktemp() name in Windows remains
secure, since every user has a separate temp directory that's only
accessible by privileged users such as SYSTEM, Administrators, and
Backup Operators (with SeBackupPrivilege and SeRestorePrivilege
enabled). The primary issue with a short name is an accidental name
collision with another program that's not as careful as Python's
tempfile. Using a longer name decreases the chance of this to
practically nothing.

From songofacandy at gmail.com  Sat Mar 23 21:51:32 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Sun, 24 Mar 2019 10:51:32 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAP1=2W5V7Oj2L-LswfikZqbHtP3DMWdWaTf=E-X-uUjMKx0+PA@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CA+3bQGGC17vzNOgaex92vF1yG3YJ0n4idHC-yBO1BHAQvwnX8Q@mail.gmail.com>
 <CAP1=2W6uE+YXTg8j4o+3z8osipdOAd+rkCZx2QZ-hKmM5A4tDw@mail.gmail.com>
 <CAEfz+Tzqnrx3Jfcf8hCia4DBCV4hVWxpHhEW8-f8LsfYps1uyQ@mail.gmail.com>
 <CAP1=2W4+WckF3FmM1RjX7DtbU1HN5V=WqnJawXCeV+evus0Wfw@mail.gmail.com>
 <CAEfz+TyUGnaABJGph0p9bAdob_Diq5bibPNpv4uGk0=6P79rqA@mail.gmail.com>
 <CAP1=2W5V7Oj2L-LswfikZqbHtP3DMWdWaTf=E-X-uUjMKx0+PA@mail.gmail.com>
Message-ID: <CAEfz+TzN7uGEK+YYponGeX3iGCm1hi7D7f1pc8COhHNT2CsTWQ@mail.gmail.com>

On Sun, Mar 24, 2019 at 8:07 AM Brett Cannon <brett at python.org> wrote:
>
>
> I've made the same mistake of assuming something that made no sense to me wouldn't make sense to anyone else and I have been proven wrong nearly every time. ;)
>
> -Brett
>

And beta and RC phase can be used to detect such breakage.

When no real world pain is assumed, we can chose simple solution.
Then, if real world pain is found, we can move to more complicated,
but backward compatible way.

Of course, keeping PendingDeprecatedWarning is not complicated way
in this time.  So I don't push the removal.  It was just an idea.

-- 
Inada Naoki  <songofacandy at gmail.com>

From ncoghlan at gmail.com  Sun Mar 24 07:57:46 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Mar 2019 21:57:46 +1000
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
Message-ID: <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>

On Fri, 22 Mar 2019 at 12:45, Inada Naoki <songofacandy at gmail.com> wrote:
>
> Hi, all.
>
> I'm thinking about removing PendingDeprecationWarning.
> (previous discussion:
> https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038)
>
> It was added "not be printed by default" version of DeprecationWarning.
> But DeprecationWarning is not printed by default now.

No, this was covered in PEP 565, and PendingDeprecationWarning was
explicitly kept as a way of opting out of the revised semantics of
DeprecationWarning.

In Python 3.7 and above, the semantics are:

* PendingDeprecationWarning: never shown by default
* DeprecationWarning: shown by default in the __main__ module
* FutureWarning: shown by default everywhere

PEP section: https://www.python.org/dev/peps/pep-0565/#additional-use-case-for-futurewarning

The documentation was also updated to match:

* https://docs.python.org/3/library/warnings.html#warning-categories
* https://docs.python.org/3/library/exceptions.html#warnings

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Mar 24 08:21:22 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Mar 2019 22:21:22 +1000
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
Message-ID: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>

Hi folks,

With the revised PEP 1 published, the Steering Council members have
been working through the backlog of open PEPs, figuring out which ones
are at a stage of maturity where we think it makes sense to appoint a
BDFL-Delegate to continue moving the PEP through the review process,
and eventually make the final decision on whether or not to accept or
reject the change.

We'll be announcing those appointments as we go, so I'm happy to
report that I will be handling the BDFL-Delegate responsibilities for
the following PEPs:

* PEP 499: Binding "-m" executed modules under their module name as
well as `__main__`
* PEP 574: Pickle protocol 5 with out of band data

I'm also pleased to report that Petr Viktorin has agreed to take on
the responsibility of reviewing the competing proposals to improve the
way CPython's C API exposes callables for direct invocation by third
party low level code:

* PEP 576: Exposing the internal FastCallKeywords convention to 3rd
party modules
* PEP 580: Revising the callable struct hierarchy internally and in
the public C API
* PEP 579: Background information for the problems the other two PEPs
are attempting to address

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mark at hotpy.org  Sun Mar 24 11:22:35 2019
From: mark at hotpy.org (Mark Shannon)
Date: Sun, 24 Mar 2019 15:22:35 +0000
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
Message-ID: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>

Hi Petr,

Regarding PEPs 576 and 580.
Over the new year, I did a thorough analysis of possible approaches to 
possible calling conventions for use in the CPython ecosystems and came 
up with a new PEP.
The draft can be found here:
https://github.com/markshannon/peps/blob/new-calling-convention/pep-9999.rst

I was hoping to profile a branch with the various experimental changes 
cherry-picked together, but don't seemed to have found the time :(

I'd like to have a testable branch, before formally submitting the PEP,
but I'd thought you should be aware of the PEP.

Cheers,
Mark.


On 24/03/2019 12:21 pm, Nick Coghlan wrote:
> Hi folks,
> 
> With the revised PEP 1 published, the Steering Council members have
> been working through the backlog of open PEPs, figuring out which ones
> are at a stage of maturity where we think it makes sense to appoint a
> BDFL-Delegate to continue moving the PEP through the review process,
> and eventually make the final decision on whether or not to accept or
> reject the change.
> 
> We'll be announcing those appointments as we go, so I'm happy to
> report that I will be handling the BDFL-Delegate responsibilities for
> the following PEPs:
> 
> * PEP 499: Binding "-m" executed modules under their module name as
> well as `__main__`
> * PEP 574: Pickle protocol 5 with out of band data
> 
> I'm also pleased to report that Petr Viktorin has agreed to take on
> the responsibility of reviewing the competing proposals to improve the
> way CPython's C API exposes callables for direct invocation by third
> party low level code:
> 
> * PEP 576: Exposing the internal FastCallKeywords convention to 3rd
> party modules
> * PEP 580: Revising the callable struct hierarchy internally and in
> the public C API
> * PEP 579: Background information for the problems the other two PEPs
> are attempting to address
> 
> Regards,
> Nick.
> 

From J.Demeyer at UGent.be  Sun Mar 24 13:00:00 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Sun, 24 Mar 2019 18:00:00 +0100
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
 <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
Message-ID: <5C97B790.3070009@UGent.be>

On 2019-03-24 16:22, Mark Shannon wrote:
> Hi Petr,
>
> Regarding PEPs 576 and 580.
> Over the new year, I did a thorough analysis of possible approaches to
> possible calling conventions for use in the CPython ecosystems and came
> up with a new PEP.
> The draft can be found here:
> https://github.com/markshannon/peps/blob/new-calling-convention/pep-9999.rst

Thanks for that. Is this new PEP meant to supersede PEP 576?

> I'd like to have a testable branch, before formally submitting the PEP,
> but I'd thought you should be aware of the PEP.

If you want to bring up this PEP now during the PEP 576/580 discussion, 
maybe it's best to formally submit it now? Having an official PEP number 
might simplify the discussion. If it turns out to be a bad idea after 
all, you can still withdraw it.

In the mean time, I remind you that PEP 576 also doesn't have a complete 
reference implementation (the PEP links to a "reference implementation" 
but it doesn't correspond to the text of the PEP).


Jeroen.

From J.Demeyer at UGent.be  Sun Mar 24 13:25:05 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Sun, 24 Mar 2019 18:25:05 +0100
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
 <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
Message-ID: <5C97BD71.8060601@UGent.be>

On 2019-03-24 16:22, Mark Shannon wrote:
> The draft can be found here:
> https://github.com/markshannon/peps/blob/new-calling-convention/pep-9999.rst

I think that this is basically a better version of PEP 576. The idea is 
the same as PEP 576, but the details are better. Since it's not 
fundamentally different from PEP 576, I think that this comparison still 
stands:

https://mail.python.org/pipermail/python-dev/2018-July/154238.html

From J.Demeyer at UGent.be  Sun Mar 24 15:05:36 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Sun, 24 Mar 2019 20:05:36 +0100
Subject: [Python-Dev] PEP 576/580: on the complexity of function calls
Message-ID: <5C97D500.905@UGent.be>

Now that the discussion on PEP 576/580 has been opened again, let me 
write something about the complexity of function calls (*), which is 
probably the most frequently given reason against PEP 580.

An important fact is the following: *the status-quo is complex*.

Over time, many performance improvements have been made to function 
calls. Each of these was a relatively small incremental change (for 
example, METH_FASTCALL with *args only was added before 
METH_FASTCALL|METH_KEYWORDS with *args and **kwargs). In the end, all 
these small changes add up to quite a bit of complexity. The fact that 
this complexity isn't documented anywhere and that it's distributed over 
several .c files in the CPython sources makes it perhaps not obvious 
that it's there.

Neither PEP 576 nor PEP 580 tries to remove this complexity. Indeed, the 
complexity is there for good reasons, as it improves performance of 
function calls in many ways. But the PEPs handle it in very different ways.

On the one hand, PEP 580 concentrates all the complexity in the 
protocol. So the protocol looks complex, even though most of it is 
really just formulating existing complexity. More importantly, since the 
complexity is moved to the protocol, it becomes quite easy to use PEP 
580 in a class: you don't need to understand the implementation of PEP 
580 for that.

On the other hand, PEP 576 keeps the existing complexity out of the 
protocol. This means that the implementations of classes using PEP 576 
become more complex, as the existing complexity needs to be implemented 
somewhere. In fact, with PEP 576 the existing complexity needs to be 
implemented in many places, leading for example to code duplication 
between builtin_function_or_method and method_descriptor. This kind of 
code duplication would again occur for third-party method-like classes.

Note that everything I said above about PEP 576 also applies to the 
not-yet-PEP
https://github.com/markshannon/peps/blob/new-calling-convention/pep-9999.rst

Best wishes,
Jeroen.


(*) With "function calls", I mean most importantly calls of instances of 
builtin_function_or_method, method, method_descriptor and function. But 
since PEP 576/580 are meant for third-party function/method classes, 
also those should be considered.

From stephane at wirtel.be  Sun Mar 24 15:52:42 2019
From: stephane at wirtel.be (=?UTF-8?Q?St=c3=a9phane_Wirtel?=)
Date: Sun, 24 Mar 2019 20:52:42 +0100
Subject: [Python-Dev] introduction of __attribute__(deprecated) ?
Message-ID: <56b8c4f7-510d-2d8f-8c71-4b6bae04c0b7@wirtel.be>

Hi,

I have created the issue https://bugs.python.org/issue36347 because I
wanted to add a missing macro for the PyMemberDef.flags attribute.

In the Modules/*.c files, we can find descriptions with
PyMemberDef where the access flag has the 0 value.

Example:

static PyMemberDef members[] = {
    {"name", T_OBJECT, offsetof(MyObject, name), 0, NULL},
    {NULL},
};

So, I didn't know the meaning of this magic number (0) and after a small
search in structmember.h, this is the default value for an "READWRITE"
attribute but there is no associated macro to this magic number.

solution: add the macro for the READWRITE mode.

so, my first PR has added the new macro READWRITE, like that

#define READWRITE 0

and improve the documentation in Doc/c-api/structures.rst.

but after a review [1], Serhiy proposed to rename it and the other ones
because it did not follow the convention for names in the C API.
Use a prefix for the public names, example Py_ or PY_.

I chose PY_ because PY_WRITE_RESTRICTED already existed.

the next steps were:
* add the new macros
* keep the backward-compatibility
* updated the documentation

I haved pushed the PR for another review

so, but I don't like this definition of READONLY because there is no way
to generate a warning at the compile time when we use the READONLY macro.

#define PY_READONLY	0
#define READONLY 	PY_READONLY

after that, I have closed my PR because I was not proud with my "solution".


Today, Ronald Oussoren [2] has proposed an other solution based on an
enum and a new __attribute__, aka deprecated.

I have checked with gcc and clang and this option is interesting because
we generate a warning when we try to compile a code if we use the
deprecated READONLY [3]

In the clang documantion, the __attribute__(deprecated) is defined in
GNU and C++11 standard, like __attribute__(unused).

In the gcc documentation, this attribute also exists

Because we already use __attribute__(unused) we could add
__attribute__(deprecated).


LC_ALL=C clang test.c -o test
test.c:14:13: warning: 'READONLY' is deprecated: use PY_READONLY
[-Wdeprecated-declarations]
    int i = READONLY;
            ^
test.c:8:27: note: 'READONLY' has been explicitly marked deprecated here
    READONLY __attribute((deprecated("use PY_READONLY"))) = PY_READONLY,
                          ^
1 warning generated.

LC_ALL=C clang --version test.c -o test
clang version 7.0.1 (Fedora 7.0.1-6.fc29)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

### GCC
LC_ALL=C gcc --version test.c -o test
gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

LC_ALL=C gcc test.c -o test
test.c: In function 'main':
test.c:14:5: warning: 'READONLY' is deprecated: use PY_READONLY
[-Wdeprecated-declarations]
     int i = READONLY;
     ^~~
test.c:8:5: note: declared here
     READONLY __attribute((deprecated("use PY_READONLY"))) = PY_READONLY,
     ^~~~~~~~

So my question is, can we use/add __attribute__(deprecated) in our
"development" kit?

[1] https://bugs.python.org/issue36347#msg338261
[2] https://bugs.python.org/issue36347#msg338731
[3] https://bugs.python.org/issue36347#msg338745

Thank you,

St?phane

From stephane at wirtel.be  Sun Mar 24 16:00:44 2019
From: stephane at wirtel.be (=?UTF-8?Q?St=c3=a9phane_Wirtel?=)
Date: Sun, 24 Mar 2019 21:00:44 +0100
Subject: [Python-Dev] introduction of __attribute__(deprecated) ?
In-Reply-To: <56b8c4f7-510d-2d8f-8c71-4b6bae04c0b7@wirtel.be>
References: <56b8c4f7-510d-2d8f-8c71-4b6bae04c0b7@wirtel.be>
Message-ID: <e84a8892-c003-e134-a03e-fb1baf855dc5@wirtel.be>

And use an enumeration for the constants.

For example:

#include <stdio.h>
#include <stdlib.h>

enum {
    PY_READWRITE = 0,
    PY_READONLY = 1,

    READONLY __attribute((deprecated("use PY_READONLY"))) = PY_READONLY,
    READWRITE __attribute((deprecated("use PY_READWRITE"))) = PY_READWRITE,
};



Le 24/03/19 ? 20:52, St?phane Wirtel a ?crit?:
> Hi,
> 
> I have created the issue https://bugs.python.org/issue36347 because I
> wanted to add a missing macro for the PyMemberDef.flags attribute.
> 
> In the Modules/*.c files, we can find descriptions with
> PyMemberDef where the access flag has the 0 value.
> 
> Example:
> 
> static PyMemberDef members[] = {
>     {"name", T_OBJECT, offsetof(MyObject, name), 0, NULL},
>     {NULL},
> };
> 
> So, I didn't know the meaning of this magic number (0) and after a small
> search in structmember.h, this is the default value for an "READWRITE"
> attribute but there is no associated macro to this magic number.
> 
> solution: add the macro for the READWRITE mode.
> 
> so, my first PR has added the new macro READWRITE, like that
> 
> #define READWRITE 0
> 
> and improve the documentation in Doc/c-api/structures.rst.
> 
> but after a review [1], Serhiy proposed to rename it and the other ones
> because it did not follow the convention for names in the C API.
> Use a prefix for the public names, example Py_ or PY_.
> 
> I chose PY_ because PY_WRITE_RESTRICTED already existed.
> 
> the next steps were:
> * add the new macros
> * keep the backward-compatibility
> * updated the documentation
> 
> I haved pushed the PR for another review
> 
> so, but I don't like this definition of READONLY because there is no way
> to generate a warning at the compile time when we use the READONLY macro.
> 
> #define PY_READONLY	0
> #define READONLY 	PY_READONLY
> 
> after that, I have closed my PR because I was not proud with my "solution".
> 
> 
> Today, Ronald Oussoren [2] has proposed an other solution based on an
> enum and a new __attribute__, aka deprecated.
> 
> I have checked with gcc and clang and this option is interesting because
> we generate a warning when we try to compile a code if we use the
> deprecated READONLY [3]
> 
> In the clang documantion, the __attribute__(deprecated) is defined in
> GNU and C++11 standard, like __attribute__(unused).
> 
> In the gcc documentation, this attribute also exists
> 
> Because we already use __attribute__(unused) we could add
> __attribute__(deprecated).
> 
> 
> LC_ALL=C clang test.c -o test
> test.c:14:13: warning: 'READONLY' is deprecated: use PY_READONLY
> [-Wdeprecated-declarations]
>     int i = READONLY;
>             ^
> test.c:8:27: note: 'READONLY' has been explicitly marked deprecated here
>     READONLY __attribute((deprecated("use PY_READONLY"))) = PY_READONLY,
>                           ^
> 1 warning generated.
> 
> LC_ALL=C clang --version test.c -o test
> clang version 7.0.1 (Fedora 7.0.1-6.fc29)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix
> InstalledDir: /usr/bin
> 
> ### GCC
> LC_ALL=C gcc --version test.c -o test
> gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)
> Copyright (C) 2018 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> 
> LC_ALL=C gcc test.c -o test
> test.c: In function 'main':
> test.c:14:5: warning: 'READONLY' is deprecated: use PY_READONLY
> [-Wdeprecated-declarations]
>      int i = READONLY;
>      ^~~
> test.c:8:5: note: declared here
>      READONLY __attribute((deprecated("use PY_READONLY"))) = PY_READONLY,
>      ^~~~~~~~
> 
> So my question is, can we use/add __attribute__(deprecated) in our
> "development" kit?
> 
> [1] https://bugs.python.org/issue36347#msg338261
> [2] https://bugs.python.org/issue36347#msg338731
> [3] https://bugs.python.org/issue36347#msg338745
> 
> Thank you,
> 
> St?phane
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/stephane%40wirtel.be
> 

From tjreedy at udel.edu  Sun Mar 24 17:02:34 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 24 Mar 2019 17:02:34 -0400
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
Message-ID: <q78r9d$uvr$1@blaine.gmane.org>

On 3/24/2019 8:21 AM, Nick Coghlan wrote:

> We'll be announcing those appointments as we go, so I'm happy to
> report that I will be handling the BDFL-Delegate responsibilities for
> the following PEPs:

Where do we discuss these?

If a delegate has a provisional view, it might help focus discussion if 
that were known.

> * PEP 499: Binding "-m" executed modules under their module name as
> well as `__main__`

My brief response: +1 unless there is a good reason not.  There have 
been multiple double module problems reported on python-list and likely 
stackoverflow.  And would there be any impact on circular imports?

-- 
Terry Jan Reedy


From vstinner at redhat.com  Sun Mar 24 18:07:37 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Sun, 24 Mar 2019 23:07:37 +0100
Subject: [Python-Dev] introduction of __attribute__(deprecated) ?
In-Reply-To: <56b8c4f7-510d-2d8f-8c71-4b6bae04c0b7@wirtel.be>
References: <56b8c4f7-510d-2d8f-8c71-4b6bae04c0b7@wirtel.be>
Message-ID: <CA+3bQGGaUC+1k03=t8mqUo1fhfrRrZ=zUNHBhBcE93Dt9TH16g@mail.gmail.com>

Le dim. 24 mars 2019 ? 20:56, St?phane Wirtel <stephane at wirtel.be> a ?crit :
> So my question is, can we use/add __attribute__(deprecated) in our
> "development" kit?

There is already Py_DEPRECATED(VERSION_UNUSED) macro which uses
__attribute__((__deprecated__)) on GCC >= 3.1.

You can find the doc of a few macros in
https://docs.python.org/dev/c-api/intro.html#useful-macros but this
macro isn't documented there.

pyport.h and pymacro.h contain most of the "macros".

Victor

From stephane at wirtel.be  Sun Mar 24 18:44:19 2019
From: stephane at wirtel.be (=?UTF-8?Q?St=c3=a9phane_Wirtel?=)
Date: Sun, 24 Mar 2019 23:44:19 +0100
Subject: [Python-Dev] introduction of __attribute__(deprecated) ?
In-Reply-To: <CA+3bQGGaUC+1k03=t8mqUo1fhfrRrZ=zUNHBhBcE93Dt9TH16g@mail.gmail.com>
References: <56b8c4f7-510d-2d8f-8c71-4b6bae04c0b7@wirtel.be>
 <CA+3bQGGaUC+1k03=t8mqUo1fhfrRrZ=zUNHBhBcE93Dt9TH16g@mail.gmail.com>
Message-ID: <374b5c31-afc4-c0f6-9ecd-0fc14ebb2d10@wirtel.be>

Thanks Victor, I have updated my PR with the Py_DEPRECATED macro.


Le 24/03/19 ? 23:07, Victor Stinner a ?crit?:
> Le dim. 24 mars 2019 ? 20:56, St?phane Wirtel <stephane at wirtel.be> a ?crit :
>> So my question is, can we use/add __attribute__(deprecated) in our
>> "development" kit?
> 
> There is already Py_DEPRECATED(VERSION_UNUSED) macro which uses
> __attribute__((__deprecated__)) on GCC >= 3.1.
> 
> You can find the doc of a few macros in
> https://docs.python.org/dev/c-api/intro.html#useful-macros but this
> macro isn't documented there.
> 
> pyport.h and pymacro.h contain most of the "macros".
> 
> Victor
> 

From cs at cskk.id.au  Sun Mar 24 19:00:36 2019
From: cs at cskk.id.au (Cameron Simpson)
Date: Mon, 25 Mar 2019 10:00:36 +1100
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <q78r9d$uvr$1@blaine.gmane.org>
References: <q78r9d$uvr$1@blaine.gmane.org>
Message-ID: <20190324230036.GA72432@cskk.homeip.net>

On 24Mar2019 17:02, Terry Reedy <tjreedy at udel.edu> wrote:
>On 3/24/2019 8:21 AM, Nick Coghlan wrote:
>>* PEP 499: Binding "-m" executed modules under their module name as
>>well as `__main__`
>
>My brief response: +1 unless there is a good reason not.

There turn out to be some subtle side effects. The test suite turned up 
one (easily fixed) in pdb, but there are definitely some more things to 
investigate.  Nick has pointed out pickle and the "python -i" option.  
I'm digging into these. (Naturally, I have _never_ before used the pdb 
or pickle modules, or the -i option :-)

>There have been multiple double module problems reported on python-list 
>and likely stackoverflow.  And would there be any impact on circular 
>imports?

Well, by binding the -m module to both __main__ and its name as denoted 
on the command line one circular import is directly short circuited.  
Aside from the -m module itself, I don't think there should be any other 
direct effect on circular imports.

Did you have a specific scenario in mind?

Cheers,
Cameron Simpson <cs at cskk.id.au>

From tjreedy at udel.edu  Sun Mar 24 22:01:14 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 24 Mar 2019 22:01:14 -0400
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <20190324230036.GA72432@cskk.homeip.net>
References: <q78r9d$uvr$1@blaine.gmane.org>
 <20190324230036.GA72432@cskk.homeip.net>
Message-ID: <q79cpe$5b4d$1@blaine.gmane.org>

On 3/24/2019 7:00 PM, Cameron Simpson wrote:
> On 24Mar2019 17:02, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 3/24/2019 8:21 AM, Nick Coghlan wrote:
>>> * PEP 499: Binding "-m" executed modules under their module name as
>>> well as `__main__`
>>
>> My brief response: +1 unless there is a good reason not.
> 
> There turn out to be some subtle side effects. The test suite turned up 
> one (easily fixed) in pdb, but there are definitely some more things to 
> investigate.? Nick has pointed out pickle and the "python -i" option. 
> I'm digging into these. (Naturally, I have _never_ before used the pdb 
> or pickle modules, or the -i option :-)
> 
>> There have been multiple double module problems reported on 
>> python-list and likely stackoverflow.? And would there be any impact 
>> on circular imports?
> 
> Well, by binding the -m module to both __main__ and its name as denoted 
> on the command line one circular import is directly short circuited. 
> Aside from the -m module itself, I don't think there should be any other 
> direct effect on circular imports.
> 
> Did you have a specific scenario in mind?

I was thinking about IDLE and its tangled web of circular inports, but I 
am now convinced that this change will not affect it.  Indeed, 
idlelib/pyshell.py already implements idea of the proposal, ending with

if __name__ == "__main__":
     sys.modules['pyshell'] = sys.modules['__main__']
     main()

(It turns out that this fails for other reasons, which I am looking 
into.  The current recommendation is to start IDLE by runing any of 
__main__.py (via python -m idlelib), idle.py, idlew.py, or idle.bat.)

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Sun Mar 24 23:22:39 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 24 Mar 2019 23:22:39 -0400
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <q79cpe$5b4d$1@blaine.gmane.org>
References: <q78r9d$uvr$1@blaine.gmane.org>
 <20190324230036.GA72432@cskk.homeip.net> <q79cpe$5b4d$1@blaine.gmane.org>
Message-ID: <q79hi1$6vdj$1@blaine.gmane.org>

On 3/24/2019 10:01 PM, Terry Reedy wrote:
> On 3/24/2019 7:00 PM, Cameron Simpson wrote:

>> Did you have a specific scenario in mind?
> 
> I was thinking about IDLE and its tangled web of circular inports, but I 
> am now convinced that this change will not affect it.? Indeed, 
> idlelib/pyshell.py already implements idea of the proposal, ending with
> 
> if __name__ == "__main__":
>  ??? sys.modules['pyshell'] = sys.modules['__main__']
>  ??? main()

After more investigation, I realized that to stop having duplicate modulue:
1. The alias should be 'idlelib.pyshell', not 'pyshell', at least when 
imports are all absolute.
2. It should be done at the top of the file, before the import of 
modules that import pyshell.

If I run python f:/dev/3x/lib/idlelib/pyshell.py, the PEP patch would 
have to notice that pyshell is a module within idlelib and alias 
'__main__' to 'idlelib.pyshell', not 'pyshell'.  Would the same be true 
if within-package import were all relative?

> (It turns out that this fails for other reasons, which I am looking 
> into.

Since starting IDLE with pyshell once worked in the past, it appears to 
be because the startup command for run.py was outdated.  Will fix.

-- 
Terry Jan Reedy



From cs at cskk.id.au  Mon Mar 25 00:27:46 2019
From: cs at cskk.id.au (Cameron Simpson)
Date: Mon, 25 Mar 2019 15:27:46 +1100
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <q79hi1$6vdj$1@blaine.gmane.org>
References: <q79hi1$6vdj$1@blaine.gmane.org>
Message-ID: <20190325042746.GA22616@cskk.homeip.net>

On 24Mar2019 23:22, Terry Reedy <tjreedy at udel.edu> wrote:
>On 3/24/2019 10:01 PM, Terry Reedy wrote:
>>On 3/24/2019 7:00 PM, Cameron Simpson wrote:
>>>Did you have a specific scenario in mind?
>>
>>I was thinking about IDLE and its tangled web of circular inports, 
>>but I am now convinced that this change will not affect it.? Indeed, 
>>idlelib/pyshell.py already implements idea of the proposal, ending 
>>with
>>
>>if __name__ == "__main__":
>> ??? sys.modules['pyshell'] = sys.modules['__main__']
>> ??? main()
>
>After more investigation, I realized that to stop having duplicate modulue:
>1. The alias should be 'idlelib.pyshell', not 'pyshell', at least when 
>imports are all absolute.

The PEP499 patch effectively uses __main__.__spec__.name for the name of 
the alias. Does that simplify your issue?

The current PR is here if you want to look at it:

    https://github.com/python/cpython/pull/12490

>2. It should be done at the top of the file, before the import of 
>modules that import pyshell.

Hmm, if PEP499 comes in you shouldn't need to do this at all. If PEP499 
gets delayed or rejected I guess you're supporting this without it. Yes, 
you'll want to do it before any other imports happen (well, as you say, 
before any which import pyshell).

What about (untested):

    if __name__ == '__main__':
        if __spec__.name not in sys.modules:
            sys.modules[__spec__.name] = sys.modules['__main__']

as a forward compatible setup?

>If I run python f:/dev/3x/lib/idlelib/pyshell.py, the PEP patch would 
>have to notice that pyshell is a module within idlelib and alias 
>'__main__' to 'idlelib.pyshell', not 'pyshell'.  Would the same be true 
>if within-package import were all relative?

I think so because we're using .__spec__.name, which I though was post 
import name resolution.

Testing in my PEP499 branch:

Test 1:

    [~/src/cpython-cs at github(git:PEP499-cs)]fleet*> ./python.exe -i  Lib/idlelib/pyshell.py
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'run'
    >>> sys.modules['__main__']
    <module '__main__' (<_frozen_importlib_external.SourceFileLoader object at 0x1088e6040>)>
    >>> sys.modules['pyshell']
    <module '__main__' (<_frozen_importlib_external.SourceFileLoader object at 0x1088e6040>)>
    >>> sys.modules['idlelib.pyshell']
    <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>

So pyshell and idlelib.pyshell are distinct here. __main__ and pyshell 
are the same module, courtesy of your sys.modules assignment at the 
bottom of pyshell.py. Test 3 below will be with that commented out.

Test 2:

    [~/src/cpython-cs at github(git:PEP499-cs)]fleet*> PYTHONPATH=$PWD/Lib ./python.exe -i -m idlelib.pyshell
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'run'
    >>> sys.modules['__main__']
    <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
    >>> sys.modules['pyshell']
    <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
    >>> sys.modules['idlelib.pyshell']
    <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
    >>> id(sys.modules['__main__'])
    4551072712
    >>> id(sys.modules['pyshell'])
    4551072712
    >>> id(sys.modules['idlelib.pyshell'])
    4551072712

So this has __main__ and idlelib.pyshell the same module from the PEP499 
patch and pyshell also the same from your sys.modules assignment.

Test 3, with the pyshell.py sys.modules assignment commented out:

    [~/src/cpython-cs at github(git:PEP499-cs)]fleet*> PYTHONPATH=$PWD/Lib ./python.exe -i -m idlelib.pyshell
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'run'
    >>> sys.modules['__main__']
    <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
    >>> sys.modules['pyshell']
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    KeyError: 'pyshell'
    >>> sys.modules['idlelib.pyshell']
    <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
    >>> id(sys.modules['__main__'])
    4552379336
    >>> id(sys.modules['idlelib.pyshell'])
    4552379336

Here we've got __main__ and idlelib.pyshell the same module and no 
'pyshell' in sys.modules.

I don't think I understand your "relative import" scenario.

Cheers,
Cameron Simpson <cs at cskk.id.au>

From songofacandy at gmail.com  Mon Mar 25 00:39:11 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Mon, 25 Mar 2019 13:39:11 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
Message-ID: <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>

On Sun, Mar 24, 2019 at 8:57 PM Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> >
> > It was added "not be printed by default" version of DeprecationWarning.
> > But DeprecationWarning is not printed by default now.
>
> No, this was covered in PEP 565, and PendingDeprecationWarning was
> explicitly kept as a way of opting out of the revised semantics of
> DeprecationWarning.

I know PEP 565.  And I don't think difference between PendingDeprecationWarning
and DeprecationWarning is still too small to have two categories.  For example,

$ cat foo.py
import warnings
def foo():
    warnings.warn("foo", DeprecationWarning)

$ python3
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import foo
>>> foo.foo()
>>>
(no warning shown)

$ cat >bar.py
import foo
foo.foo()

$ python3 bar.py
(no warning shown)


I can't find I'm using deprecated APIs even when I'm using REPL.
When people want to check use of deprecated APIs, they need to
use -X dev or -Wd option anyway.

We have many ways to deprecation:

* Document only deprecation (no warning) -- no actual removal is planned.
* FutureWarning -- to warn end users.
* DeprecationWarning -- to warn Python developers.
* PendingDeprecationWarning -- to warn Python developers.


-- 
Inada Naoki  <songofacandy at gmail.com>

From tjreedy at udel.edu  Mon Mar 25 03:52:45 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 25 Mar 2019 03:52:45 -0400
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <20190325042746.GA22616@cskk.homeip.net>
References: <q79hi1$6vdj$1@blaine.gmane.org>
 <20190325042746.GA22616@cskk.homeip.net>
Message-ID: <09eceb89-a207-b65b-2990-18a1ace92a79@udel.edu>

On 3/25/2019 12:27 AM, Cameron Simpson wrote:
> On 24Mar2019 23:22, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 3/24/2019 10:01 PM, Terry Reedy wrote:
>>> On 3/24/2019 7:00 PM, Cameron Simpson wrote:
>>>> Did you have a specific scenario in mind?
>>>
>>> I was thinking about IDLE and its tangled web of circular inports, 
>>> but I am now convinced that this change will not affect it.? Indeed, 
>>> idlelib/pyshell.py already implements idea of the proposal, ending with
>>>
>>> if __name__ == "__main__":
>>> ??? sys.modules['pyshell'] = sys.modules['__main__']
>>> ??? main()
>>
>> After more investigation, I realized that to stop having duplicate 
>> modulue:
>> 1. The alias should be 'idlelib.pyshell', not 'pyshell', at least when 
>> imports are all absolute.
> 
> The PEP499 patch effectively uses __main__.__spec__.name for the name of 
> the alias. Does that simplify your issue?
> 
> The current PR is here if you want to look at it:
> 
>  ?? https://github.com/python/cpython/pull/12490

The new test passes on Win10.

>> 2. It should be done at the top of the file, before the import of 
>> modules that import pyshell.
> 
> Hmm, if PEP499 comes in you shouldn't need to do this at all. If PEP499 
> gets delayed or rejected I guess you're supporting this without it. Yes, 
> you'll want to do it before any other imports happen (well, as you say, 
> before any which import pyshell).
> 
> What about (untested):
> 
>  ?? if __name__ == '__main__':
>  ?????? if __spec__.name not in sys.modules:

When I start pyshell in my master repository directory on windows with
   python -m idlelib.pyshell
__spec__.name is 'idlelib.pyshell, which I currently hard-coded.
When I start with what should be equivalent
   python f:/dev/3x/lib/idlelib/pyshell.py
__spec__ is None and __spec__.name an attribute error.

>  ?????????? sys.modules[__spec__.name] = sys.modules['__main__']
> 
> as a forward compatible setup?
> 
>> If I run python f:/dev/3x/lib/idlelib/pyshell.py, the PEP patch would 
>> have to notice that pyshell is a module within idlelib and alias 
>> '__main__' to 'idlelib.pyshell', not 'pyshell'.? Would the same be 
>> true if within-package import were all relative?
> 
> I think so because we're using .__spec__.name, which I though was post 
> import name resolution.

You must be doing something different when __spec__ is None ;-).  I 
tested the patch and it does not raise AttributeError with the command 
above.

> Testing in my PEP499 branch:
> 
> Test 1:
> 
>  ?? [~/src/cpython-cs at github(git:PEP499-cs)]fleet*> ./python.exe -i  
> Lib/idlelib/pyshell.py
>  ?? Traceback (most recent call last):
>  ???? File "<string>", line 1, in <module>
>  ?? ModuleNotFoundError: No module named 'run'

This is because of an obsolete 'command = ...' around 420.  The if line 
is correct always and the if/then not needed.

>  ?? >>> sys.modules['__main__']
>  ?? <module '__main__' (<_frozen_importlib_external.SourceFileLoader 
> object at 0x1088e6040>)>
>  ?? >>> sys.modules['pyshell']
>  ?? <module '__main__' (<_frozen_importlib_external.SourceFileLoader 
> object at 0x1088e6040>)>
>  ?? >>> sys.modules['idlelib.pyshell']
>  ?? <module 'idlelib.pyshell' from 
> '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
> 
> So pyshell and idlelib.pyshell are distinct here.

I verified that the module was being executed twice by putting 
print('running') at the top.

  __main__ and pyshell
> are the same module, courtesy of your sys.modules assignment at the 
> bottom of pyshell.py.

Obsolete and removed.

  Test 3 below will be with that commented out.
> 
> Test 2:
> 
>  ?? [~/src/cpython-cs at github(git:PEP499-cs)]fleet*> PYTHONPATH=$PWD/Lib 
> ./python.exe -i -m idlelib.pyshell
>  ?? Traceback (most recent call last):
>  ???? File "<string>", line 1, in <module>
>  ?? ModuleNotFoundError: No module named 'run'
>  ?? >>> sys.modules['__main__']
>  ?? <module 'idlelib.pyshell' from 
> '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
>  ?? >>> sys.modules['pyshell']
>  ?? <module 'idlelib.pyshell' from 
> '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
>  ?? >>> sys.modules['idlelib.pyshell']
>  ?? <module 'idlelib.pyshell' from 
> '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
>  ?? >>> id(sys.modules['__main__'])
>  ?? 4551072712
>  ?? >>> id(sys.modules['pyshell'])
>  ?? 4551072712
>  ?? >>> id(sys.modules['idlelib.pyshell'])
>  ?? 4551072712
> 
> So this has __main__ and idlelib.pyshell the same module from the PEP499 
> patch and pyshell also the same from your sys.modules assignment.
> 
> Test 3, with the pyshell.py sys.modules assignment commented out:
> 
>  ?? [~/src/cpython-cs at github(git:PEP499-cs)]fleet*> PYTHONPATH=$PWD/Lib 
> ./python.exe -i -m idlelib.pyshell
>  ?? Traceback (most recent call last):
>  ???? File "<string>", line 1, in <module>
>  ?? ModuleNotFoundError: No module named 'run'
>  ?? >>> sys.modules['__main__']
>  ?? <module 'idlelib.pyshell' from 
> '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
>  ?? >>> sys.modules['pyshell']
>  ?? Traceback (most recent call last):
>  ???? File "<stdin>", line 1, in <module>
>  ?? KeyError: 'pyshell'
>  ?? >>> sys.modules['idlelib.pyshell']
>  ?? <module 'idlelib.pyshell' from 
> '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
>  ?? >>> id(sys.modules['__main__'])
>  ?? 4552379336
>  ?? >>> id(sys.modules['idlelib.pyshell'])
>  ?? 4552379336
> 
> Here we've got __main__ and idlelib.pyshell the same module and no 
> 'pyshell' in sys.modules.

> I don't think I understand your "relative import" scenario.

If files other that pyshell used relative 'import ./pyshell' instead of 
absolute 'import idlelib.pyshell', would the sys.modules key still be 
'idlelib.pyshell' or 'pyshell'?   Which is to ask, would the alias 
needed to avoid a second pyshell module still be 'idlelib.pyshell' or 
'pyshell'?



From cs at cskk.id.au  Mon Mar 25 06:30:22 2019
From: cs at cskk.id.au (Cameron Simpson)
Date: Mon, 25 Mar 2019 21:30:22 +1100
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <09eceb89-a207-b65b-2990-18a1ace92a79@udel.edu>
References: <09eceb89-a207-b65b-2990-18a1ace92a79@udel.edu>
Message-ID: <20190325103022.GA39362@cskk.homeip.net>

On 25Mar2019 03:52, Terry Reedy <tjreedy at udel.edu> wrote:
>On 3/25/2019 12:27 AM, Cameron Simpson wrote:
>>>>I was thinking about IDLE and its tangled web of circular inports, 
>>>>but I am now convinced that this change will not affect it.? Indeed, 
>>>>idlelib/pyshell.py already implements idea of the proposal, ending 
>>>>with
>>>>
>>>>if __name__ == "__main__":
>>>>??? sys.modules['pyshell'] = sys.modules['__main__']
>>>>??? main()
>>>
>>>After more investigation, I realized that to stop having duplicate 
>>>modulue:
>>>1. The alias should be 'idlelib.pyshell', not 'pyshell', at least 
>>>when imports are all absolute.
>>
>>The PEP499 patch effectively uses __main__.__spec__.name for the name 
>>of the alias. Does that simplify your issue?
[...]
>>What about (untested):
>>
>> ?? if __name__ == '__main__':
>> ?????? if __spec__.name not in sys.modules:
>> ?????????? sys.modules[__spec__.name] = sys.modules['__main__']
>
>When I start pyshell in my master repository directory on windows with
>  python -m idlelib.pyshell
>__spec__.name is 'idlelib.pyshell, which I currently hard-coded.
>When I start with what should be equivalent
>  python f:/dev/3x/lib/idlelib/pyshell.py
>__spec__ is None and __spec__.name an attribute error.

Um, yes. I presume that since no "import" has been done, there's no 
import spec (.__spec__).

Clearly the above needs to accomodate this, possibly with a fallback 
guess. Is sniffing the end components of __file__ at all sane? ending in 
idlelib/pyshell.py or pyshell.py? Or is that just getting baroque?

I don't think these are strictly the same from some kind of purist 
viewpoint: the path might be anything - _is_ it reasonable to suppose 
that it has a module name (== importable/finding through the import 
path)?

>>>If I run python f:/dev/3x/lib/idlelib/pyshell.py, the PEP patch would 
>>>have to notice that pyshell is a module within idlelib and alias 
>>>'__main__' to 'idlelib.pyshell', not 'pyshell'.? Would the same be 
>>>true if within-package import were all relative?
>>
>>I think so because we're using .__spec__.name, which I though was 
>>post import name resolution.
>
>You must be doing something different when __spec__ is None ;-).  I 
>tested the patch and it does not raise AttributeError with the command 
>above.

Indeed. I may have fudged a bit when I said "The PEP499 patch 
effectively uses __main__.__spec__.name". It modifies runpy.py's 
_run_module_as_main function, and that is called for the "python -m 
module_name" invocation, so it can get the module spec because it has a 
module name.

So the patch doesn't have to cope with __spec__ being None.

As you say, __spec__ is None for "python path/to/file.py" so __spec__ 
isn't any use there. Apologies.

[...]
>>Test 3, with the pyshell.py sys.modules assignment commented out:
>>
>> ?? [~/src/cpython-cs at github(git:PEP499-cs)]fleet*> PYTHONPATH=$PWD/Lib ./python.exe -i -m idlelib.pyshell
[...]
>> ?? >>> sys.modules['__main__']
>> ?? <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
>> ?? >>> sys.modules['pyshell']
>> ?? Traceback (most recent call last):
>> ???? File "<stdin>", line 1, in <module>
>> ?? KeyError: 'pyshell'
>> ?? >>> sys.modules['idlelib.pyshell']
>> ?? <module 'idlelib.pyshell' from '/Users/cameron/src/cpython-cs at github/Lib/idlelib/pyshell.py'>
>> ?? >>> id(sys.modules['__main__'])
>> ?? 4552379336
>> ?? >>> id(sys.modules['idlelib.pyshell'])
>> ?? 4552379336
>>
>>Here we've got __main__ and idlelib.pyshell the same module and no 
>>'pyshell' in sys.modules.
>
>>I don't think I understand your "relative import" scenario.
>
>If files other that pyshell used relative 'import ./pyshell' instead 
>of absolute 'import idlelib.pyshell', would the sys.modules key still 
>be 'idlelib.pyshell' or 'pyshell'?   Which is to ask, would the alias 
>needed to avoid a second pyshell module still be 'idlelib.pyshell' or 
>'pyshell'?

Ok.

As I understand it Python 3 imports are absolute: without a leading dot 
a name is absolute, so "import pyshell" should install 
sys.module['pyshell'] _provided_ that 'pyshell' can be found in the 
module search path.

Conversely, an "import .pyshell" is an import relative to the current 
module's package name, equivalent to an import of the absolute path 
"package.name.pyshell", for whatever the package name is. So (a) you can 
only import '.pyshell' from within a package containing a 'pyshell.py' 
file and (b) you can't import import '.pyshell' if you're not in a 
package.

I stuffed a "test2.py" into the local idlelib like this:

    import sys
    print("running", __file__, __name__)
    print(repr(sorted(sys.modules)))
    print(repr(sys.paht))
    from pyshell import idle_showwarning
    print(repr(sorted(sys.modules)))

and fiddled with the "from pyshell import idle_showwarning" line.

(I'm presuming this is what you have in mind, since "import ./pyshell" 
elicits a syntax error.)

Using "./python.exe -m idlelib.test2":

  Plain "pyshell" gets an ImportError - no such module.

  Using ".pyshell" imports the pyshell module as "idlelib.pyshell" in 
  sys.modules.

Which was encouraging until I went "./python.exe Lib/idlelib/test2.py".  
This puts Lib/idlelib (as an absolute path) at the start of sys.path.

A plain "pyshell" import works and installs sys.modules['pyshell'].

Conversely, trying the ".pyshell" import gets:

  ModuleNotFoundError: No module named '__main__.pyshell'; '__main__' is not a package

So we can get 'pyshell' or 'idlelib.pyshell' into sys.modules depending 
how we invoke python.

HOWEVER, if you're importing the 'pyshell' from idlelib _as found in the 
module search path_, whether absolutely as 'idlelib.pyshell' or 
relatives as '.pyshell' from within the idlelib package, you should 
always get 'idlelib.pyshell' in the sys.modules map.

And I don't think you should need to worry about a circular import 
importing some top level name 'pyshell' because that's not using the 
idlelib package, so I'd argue it isn't your problem.

Thoughts?

Cheers,
Cameron Simpson <cs at cskk.id.au>

From ncoghlan at gmail.com  Mon Mar 25 07:53:07 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 25 Mar 2019 21:53:07 +1000
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
Message-ID: <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>

On Mon, 25 Mar 2019 at 14:39, Inada Naoki <songofacandy at gmail.com> wrote:
> We have many ways to deprecation:
>
> * Document only deprecation (no warning) -- no actual removal is planned.
> * FutureWarning -- to warn end users.
> * DeprecationWarning -- to warn Python developers.
> * PendingDeprecationWarning -- to warn Python developers.

The key difference between the last two:

* DeprecationWarning: the decision has already been made, you have
little chance of getting the deprecation reversed
* PendingDeprecationWarning: full deprecation is being considered, now
is the time to get in touch if you'd like to avoid full deprecation

Hence the last one only showing up to those folks that are running
automated tests or otherwise enabling all warnings.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Mon Mar 25 08:51:05 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Mar 2019 13:51:05 +0100
Subject: [Python-Dev] Removing PendingDeprecationWarning
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
 <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
Message-ID: <20190325135105.7091ec67@fsol>

On Mon, 25 Mar 2019 21:53:07 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Mon, 25 Mar 2019 at 14:39, Inada Naoki <songofacandy at gmail.com> wrote:
> > We have many ways to deprecation:
> >
> > * Document only deprecation (no warning) -- no actual removal is planned.
> > * FutureWarning -- to warn end users.
> > * DeprecationWarning -- to warn Python developers.
> > * PendingDeprecationWarning -- to warn Python developers.  
> 
> The key difference between the last two:
> 
> * DeprecationWarning: the decision has already been made, you have
> little chance of getting the deprecation reversed
> * PendingDeprecationWarning: full deprecation is being considered, now
> is the time to get in touch if you'd like to avoid full deprecation

Are people actually aware of this difference?  Personally I am (was)
not.

Regards

Antoine.



From songofacandy at gmail.com  Mon Mar 25 09:11:57 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Mon, 25 Mar 2019 22:11:57 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
 <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
Message-ID: <CAEfz+Tzj5uh3K9yOF-k+1S1_s_32CSz2_Ad4kB6WH+Y7m_c1BA@mail.gmail.com>

On Mon, Mar 25, 2019 at 8:53 PM Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On Mon, 25 Mar 2019 at 14:39, Inada Naoki <songofacandy at gmail.com> wrote:
> > We have many ways to deprecation:
> >
> > * Document only deprecation (no warning) -- no actual removal is planned.
> > * FutureWarning -- to warn end users.
> > * DeprecationWarning -- to warn Python developers.
> > * PendingDeprecationWarning -- to warn Python developers.
>
> The key difference between the last two:
>
> * DeprecationWarning: the decision has already been made, you have
> little chance of getting the deprecation reversed
> * PendingDeprecationWarning: full deprecation is being considered, now
> is the time to get in touch if you'd like to avoid full deprecation
>
> Hence the last one only showing up to those folks that are running
> automated tests or otherwise enabling all warnings.
>
> Cheers,
> Nick.
>

PendingDeprecationWarning was added when DeprecationWarning is
shown by default and it is too noisy for end users.

After Python 2.7 and 3.2, there are no difference between two classes.

PEP 565 tried to introduce difference again.  But the difference is
still too little.

I have not seen DeprecationWarning enabled by PEP 565.
I see both warning when I'm using PYTHONWARNINGS=default or -Wd,
or running test.

I don't think it's worth enough to try giving meaning to
PendingDeprecationWarning
again.  C, Rust, Java, Ruby, PHP, don't have PendingDeprecation.
Programmers only need Deprecation.  Why programmers need PendingDeprecation
only in Python?

If it is really important, I think we need easy option to enable only
DeprecationWarning.
It must be as easy as -Wd.

-- 
Inada Naoki  <songofacandy at gmail.com>

From gzlist at googlemail.com  Mon Mar 25 11:12:00 2019
From: gzlist at googlemail.com (Martin (gzlist))
Date: Mon, 25 Mar 2019 15:12:00 +0000
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <543c3748-42d9-44bb-d18f-fe8e3c134584@python.org>
References: <5C948F55.4040108@canterbury.ac.nz>
 <q72gqf$81gc$1@blaine.gmane.org> <20190322123301.7489ae78@fsol>
 <543c3748-42d9-44bb-d18f-fe8e3c134584@python.org>
Message-ID: <CAOG+YdrUv8H2PXCuhGfF7q32BFtzhsH5oYv4fwATRb4G4tFmng@mail.gmail.com>

On Fri, 22 Mar 2019 at 16:12, Steve Dower <steve.dower at python.org> wrote:
>
> On 22Mar2019 0433, Antoine Pitrou wrote:
> > The question is: why would you use a array.array() with a Windows C API?
>
> I started replying to this with a whole lot of examples, and eventually
> convinced myself that you wouldn't (or shouldn't).
>
> That said, I see value in having a type for array/struct/memoryview that
> "is the same size as wchar_t", since that will avoid people having to
> guess (the docs for array in particular are deliberately vague about the
> actual size of the various types).

This is pretty much what ctypes provides for dealing with unicode?

https://docs.python.org/3/library/ctypes.html#ctypes.create_unicode_buffer

Seems a fine place to have things that help with win32 api interactions.

Martin

From steve.dower at python.org  Mon Mar 25 11:48:59 2019
From: steve.dower at python.org (Steve Dower)
Date: Mon, 25 Mar 2019 08:48:59 -0700
Subject: [Python-Dev] Replacement for array.array('u')?
In-Reply-To: <CAOG+YdrUv8H2PXCuhGfF7q32BFtzhsH5oYv4fwATRb4G4tFmng@mail.gmail.com>
References: <5C948F55.4040108@canterbury.ac.nz>
 <q72gqf$81gc$1@blaine.gmane.org> <20190322123301.7489ae78@fsol>
 <543c3748-42d9-44bb-d18f-fe8e3c134584@python.org>
 <CAOG+YdrUv8H2PXCuhGfF7q32BFtzhsH5oYv4fwATRb4G4tFmng@mail.gmail.com>
Message-ID: <e1fc1993-7389-cb3d-993f-93419e15ed98@python.org>

On 25Mar2019 0812, Martin (gzlist) wrote:
> On Fri, 22 Mar 2019 at 16:12, Steve Dower <steve.dower at python.org> wrote:
>>
>> On 22Mar2019 0433, Antoine Pitrou wrote:
>>> The question is: why would you use a array.array() with a Windows C API?
>>
>> I started replying to this with a whole lot of examples, and eventually
>> convinced myself that you wouldn't (or shouldn't).
>>
>> That said, I see value in having a type for array/struct/memoryview that
>> "is the same size as wchar_t", since that will avoid people having to
>> guess (the docs for array in particular are deliberately vague about the
>> actual size of the various types).
> 
> This is pretty much what ctypes provides for dealing with unicode?
> 
> https://docs.python.org/3/library/ctypes.html#ctypes.create_unicode_buffer
> 
> Seems a fine place to have things that help with win32 api interactions.

Sure, though there are other reasons to deal with "pure" data that would 
benefit from having the data type in array. I don't need to directly 
refer to an existing buffer in memory, just to easily create/reinterpret 
bytes (for which memoryview is often better, though it inherits its 
codes from struct, which has no 'u' code, which is probably why I end up 
using array instead ;) )

Also, I keep finding that every time I deploy Python these days, it's 
critical to remove ctypes to reduce the attack surface area (I'm getting 
it into more and more high value systems where the rules are more 
strict). So I'm a big fan of treating ctypes as optional.

Cheers,
Steve

From brett at python.org  Mon Mar 25 14:22:41 2019
From: brett at python.org (Brett Cannon)
Date: Mon, 25 Mar 2019 11:22:41 -0700
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <q78r9d$uvr$1@blaine.gmane.org>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
 <q78r9d$uvr$1@blaine.gmane.org>
Message-ID: <CAP1=2W4-MTa4==eorSyRrmXNqgSyoqE_bpR6UL2_C0ruy7yZZQ@mail.gmail.com>

On Sun, Mar 24, 2019 at 2:03 PM Terry Reedy <tjreedy at udel.edu> wrote:

> On 3/24/2019 8:21 AM, Nick Coghlan wrote:
>
> > We'll be announcing those appointments as we go, so I'm happy to
> > report that I will be handling the BDFL-Delegate responsibilities for
> > the following PEPs:
>
> Where do we discuss these?
>

Either as a topic on discuss.python.org or as a separate thread here on
python-dev.

-Brett


>
> If a delegate has a provisional view, it might help focus discussion if
> that were known.
>
> > * PEP 499: Binding "-m" executed modules under their module name as
> > well as `__main__`
>
> My brief response: +1 unless there is a good reason not.  There have
> been multiple double module problems reported on python-list and likely
> stackoverflow.  And would there be any impact on circular imports?
>
> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190325/8949a800/attachment.html>

From nad at python.org  Mon Mar 25 20:05:40 2019
From: nad at python.org (Ned Deily)
Date: Mon, 25 Mar 2019 20:05:40 -0400
Subject: [Python-Dev] [RELEASE] Python 3.7.3 is now available
Message-ID: <A942C00C-74D5-489D-93E2-0AA176F67CA8@python.org>

https://blog.python.org/2019/03/python-373-is-now-available.html

Python 3.7.3 is now available. Python 3.7.3 is the next
maintenance release of Python 3.7, the latest feature release of Python.
You can find Python 3.7.3 here:
   https://www.python.org/downloads/release/python-373/

See the What?s New In Python 3.7 document for more information about the
many new features and optimizations included in the 3.7 series. Detailed
information about the changes made in 3.7.3 can be found in its change log.

Thanks to all of the many volunteers who help make Python Development and
these releases possible! Please consider supporting our efforts by
volunteering yourself or through organization contributions to the Python
Software Foundation.

https://docs.python.org/3.7/whatsnew/3.7.html
https://docs.python.org/3.7/whatsnew/changelog.html#python-3-7-3-final
https://www.python.org/psf/

--
  Ned Deily
  nad at python.org -- []


From Peixing.Xin at windriver.com  Tue Mar 26 03:57:18 2019
From: Peixing.Xin at windriver.com (Xin, Peixing)
Date: Tue, 26 Mar 2019 07:57:18 +0000
Subject: [Python-Dev] Anyone can help to review these 2 PRs?
Message-ID: <8488FBC4EAAC5941BA4B85DD1ECCF1870133BA4559@ALA-MBD.corp.ad.wrs.com>

Hi, Experts:

I have 2 PRs to get review for days. Anyone can give your hands? Thanks in advance. BTW, they are about CPython working on VxWorks RTOS.

https://github.com/python/cpython/pull/12394
https://github.com/python/cpython/pull/12428

Thanks,
Peixing

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190326/ceeecc89/attachment.html>

From lukasz at langa.pl  Tue Mar 26 05:11:55 2019
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Tue, 26 Mar 2019 10:11:55 +0100
Subject: [Python-Dev] [RELEASE] Python 3.8.0a3 is now available for testing
Message-ID: <17D69CA4-6E33-4FA4-BFF8-0642EC620639@langa.pl>

It's time for the third alpha of Python 3.8.0. Go get it here:
https://www.python.org/downloads/release/python-380a3/

Python 3.8.0a3 is the third of four planned alpha releases of Python 3.8,
the next feature release of Python.  During the alpha phase, Python 3.8
remains under heavy development: additional features will be added
and existing features may be modified or deleted.  Please keep in mind
that this is a preview release and its use is not recommended for
production environments.  The last alpha release, 3.8.0a4, is planned
for 2019-04-29.

I am happy to say that this time all buildbots were properly green. Thank you to everybody who worked on that.

- ?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190326/1a72a4c7/attachment.sig>

From stephane at wirtel.be  Tue Mar 26 07:23:26 2019
From: stephane at wirtel.be (=?UTF-8?Q?St=c3=a9phane_Wirtel?=)
Date: Tue, 26 Mar 2019 12:23:26 +0100
Subject: [Python-Dev] [RELEASE] Python 3.8.0a3 is now available for
 testing
In-Reply-To: <17D69CA4-6E33-4FA4-BFF8-0642EC620639@langa.pl>
References: <17D69CA4-6E33-4FA4-BFF8-0642EC620639@langa.pl>
Message-ID: <1cc7d757-6fc3-fc75-6ac3-f26357e7c3cb@wirtel.be>

?ukasz,

I have proposed a PR for the update of this docker image:
quay.io/python-devs/ci-image

I have also updated the versions to 3.4.10, 3.5.7, 3.7.3 and of course
to Python 3.8.0a3.

By the way, this image will contain the last versions of Python 3.x

https://gitlab.com/python-devs/ci-images/merge_requests/9

Bonne journ?e,

St?phane

Le 26/03/19 ? 10:11, ?ukasz Langa a ?crit?:
> It's time for the third alpha of Python 3.8.0. Go get it here:
> https://www.python.org/downloads/release/python-380a3/
> 
> Python 3.8.0a3 is the third of four planned alpha releases of Python 3.8,
> the next feature release of Python.  During the alpha phase, Python 3.8
> remains under heavy development: additional features will be added
> and existing features may be modified or deleted.  Please keep in mind
> that this is a preview release and its use is not recommended for
> production environments.  The last alpha release, 3.8.0a4, is planned
> for 2019-04-29.
> 
> I am happy to say that this time all buildbots were properly green. Thank you to everybody who worked on that.

From barry at python.org  Tue Mar 26 12:44:45 2019
From: barry at python.org (Barry Warsaw)
Date: Tue, 26 Mar 2019 09:44:45 -0700
Subject: [Python-Dev] [RELEASE] Python 3.8.0a3 is now available for
 testing
In-Reply-To: <1cc7d757-6fc3-fc75-6ac3-f26357e7c3cb@wirtel.be>
References: <17D69CA4-6E33-4FA4-BFF8-0642EC620639@langa.pl>
 <1cc7d757-6fc3-fc75-6ac3-f26357e7c3cb@wirtel.be>
Message-ID: <FEEEC638-C460-4E6C-9A68-1FEA213BAD30@python.org>

On Mar 26, 2019, at 04:23, St?phane Wirtel <stephane at wirtel.be> wrote:
> 
> I have proposed a PR for the update of this docker image:
> quay.io/python-devs/ci-image
> 
> I have also updated the versions to 3.4.10, 3.5.7, 3.7.3 and of course
> to Python 3.8.0a3.
> 
> By the way, this image will contain the last versions of Python 3.x
> 
> https://gitlab.com/python-devs/ci-images/merge_requests/9

Thank you St?phane!  This is now merged and the updated image is available.

Cheers,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190326/801f5dc3/attachment.sig>

From stephane at wirtel.be  Tue Mar 26 13:15:57 2019
From: stephane at wirtel.be (Stephane Wirtel)
Date: Tue, 26 Mar 2019 18:15:57 +0100
Subject: [Python-Dev] [RELEASE] Python 3.8.0a3 is now available for
 testing
In-Reply-To: <FEEEC638-C460-4E6C-9A68-1FEA213BAD30@python.org>
References: <17D69CA4-6E33-4FA4-BFF8-0642EC620639@langa.pl>
 <1cc7d757-6fc3-fc75-6ac3-f26357e7c3cb@wirtel.be>
 <FEEEC638-C460-4E6C-9A68-1FEA213BAD30@python.org>
Message-ID: <20190326171557.GB11340@xps>

>Thank you St?phane!  This is now merged and the updated image is available.
Thank you to you Barry.

I have proposed an idea where we could generate the new image without
any human action, excepted when there is a fail with the build
step.


Cheers,

St?phane

-- 
St?phane Wirtel - https://wirtel.be - @matrixise

From songofacandy at gmail.com  Wed Mar 27 06:24:57 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Wed, 27 Mar 2019 19:24:57 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+Tzj5uh3K9yOF-k+1S1_s_32CSz2_Ad4kB6WH+Y7m_c1BA@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
 <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
 <CAEfz+Tzj5uh3K9yOF-k+1S1_s_32CSz2_Ad4kB6WH+Y7m_c1BA@mail.gmail.com>
Message-ID: <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>

On Mon, Mar 25, 2019 at 10:11 PM Inada Naoki <songofacandy at gmail.com> wrote:
>
> C, Rust, Java, Ruby, PHP, don't have PendingDeprecation.
> Programmers only need Deprecation.  Why programmers need PendingDeprecation
> only in Python?
>

Any comments about this?

I want to document PendingDeprecationWarning is deprecated [1].
May I merge the PR?  Should I start poll on discuss.python.org?

[1]: https://github.com/python/cpython/pull/12505/files

-- 
Inada Naoki  <songofacandy at gmail.com>

From encukou at gmail.com  Wed Mar 27 09:50:21 2019
From: encukou at gmail.com (Petr Viktorin)
Date: Wed, 27 Mar 2019 14:50:21 +0100
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
 <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
Message-ID: <CA+=+wqBsUKWHRqjhRFShmY77vaxDPTFaafWtOB8dnyXQSthiZw@mail.gmail.com>

On Sun, Mar 24, 2019 at 4:22 PM Mark Shannon <mark at hotpy.org> wrote:
>
> Hi Petr,
>
> Regarding PEPs 576 and 580.
> Over the new year, I did a thorough analysis of possible approaches to
> possible calling conventions for use in the CPython ecosystems and came
> up with a new PEP.
> The draft can be found here:
> https://github.com/markshannon/peps/blob/new-calling-convention/pep-9999.rst
>
> I was hoping to profile a branch with the various experimental changes
> cherry-picked together, but don't seemed to have found the time :(
>
> I'd like to have a testable branch, before formally submitting the PEP,
> but I'd thought you should be aware of the PEP.
>
> Cheers,
> Mark.

Hello Mark,
Thank you for letting me know! I wish I knew of this back in January,
when you committed the first draft. This is unfair to the competing
PEP, which is ready and was waiting for the new govenance. We have
lost three months that could be spent pondering the ideas in the
pre-PEP.
Do you think you will find the time to piece things together? Is there
anything that you already know should be changed?

Do you have any comments on [Jeroen's comparison]?

The pre-PEP is simpler then PEP 580, because it solves simpler issues.
I'll need to confirm that it won't paint us into a corner -- that
there's a way to address all the issues in PEP 579 in the future.

The pre-PEP claims speedups of 2% in initial experiments, with
expected overall performance gain of 4% for the standard benchmark
suite. That's pretty big.
As far as I can see, PEP 580 claims not much improvement in CPython,
but rather large improvements for extensions (Mistune with Cython).

The pre-PEP has a complication around offsetting arguments by 1 to
allow bound methods forward calls cheaply. I fear that this optimizes
for current usage with its limitations.
PEP 580's cc_parent allows bound methods to have access to the class,
and through that, the module object where they are defined and the
corresponding module state. To support this, vector calls would need a
two-argument offset.
(That seems to illustrate the main difference between the motivations
of the two PEPs: one focuses on extensibility; the other on optimizing
existing use cases.)

The pre-PEP's "any third-party class implementing the new call
interface will not be usable as a base class" looks quite limiting.



[Jeroen's comparison]:
https://mail.python.org/pipermail/python-dev/2018-July/154238.html

From J.Demeyer at UGent.be  Wed Mar 27 11:13:10 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Wed, 27 Mar 2019 16:13:10 +0100
Subject: [Python-Dev] PEP 576bis discussion
In-Reply-To: <CA+=+wqBsUKWHRqjhRFShmY77vaxDPTFaafWtOB8dnyXQSthiZw@mail.gmail.com>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
 <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
 <CA+=+wqBsUKWHRqjhRFShmY77vaxDPTFaafWtOB8dnyXQSthiZw@mail.gmail.com>
Message-ID: <5C9B9306.3050502@UGent.be>

By lack of a better name, I'm using the name PEP 576bis to refer to
https://github.com/markshannon/peps/blob/new-calling-convention/pep-9999.rst 
(This is why this should get a PEP number soon, even if the PEP is not 
completely done yet).

On 2019-03-27 14:50, Petr Viktorin wrote:
> The pre-PEP is simpler then PEP 580, because it solves simpler issues.
> I'll need to confirm that it won't paint us into a corner -- that
> there's a way to address all the issues in PEP 579 in the future.

One potential issue is calling bound methods (in the duck typing sense) 
when the LOAD_METHOD optimization is *not* used. This would happen for 
example when storing a bound method object somewhere and then calling it 
(possibly repeatedly). Perhaps that's not a very common thing and we 
should just live with that. However, since __self__ is part of the PEP 
580 protocol, it allows calling a bound method object without any 
performance penalty compared to calling the underlying function directly.

Similarly, a follow-up of PEP 580 could allow zero-overhead calling of 
static/class methods (I didn't put this in PEP 580 because it's already 
too long).

> As far as I can see, PEP 580 claims not much improvement in CPython,
> but rather large improvements for extensions (Mistune with Cython).

Cython is indeed the main reason for PEP 580.

> The pre-PEP has a complication around offsetting arguments by 1 to
> allow bound methods forward calls cheaply.

I honestly don't understand what this "offset by one" means or why it's 
useful. It should be better explained in the PEP.

> The pre-PEP's "any third-party class implementing the new call
> interface will not be usable as a base class" looks quite limiting.

I agree, this is pretty bad. However, I don't think that there is a need 
for this limitation. PEP 580 solves this by only inheriting the 
Py_TPFLAGS_HAVE_CCALL flag in specific cases. PEP 576bis could do 
something similar.

Finally, I don't agree with this sentence from PEP 576bis: PEP 580 is 
specifically targetted at function-like objects, and doesn't support 
other callables like classes, partial functions, or proxies.

It's true that classes are not supported (and I wonder how PEP 576bis 
deals with that, it would be good to explain that more explicitly) but 
other callables are not a problem.


Jeroen.

From steve.dower at python.org  Wed Mar 27 11:26:33 2019
From: steve.dower at python.org (Steve Dower)
Date: Wed, 27 Mar 2019 08:26:33 -0700
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
 <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
 <CAEfz+Tzj5uh3K9yOF-k+1S1_s_32CSz2_Ad4kB6WH+Y7m_c1BA@mail.gmail.com>
 <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>
Message-ID: <18ccdacf-8fc8-5130-b4ba-89df84e02987@python.org>

On 27Mar2019 0324, Inada Naoki wrote:
> On Mon, Mar 25, 2019 at 10:11 PM Inada Naoki <songofacandy at gmail.com> wrote:
>>
>> C, Rust, Java, Ruby, PHP, don't have PendingDeprecation.
>> Programmers only need Deprecation.  Why programmers need PendingDeprecation
>> only in Python?
>>
> 
> Any comments about this?
> 
> I want to document PendingDeprecationWarning is deprecated [1].
> May I merge the PR?  Should I start poll on discuss.python.org?
> 
> [1]: https://github.com/python/cpython/pull/12505/files

There are two questions here:

1. Are we going to stop using it in CPython and the standard library?
2. Do we want everyone else to stop using it for their own purposes?

I think the first one is easy for us to answer (and personally, I don't 
care which way we go with it). However, deciding to stop using it 
ourselves is not the same as deprecating the class. So in this case, we 
would document that it is no longer used in CPython but is still 
available for other projects.

The second question is harder to answer, and in the absence of input 
from those who are already using it (or the absence of evidence that 
nobody else is using it), the best we can do is evaluate how much of a 
maintenance burden the class is.

In my opinion, it is a very low maintenance burden, and a very low 
cognitive burden on users, and so the cost of deprecating it on 
third-parties who are using it vastly outweighs the cost of leaving it 
alone.

If someone can show that either no third-parties are using it, or all 
those that do will probably never explicitly support Python 3.8 or 
later, or all those that do would prefer to stop using it, then I'll 
happily change my mind here. But right now, it seems like deprecating it 
will cause an unknown amount of churn for minimal benefit.

At most, I'd document it as "this will probably not survive to the next 
major change release, even though we aren't planning to do one yet" 
(much like the Py_UNICODE APIs). Ironically, PendingDeprecationWarning 
seems a pretty accurate way of indicating this state.

Cheers,
Steve

From p.f.moore at gmail.com  Wed Mar 27 11:45:15 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Mar 2019 15:45:15 +0000
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <18ccdacf-8fc8-5130-b4ba-89df84e02987@python.org>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
 <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
 <CAEfz+Tzj5uh3K9yOF-k+1S1_s_32CSz2_Ad4kB6WH+Y7m_c1BA@mail.gmail.com>
 <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>
 <18ccdacf-8fc8-5130-b4ba-89df84e02987@python.org>
Message-ID: <CACac1F-bNawCZce_HJpToWPTh=_WPtZ1aFN43UpTOVRb4SyLvw@mail.gmail.com>

On Wed, 27 Mar 2019 at 15:29, Steve Dower <steve.dower at python.org> wrote:
> If someone can show that either no third-parties are using it, or all
> those that do will probably never explicitly support Python 3.8 or
> later, or all those that do would prefer to stop using it, then I'll
> happily change my mind here. But right now, it seems like deprecating it
> will cause an unknown amount of churn for minimal benefit.

FWIW, https://searchcode.com/?q=PendingDeprecationWarning&lan=19 says
there are about 6.786 results. I haven't looked extensively, but
Django seems to be in there. The msgpack library that is vendored by
pip (ironically, maintained by INADA-San) also uses it in one place.

Personally, I don't think deprecation is worth the disruption.
Paul

From brett at python.org  Wed Mar 27 12:58:21 2019
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Mar 2019 09:58:21 -0700
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
 <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
 <CAEfz+Tzj5uh3K9yOF-k+1S1_s_32CSz2_Ad4kB6WH+Y7m_c1BA@mail.gmail.com>
 <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>
Message-ID: <CAP1=2W693W0y0N-J7HFOUiEOkSXbYsJrOE-EjqRM1XAxJCdgvw@mail.gmail.com>

On Wed, Mar 27, 2019 at 3:26 AM Inada Naoki <songofacandy at gmail.com> wrote:

> On Mon, Mar 25, 2019 at 10:11 PM Inada Naoki <songofacandy at gmail.com>
> wrote:
> >
> > C, Rust, Java, Ruby, PHP, don't have PendingDeprecation.
> > Programmers only need Deprecation.  Why programmers need
> PendingDeprecation
> > only in Python?
> >
>
> Any comments about this?
>

You know my views on this. :) Leave it and we start to consistently use if
for deprecations that are more than a release away from removal (and for
those not reading discuss.python.org, I proposed over there introducing a
warnings._deprecate() function for the stdlib which will do the right thing
based on sys.version_info() and the stated start/end for the deprecation).

It might also make sense to hold off on anything official until we decide
on something like PEP 387 and how we want to officially communicate
deprecations.

-Brett


>
> I want to document PendingDeprecationWarning is deprecated [1].
> May I merge the PR?  Should I start poll on discuss.python.org?
>
> [1]: https://github.com/python/cpython/pull/12505/files
>
> --
> Inada Naoki  <songofacandy at gmail.com>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190327/32bbf300/attachment.html>

From vstinner at redhat.com  Wed Mar 27 13:48:59 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 27 Mar 2019 18:48:59 +0100
Subject: [Python-Dev] New Python Initialization API
Message-ID: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>

Hi,

I would like to add a new C API to initialize Python. I would like
your opinion on the whole API before making it public. The code is
already implemented. Doc of the new API:

   https://pythondev.readthedocs.io/init_config.html


To make the API public, _PyWstrList, _PyInitError, _PyPreConfig,
_PyCoreConfig and related functions should be made public.

By the way, I would suggest to rename "_PyCoreConfig" to just
"PyConfig" :-) I don't think that "core init" vs "main init" is really
relevant: more about that below.


Let's start with two examples using the new API.

Example of simple initialization to enable isolated mode:

    _PyCoreConfig config = _PyCoreConfig_INIT;
    config.isolated = 1;

    _PyInitError err = _Py_InitializeFromConfig(&config);
    if (_Py_INIT_FAILED(err)) {
        _Py_ExitInitError(err);
    }
    /* ... use Python API here ... */
    Py_Finalize();

Example using the pre-initialization to enable the UTF-8 Mode (and use the
"legacy" Py_Initialize() function):

    _PyPreConfig preconfig = _PyPreConfig_INIT;
    preconfig.utf8_mode = 1;

    _PyInitError err = _Py_PreInitialize(&preconfig);
    if (_Py_INIT_FAILED(err)) {
        _Py_ExitInitError(err);
    }

    /* at this point, Python will only speak UTF-8 */

    Py_Initialize();
    /* ... use Python API here ... */
    Py_Finalize();

Since November 2017, I'm refactoring the Python Initialization code to
cleanup the code and prepare a new ("better") API to configure Python
Initialization. I just fixed the last issues that Nick Coghlan asked
me to fix (add a pre-initialization step: done, fix mojibake: done).
My work is inspired by Nick Coghlan's PEP 432, but it is not
implementing it directly. I had other motivations than Nick even if we
are somehow going towards the same direction.

Nick wants to get a half-initialized Python ("core init"), configure
Python using the Python API and Python objects, and then finish the
implementation ("main init").

I chose a different approach: put *everything* into a single C
structure (_PyCoreConfig) using C types. Using the structure, you
should be able to do what Nick wanted to do, but with C rather than
Python. Nick: please tell me if I'm wrong :-)

This work is also connected to Eric Snow's work on sub-interpreters
(PEP 554) and moving global variables into structures. For example,
I'm using his _PyRuntime structure to store a new "preconfig" state
(pre-initialization configuration, more about that below).

In November 2017, when I started to work on the Python Initialization
(bpo-32030), I identified the following problems:

* Many parts of the code were interdependent
* Code executed early in Py_Main() used the Python API before the Python API
  was fully initialized. Like code parsing -W command line option which
  used PyUnicode_FromWideChar() and PyList_Append().
* Error handling used Py_FatalError() which didn't let the caller to decide
  how to handle the error. Moreover, exit() was used to exit Python,
whereas libpython shouldn't do that: a library should not exit the
whole process! (imagine when Python is embedded inside an application)

One year and a half later, I implemented the following solutions:

* Py_Main() and Py_Initialize() code has been reorganized to respect
  priorities between global configuration variables (ex:
  Py_IgnoreEnvironmentFlag), environment variables (ex: PYTHONPATH), command
  line arguments (ex: -X utf8), configuration files (ex: pyenv.cfg), and the
  new _PyPreConfig and _PyCoreConfig structures which store the whole
  configuration.
* Python Initialization no longer uses the Python API but only C types
  like wchar_t* strings, a new _PyWstrList structure and PyMem_RawMalloc()
  memory allocator (PyMem_Malloc() is no longer used during init).
* The code has been modified to use a new _PyInitError structure. The caller
  of the top function gets control to cleanup everything before handling the
  error (display a fatal error message or simply exit Python).

The new _PyCoreConfig structure has the top-priority and provides a single
structure for all configuration parameters.

It becomes possible to override the code computing the "path configuration"
like sys.path to fully control where Python looks to import modules. It
becomes possible to use an empty list of paths to only allow builtin modules.

A new "pre-initialization" steps is responsible to configure the bare minimum
before the Python initialization: memory allocators and encodings
(LC_CTYPE locale
and the UTF-8 mode). The LC_CTYPE is no longer coerced and the UTF-8 Mode is
no longer enabled automatically depending on the user configuration to prevent
mojibake. Previously, calling Py_DecodeLocale() to get a Unicode wchar_t*
string from a bytes wchar* string created mojibake when called before
Py_Initialize() if the LC_CTYPE locale was coerced and/or if the UTF-8 Mode was
enabled.

The pre-initialization step ensures that the encodings and memory allocators
are well defined *before* Py_Initialize() is called.

Since the new API is currently private, I didn't document it in
Python. Moreover, the code changed a lot last year :-) But it should
now be way more stable. I started to document it in a separated
webpage:

   https://pythondev.readthedocs.io/init_config.html

The plan is to put it in the Python documentation once it becomes public.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.

From alexander.belopolsky at gmail.com  Wed Mar 27 14:34:48 2019
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 27 Mar 2019 14:34:48 -0400
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
Message-ID: <CAP7h-xZANmsqjbdOY985YWwYZCMWgZ25qmg0GuTd-_4+WYK9Mg@mail.gmail.com>

> To make the API public, _PyWstrList, _PyInitError, _PyPreConfig,
> _PyCoreConfig and related functions should be made public.

Would you consider making _Py_UnixMain public as well?

It is useful for high level embedding and not trivial for 3rd parties to
reimplement.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190327/d1984f0f/attachment.html>

From steve.dower at python.org  Wed Mar 27 15:37:00 2019
From: steve.dower at python.org (Steve Dower)
Date: Wed, 27 Mar 2019 12:37:00 -0700
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
Message-ID: <70c245c7-8f40-a51b-934b-af958f7cd849@python.org>

On 27Mar2019 1048, Victor Stinner wrote:
> Since November 2017, I'm refactoring the Python Initialization code to
> cleanup the code and prepare a new ("better") API to configure Python
> Initialization. I just fixed the last issues that Nick Coghlan asked
> me to fix (add a pre-initialization step: done, fix mojibake: done).
> My work is inspired by Nick Coghlan's PEP 432, but it is not
> implementing it directly. I had other motivations than Nick even if we
> are somehow going towards the same direction.

I this this should be its own PEP, since as you say it is not 
implementing the only PEP we have (or alternatively, maybe you should 
collaborate to update PEP 432 so that it reflects what you think we 
ought to be implementing).

Having formal writeups of both ideas is important to help decide between 
the two. It's not good to overrule a PEP by pretending that your change 
isn't big enough to need its own.

(Not trying to devalue the work you've been doing so far, since it's 
great! But internal changes are one thing, while updating the public, 
documented interfaces deserves a more thorough process.)

Cheers,
Steve

From brett at python.org  Wed Mar 27 16:23:01 2019
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Mar 2019 13:23:01 -0700
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <70c245c7-8f40-a51b-934b-af958f7cd849@python.org>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <70c245c7-8f40-a51b-934b-af958f7cd849@python.org>
Message-ID: <CAP1=2W6+udqgYbXCvGQAg4tbuKdU8LfrDeCrrvwMoxtZZ7UN1g@mail.gmail.com>

On Wed, Mar 27, 2019 at 12:39 PM Steve Dower <steve.dower at python.org> wrote:

> On 27Mar2019 1048, Victor Stinner wrote:
> > Since November 2017, I'm refactoring the Python Initialization code to
> > cleanup the code and prepare a new ("better") API to configure Python
> > Initialization. I just fixed the last issues that Nick Coghlan asked
> > me to fix (add a pre-initialization step: done, fix mojibake: done).
> > My work is inspired by Nick Coghlan's PEP 432, but it is not
> > implementing it directly. I had other motivations than Nick even if we
> > are somehow going towards the same direction.
>
> I this this should be its own PEP, since as you say it is not
> implementing the only PEP we have (or alternatively, maybe you should
> collaborate to update PEP 432 so that it reflects what you think we
> ought to be implementing).
>

I agree that if this isn't doing what PEP 432 set out but going its own way
we should probably discuss in regards to 432.

-Brett


>
> Having formal writeups of both ideas is important to help decide between
> the two. It's not good to overrule a PEP by pretending that your change
> isn't big enough to need its own.
>
> (Not trying to devalue the work you've been doing so far, since it's
> great! But internal changes are one thing, while updating the public,
> documented interfaces deserves a more thorough process.)
>
> Cheers,
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190327/d211cfcc/attachment.html>

From J.Demeyer at UGent.be  Wed Mar 27 18:18:44 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Wed, 27 Mar 2019 23:18:44 +0100
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <CA+=+wqBsUKWHRqjhRFShmY77vaxDPTFaafWtOB8dnyXQSthiZw@mail.gmail.com>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
 <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
 <CA+=+wqBsUKWHRqjhRFShmY77vaxDPTFaafWtOB8dnyXQSthiZw@mail.gmail.com>
Message-ID: <5C9BF6C4.2060808@UGent.be>

On 2019-03-27 14:50, Petr Viktorin wrote:
> The pre-PEP claims speedups of 2% in initial experiments, with
> expected overall performance gain of 4% for the standard benchmark
> suite. That's pretty big.

I re-did my earlier benchmarks for PEP 580 and these are the results:
https://gist.github.com/jdemeyer/f0d63be8f30dc34cc989cd11d43df248

In general, the PEP 580 timings seem slightly better than vanilla 
CPython, similar to what Mark got. I'm speculating that the speedup in 
both cases comes from the removal of type checks and dispatching 
depending on that, and instead using a single protocol that directly 
does what needs to be done.


Jeroen.

From brett at python.org  Wed Mar 27 18:32:00 2019
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Mar 2019 15:32:00 -0700
Subject: [Python-Dev] Rejecting PEP 502
Message-ID: <CAP1=2W5qgv4MG7rxBcjX4TffeUvYGxakJ2HvsoWxitMS5LZLoQ@mail.gmail.com>

https://www.python.org/dev/peps/pep-0502/

Thanks to Mike G. Miller for the informational PEP, but since the PEP was
written with an opinionated voice and the details of the PEP aren't
critical as they should be captured by PEP 498  anyway, we didn't think it
was worth trying to touch the PEP up to change its voice.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190327/9b61418b/attachment.html>

From greg at krypto.org  Wed Mar 27 18:59:25 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 27 Mar 2019 15:59:25 -0700
Subject: [Python-Dev] PEP 556 threaded garbage collection & linear recursion
 in gc
Message-ID: <CAGE7PN+8o8AVMpXh84ZqH7cN69r9rr3yq=-enE5JaF6eXzi2aQ@mail.gmail.com>

https://www.python.org/dev/peps/pep-0556/

This PEP is currently Deferred as nobody is actively working on a test
implementation.

A situation came up the other day where I *believe* this could've helped.

Scenario (admittedly not one most environments run into): A Python process
with a C++ extension module implementing a threaded server (threads spawned
by C++) that could call back into CPython within server request handlers.
(ie: how all threaded servers tend to work regardless of core loop
implementation language)

Python code in the application had done something (unknown to me, I didn't
dive into their code) that built up large enough presumably nested or
recursive data structures that the garbage collector, when triggered, would
wind up in very deep recursion.  This caused a stack overflow as the C++
spawned threads were only being given a 256k stack (to conserve virtual
address space - there can potentially be a _ton_ of threads in this code).

That had a C++ stack trace 1000+ levels deep repeating the pattern of

...
    @     0x564d59bd21de         32  func_dealloc
    @     0x564d59bce0c1         32  cell_dealloc
    @     0x564d5839db41         48  tupledealloc
    @     0x564d59bd21de         32  func_dealloc
    @     0x564d59bce0c1         32  cell_dealloc
    @     0x564d5839db41         48  tupledealloc
...

If our gc were done on a thread of its own spawned by Python, with a
typical normal larger default stack size (8mb) this would've been less
likely to trigger a crash (though obviously still possible if the recursion
is linear).

I know, there are obvious workarounds to this situation of all sorts.  My
point is more that synchronously triggering gc _within_ a thread that
happens to invoke the periodic "hey, lets do another gc run" logic the eval
loop is undesirable.

I'm not making any plans to work on an implementation for this PEP,
deferred seems accurate. Just dropping a note that I'd still be interested
in seeing something other than synchronous gc in arbitrary threads happen
when a process has multiple threads.

*Another* take away from this is that it *appears* possible to cause our gc
to go into linear recursion.  Yuck!  I'd file an issue on that, but doing
so requires making some example code to construct such a scenario first...

Food for thought in case these kinds of things are something anyone else
has encountered.

-Greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190327/8b90cf69/attachment.html>

From tim.peters at gmail.com  Wed Mar 27 20:38:01 2019
From: tim.peters at gmail.com (Tim Peters)
Date: Wed, 27 Mar 2019 19:38:01 -0500
Subject: [Python-Dev] PEP 556 threaded garbage collection & linear
 recursion in gc
In-Reply-To: <CAGE7PN+8o8AVMpXh84ZqH7cN69r9rr3yq=-enE5JaF6eXzi2aQ@mail.gmail.com>
References: <CAGE7PN+8o8AVMpXh84ZqH7cN69r9rr3yq=-enE5JaF6eXzi2aQ@mail.gmail.com>
Message-ID: <CAExdVN=JCWJQOPUxd=ApSUDzBMm58FzKBeS0ELLBuEmvwVvdFQ@mail.gmail.com>

[Gregory P. Smith <greg at krypto.org>]
> ...
> A situation came up the other day where I believe this could've helped.
>
> Scenario (admittedly not one most environments run into): A Python process
> with a C++ extension module implementing a threaded server (threads
> spawned by C++) that could call back into CPython within server request
> handlers.  (ie: how all threaded servers tend to work regardless of core
> loop implementation language)
>
> Python code in the application had done something (unknown to me, I
> didn't dive into their code) that built up large enough presumably nested
> or recursive data structures that the garbage collector, when triggered,
> would wind up in very deep recursion.  This caused a stack overflow
>  as the C++ spawned threads were only being given a 256k stack (to
> conserve virtual address space - there can potentially be a _ton_ of
> threads in this code).
>
> That had a C++ stack trace 1000+ levels deep repeating the pattern of
>
> ...
>     @     0x564d59bd21de         32  func_dealloc
>     @     0x564d59bce0c1         32  cell_dealloc
>     @     0x564d5839db41         48  tupledealloc
>     @     0x564d59bd21de         32  func_dealloc
>     @     0x564d59bce0c1         32  cell_dealloc
>     @     0x564d5839db41         48  tupledealloc
> ...
>
> If our gc were done on a thread of its own spawned by Python, with a typical
> normal larger default stack size (8mb) this would've been less likely
> to trigger a crash (though obviously still possible if the recursion is linear).

Just noting that gc is probably a red herring in that scenario.  gc
(cleverly!) determines the set of trash objects without needing any
recursive graph crawls.  And gc doesn't deallocate anything - not
directly.  It simply applies Py_DECREF to trash objects, once for each
PyObject* found pointing to them.  _If_ deallocation occurs as a
result, it's the same as if the user had done `del` on the appropriate
object manually.  The recursion then occurs in the chain of
XXX_dealloc calls, which in turn do more Py_DECREFs of their own, and
which have nothing in particular to do with gc.  Approximately the
same stack depth would be hit in a Python built without gc if the user
did the same sequence of `del`s by hand to break trash cycles.

The good news is that the "trashcan" mechanism - which predates gc -
was introduced to limit call depth in the presence of some potentially
unbounded chains of dealloc calls.  So teach trashcan about the
pattern here, and the stack depth problem goes away, with or without
gc.

The bad news is that the traschcan mechanism is excruciating, a
long-time source of subtle bugs of its own :-(

Note:  without actual code to run, it's possible that trashcan is
_already_ trying to limit stack depth in this specific case, but the
stack size is too small to accommodate the largest depth of recursive
calls trashcan is willing to allow before interfering.  Or some bug
snuck into trashcan, or its invoking code, that causes it to fail to
work as intended due to some unknown detail of the code above.
There's just no way to guess without code to provoke it.


> Another take away from this is that it appears possible to cause our gc
> to go into linear recursion.

As above, it's probably not gc, but deallocation as a side effect of
Py_DECREF dropping a refcount to 0.  gc is just one way Py_DECREF can
get invoked.

From vstinner at redhat.com  Wed Mar 27 21:30:17 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 28 Mar 2019 02:30:17 +0100
Subject: [Python-Dev] PEP 587: Python Initialization Configuration
Message-ID: <CA+3bQGFePnvb9E0W8p0jZYo9nt9CHkKama7xzLi2vfMn5JSfAg@mail.gmail.com>

Hi,

Since Steve Dower asked me to write a formal PEP for my proposal of a
new C API to initialize Python, here you have!

This new PEP will be shortly rendered as HTML at:
https://www.python.org/dev/peps/pep-0587/

The Rationale might be a little bit weak at this point and there are
still 3 small open questions, but the whole PEP should give you a
better overview of my proposal than my previous email :-)

I chose to start with a new PEP rather than updating PEP 432. I didn't
feel able to just "update" the PEP 432, my proposal is different. The
main difference between my PEP 587 and Nick Coghlan's PEP 432 is that
my PEP 587 only allows to configure Python *before* its
initialization, whereas Nick wanted to give control between its
"Initializing" and "Initialized" initialization phases. Moreover, my
PEP only uses C types rather Nick wanted to use Python types.

By setting the private PyConfig._init_main field to 1, my PEP 587
allows to stop Python initialization early to get a bare minimum
working Python only with builtin types and no importlib. If I
understood correctly, with my PEP, it remains possible to *extend*
Python C API later to implement what Nick wants. Nick: please correct
me if I'm wrong :-)

Victor


PEP: 587
Title: Python Initialization Configuration
Author: Victor Stinner <vstinner at redhat.com>
Discussions-To: python-dev at python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Mar-2018
Python-Version: 3.8

Abstract
========

Add a new C API to configure the Python Initialization providing finer
control on the whole configuration and better error reporting.


Rationale
=========

Python is highly configurable but its configuration evolved organically:
configuration parameters is scattered all around the code using
different ways to set them (mostly global configuration variables and
functions).  A straightforward and reliable way to configure Python is
needed. Some configuration parameters are not accessible from the C API,
or not easily.

The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as
input whereas the Python filesystem encoding is set during the
initialization.


Python Initialization C API
===========================

This PEP proposes to add the following new structures, functions and
macros.

New structures (4):

* ``PyConfig``
* ``PyInitError``
* ``PyPreConfig``
* ``PyWideCharList``

New functions (8):

* ``Py_PreInitialize(config)``
* ``Py_PreInitializeFromArgs(config, argc, argv)``
* ``Py_PreInitializeFromWideArgs(config, argc, argv)``
* ``Py_InitializeFromConfig(config)``
* ``Py_InitializeFromArgs(config, argc, argv)``
* ``Py_InitializeFromWideArgs(config, argc, argv)``
* ``Py_UnixMain(argc, argv)``
* ``Py_ExitInitError(err)``

New macros (6):

* ``Py_INIT_OK()``
* ``Py_INIT_ERR(MSG)``
* ``Py_INIT_USER_ERR(MSG)``
* ``Py_INIT_NO_MEMORY()``
* ``Py_INIT_EXIT(EXITCODE)``
* ``Py_INIT_FAILED(err)``


PyWideCharList
--------------

``PyWideCharList`` is a list of ``wchar_t*`` strings.

Example to initialize a string from C static array::

    static wchar_t* argv[2] = {
        L"-c",
        L"pass",
    };
    PyWideCharList config_argv = PyWideCharList_INIT;
    config_argv.length = Py_ARRAY_LENGTH(argv);
    config_argv.items = argv;

``PyWideCharList`` structure fields:

* ``length`` (``Py_ssize_t``)
* ``items`` (``wchar_t**``)

If *length* is non-zero, *items* must be non-NULL and all strings must
be non-NULL.

.. note::
   The "WideChar" name comes from the existing Python C API. Example:
   ``PyUnicode_FromWideChar(const wchar_t *str, Py_ssize_t size)``.

PyInitError
-----------

``PyInitError`` is a structure to store an error message or an exit code
for the Python Initialization.

Example::

    PyInitError alloc(void **ptr, size_t size)
    {
        *ptr = PyMem_RawMalloc(size);
        if (*ptr == NULL) {
            return Py_INIT_NO_MEMORY();
        }
        return Py_INIT_OK();
    }

    int main(int argc, char **argv)
    {
        void *ptr;
        PyInitError err = alloc(&ptr, 16);
        if (Py_INIT_FAILED(err)) {
            Py_ExitInitError(err);
        }
        PyMem_Free(ptr);
        return 0;
    }

``PyInitError`` fields:

* ``exitcode`` (``int``): if greater or equal to zero, argument passed to
  ``exit()``
* ``msg`` (``const char*``): error message
* ``prefix`` (``const char*``): string displayed before the message,
  usually rendered as ``prefix: msg``.
* ``user_err`` (int): if non-zero, the error is caused by the user
  configuration, otherwise it's an internal Python error.

Macro to create an error:

* ``Py_INIT_OK()``: success
* ``Py_INIT_NO_MEMORY()``: memory allocation failure (out of memory)
* ``Py_INIT_ERR(MSG)``: Python internal error
* ``Py_INIT_USER_ERR(MSG)``: error caused by user configuration
* ``Py_INIT_EXIT(STATUS)``: exit Python with the specified status

Other macros and functions:

* ``Py_INIT_FAILED(err)``: Is the result an error or an exit?
* ``Py_ExitInitError(err)``: call ``exit(status)`` for
  an error created by ``Py_INIT_EXIT(status)``,
  call ``Py_FatalError(msg)`` for other errors. Must not be called for
  an error created by ``Py_INIT_OK()``.

Pre-Initialization with PyPreConfig
-----------------------------------

``PyPreConfig`` structure is used to pre-initialize Python:

* Set the memory allocator
* Configure the LC_CTYPE locale
* Set the UTF-8 mode

Example using the pre-initialization to enable the UTF-8 Mode::

    PyPreConfig preconfig = PyPreConfig_INIT;
    preconfig.utf8_mode = 1;

    PyInitError err = Py_PreInitialize(&preconfig);
    if (Py_INIT_FAILED(err)) {
        Py_ExitInitError(err);
    }

    /* at this point, Python will speak UTF-8 */

    Py_Initialize();
    /* ... use Python API here ... */
    Py_Finalize();

Functions to pre-initialize Python:

* ``PyInitError Py_PreInitialize(const PyPreConfig *config)``
* ``PyInitError Py_PreInitializeFromArgs( const PyPreConfig *config,
int argc, char **argv)``
* ``PyInitError Py_PreInitializeFromWideArgs( const PyPreConfig
*config, int argc, wchar_t **argv)``

These functions can be called with *config* set to ``NULL``. The caller
is responsible to handler error using ``Py_INIT_FAILED()`` and
``Py_ExitInitError()``.

``PyPreConfig`` fields:

* ``allocator``: name of the memory allocator (ex: ``"malloc"``)
* ``coerce_c_locale_warn``: if non-zero, emit a warning if the C locale
  is coerced.
* ``coerce_c_locale``: if equals to 2, coerce the C locale; if equals to
  1, read the LC_CTYPE to decide if it should be coerced.
* ``dev_mode``: see ``PyConfig.dev_mode``
* ``isolated``: see ``PyConfig.isolated``
* ``legacy_windows_fs_encoding`` (Windows only): if non-zero, set the
  Python filesystem encoding to ``"mbcs"``.
* ``use_environment``: see ``PyConfig.use_environment``
* ``utf8_mode``: if non-zero, enable the UTF-8 mode

The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are
disabled by default in ``PyPreConfig``. Set ``coerce_c_locale``,
``coerce_c_locale_warn`` and ``utf8_mode`` to ``-1`` to let Python
enable them depending on the user configuration.

Initialization with PyConfig
----------------------------

The ``PyConfig`` structure contains all parameters to configure Python.

Example of Python initialization enabling the isolated mode::

    PyConfig config = PyConfig_INIT;
    config.isolated = 1;

    PyInitError err = Py_InitializeFromConfig(&config);
    if (Py_INIT_FAILED(err)) {
        Py_ExitInitError(err);
    }
    /* ... use Python API here ... */
    Py_Finalize();


Functions to initialize Python:

* ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``
* ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int
argc, char **argv)``
* ``PyInitError Py_InitializeFromWideArgs(const PyConfig *config, int
argc, wchar_t **argv)``

These functions can be called with *config* set to ``NULL``. The caller
is responsible to handler error using ``Py_INIT_FAILED()`` and
``Py_ExitInitError()``.

PyConfig fields:

* ``argv``: ``sys.argv``
* ``base_exec_prefix``: ``sys.base_exec_prefix``
* ``base_prefix``: ``sys.base_prefix``
* ``buffered_stdio``: if equals to 0, enable unbuffered mode, make
  stdout and stderr streams to be unbuffered.
* ``bytes_warning``: if equals to 1, issue a warning when comparing
  ``bytes`` or ``bytearray`` with ``str``, or comparing ``bytes`` with
  ``int``. If equal or greater to 2, raise a ``BytesWarning`` exception.
* ``dll_path`` (Windows only): Windows DLL path
* ``dump_refs``: if non-zero, display all objects still alive at exit
* ``exec_prefix``: ``sys.exec_prefix``
* ``executable``: ``sys.executable``
* ``faulthandler``: if non-zero, call ``faulthandler.enable()``
* ``filesystem_encoding``: Filesystem encoding,
  ``sys.getfilesystemencoding()``
* ``filesystem_errors``: Filesystem encoding errors,
  ``sys.getfilesystemencodeerrors()``
* ``hash_seed``, ``use_hash_seed``: randomized hash function seed
* ``home``: Python home
* ``import_time``: if non-zero, profile import time
* ``inspect``: enter interactive mode after executing a script or a
  command
* ``install_signal_handlers``: install signal handlers?
* ``interactive``: interactive mode
* ``legacy_windows_stdio`` (Windows only): if non-zero, use
  ``io.FileIO`` instead of ``WindowsConsoleIO`` for ``sys.stdin``,
  ``sys.stdout`` and ``sys.stderr``.
* ``malloc_stats``: if non-zero, dump memory allocation statistics
  at exit
* ``module_search_path_env``: ``PYTHONPATH`` environment variale value
* ``module_search_paths``, ``use_module_search_paths``: ``sys.path``
* ``optimization_level``: compilation optimization level
* ``parser_debug``: if non-zero, turn on parser debugging output (for
  expert only, depending on compilation options).
* ``prefix``: ``sys.prefix``
* ``program_name``: Program name
* ``program``: ``argv[0]`` or an empty string
* ``pycache_prefix``: ``.pyc`` cache prefix
* ``quiet``: quiet mode (ex: don't display the copyright and version
  messages even in interactive mode)
* ``run_command``: ``-c COMMAND`` argument
* ``run_filename``: ``python3 SCRIPT`` argument
* ``run_module``: ``python3 -m MODULE`` argument
* ``show_alloc_count``: show allocation counts at exit
* ``show_ref_count``: show total reference count at exit
* ``site_import``: import the ``site`` module at startup?
* ``skip_source_first_line``: skip the first line of the source
* ``stdio_encoding``, ``stdio_errors``: encoding and encoding errors of
  ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``
* ``tracemalloc``: if non-zero, call ``tracemalloc.start(value)``
* ``user_site_directory``: if non-zero, add user site directory to
  ``sys.path``
* ``verbose``: if non-zero, enable verbose mode
* ``warnoptions``: options of the ``warnings`` module to build filters
* ``write_bytecode``: if non-zero, write ``.pyc`` files
* ``xoptions``: ``sys._xoptions``

There are also private fields which are for internal-usage only:

* ``_check_hash_pycs_mode``
* ``_frozen``
* ``_init_main``
* ``_install_importlib``

New Py_UnixMain() function
--------------------------

Python 3.7 provides a high-level ``Py_Main()`` function which requires
to pass command line arguments as ``wchar_t*`` strings. It is
non-trivial to use the correct encoding to decode bytes. Python has its
own set of issues with C locale coercion and UTF-8 Mode.

This PEP adds a new ``Py_UnixMain()`` function which takes command line
arguments as bytes::

    int Py_UnixMain(int argc, char **argv)

Memory allocations and Py_DecodeLocale()
----------------------------------------

New pre-initialization and initialization APIs use constant
``PyPreConfig`` or ``PyConfig`` structures. If memory is allocated
dynamically, the caller is responsible to release it.  Using static
strings is just fine.

Python memory allocation functions like ``PyMem_RawMalloc()`` must not
be used before Python pre-initialization.  Using ``malloc()`` and
``free()`` is always safe.

``Py_DecodeLocale()`` must only be used after the pre-initialization.


XXX Open Questions
==================

This PEP is still a draft with open questions which should be answered:

* Do we need to add an API for import ``inittab``?
* What about the stable ABI? Should we add a version into
  ``PyPreConfig`` and ``PyConfig`` structures somehow? The Windows API
  is known for its ABI stability and it stores the structure size into
  the structure directly. Do the same?
* The PEP 432 stores ``PYTHONCASEOK`` into the config. Do we need
  to add something for that into ``PyConfig``? How would it be exposed
  at the Python level for ``importlib``? Passed as an argument to
  ``importlib._bootstrap._setup()`` maybe?


Backwards Compatibility
=======================

This PEP only adds a new API: it leaves the existing API unchanged and
has no impact on the backwards compatibility.


Alternative: PEP 432
====================

This PEP is inspired by Nick Coghlan's PEP 432 with a main difference:
it only allows to configure Python before its initialization.

The PEP 432 uses three initialization phases: Pre-Initialization,
Initializing, Initialized. It is possible to configure Python between
Initializing and Initialized phases using Python objects.

This PEP only uses C types like ``int`` and ``wchar_t*`` (and
``PyWideCharList`` structure). All parameters must be configured at once
before the Python initialization using the ``PyConfig`` structure.


Annex: Python Configuration
===========================

Priority and Rules
------------------

Priority of configuration parameters, highest to lowest:

* ``PyConfig``
* ``PyPreConfig``
* Configuration files
* Command line options
* Environment variables
* Global configuration variables

Priority of warning options, highest to lowest:

* ``PyConfig.warnoptions``
* ``PyConfig.dev_mode`` (add ``"default"``)
* ``PYTHONWARNINGS`` environment variables
* ``-W WARNOPTION`` command line argument
* ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater
  than 1, or add ``"default::BytesWarning``)

Rules on ``PyConfig`` and ``PyPreConfig`` parameters:

* If ``isolated`` is non-zero, ``use_environment`` and
  ``user_site_directory`` are set to 0
* If ``legacy_windows_fs_encoding`` is non-zero, ``utf8_mode`` is set to
  0
* If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``,
  ``faulthandler`` is set to 1, and ``"default"`` filter is added to
  ``warnoptions``. But ``PYTHONMALLOC`` has the priority over
  ``dev_mode`` to set the memory allocator.

Configuration Files
-------------------

Python configuration files:

* ``pyvenv.cfg``
* ``python._pth`` (Windows only)
* ``pybuilddir.txt`` (Unix only)

Global Configuration Variables
------------------------------

Global configuration variables mapped to ``PyPreConfig`` fields:

========================================  ================================
Variable                                  Field
========================================  ================================
``Py_LegacyWindowsFSEncodingFlag``        ``legacy_windows_fs_encoding``
``Py_LegacyWindowsFSEncodingFlag``        ``legacy_windows_fs_encoding``
``Py_UTF8Mode``                           ``utf8_mode``
``Py_UTF8Mode``                           ``utf8_mode``
========================================  ================================

Global configuration variables mapped to ``PyConfig`` fields:

========================================  ================================
Variable                                  Field
========================================  ================================
``Py_BytesWarningFlag``                   ``bytes_warning``
``Py_DebugFlag``                          ``parser_debug``
``Py_DontWriteBytecodeFlag``              ``write_bytecode``
``Py_FileSystemDefaultEncodeErrors``      ``filesystem_errors``
``Py_FileSystemDefaultEncoding``          ``filesystem_encoding``
``Py_FrozenFlag``                         ``_frozen``
``Py_HasFileSystemDefaultEncoding``       ``filesystem_encoding``
``Py_HashRandomizationFlag``              ``use_hash_seed``, ``hash_seed``
``Py_IgnoreEnvironmentFlag``              ``use_environment``
``Py_InspectFlag``                        ``inspect``
``Py_InteractiveFlag``                    ``interactive``
``Py_IsolatedFlag``                       ``isolated``
``Py_LegacyWindowsStdioFlag``             ``legacy_windows_stdio``
``Py_NoSiteFlag``                         ``site_import``
``Py_NoUserSiteDirectory``                ``user_site_directory``
``Py_OptimizeFlag``                       ``optimization_level``
``Py_QuietFlag``                          ``quiet``
``Py_UnbufferedStdioFlag``                ``buffered_stdio``
``Py_VerboseFlag``                        ``verbose``
``_Py_HasFileSystemDefaultEncodeErrors``  ``filesystem_errors``
``Py_BytesWarningFlag``                   ``bytes_warning``
``Py_DebugFlag``                          ``parser_debug``
``Py_DontWriteBytecodeFlag``              ``write_bytecode``
``Py_FileSystemDefaultEncodeErrors``      ``filesystem_errors``
``Py_FileSystemDefaultEncoding``          ``filesystem_encoding``
``Py_FrozenFlag``                         ``_frozen``
``Py_HasFileSystemDefaultEncoding``       ``filesystem_encoding``
``Py_HashRandomizationFlag``              ``use_hash_seed``, ``hash_seed``
``Py_IgnoreEnvironmentFlag``              ``use_environment``
``Py_InspectFlag``                        ``inspect``
``Py_InteractiveFlag``                    ``interactive``
``Py_IsolatedFlag``                       ``isolated``
``Py_LegacyWindowsStdioFlag``             ``legacy_windows_stdio``
``Py_NoSiteFlag``                         ``site_import``
``Py_NoUserSiteDirectory``                ``user_site_directory``
``Py_OptimizeFlag``                       ``optimization_level``
``Py_QuietFlag``                          ``quiet``
``Py_UnbufferedStdioFlag``                ``buffered_stdio``
``Py_VerboseFlag``                        ``verbose``
``_Py_HasFileSystemDefaultEncodeErrors``  ``filesystem_errors``
========================================  ================================


``Py_LegacyWindowsFSEncodingFlag`` and ``Py_LegacyWindowsStdioFlag`` are
only available on Windows.

Command Line Arguments
----------------------

Usage::

    python3 [options]
    python3 [options] -c COMMAND
    python3 [options] -m MODULE
    python3 [options] SCRIPT


Command line options mapped to pseudo-action on ``PyConfig`` fields:

================================  ================================
Option                            ``PyConfig`` field
================================  ================================
``-b``                            ``bytes_warning++``
``-B``                            ``write_bytecode = 0``
``-c COMMAND``                    ``run_module = COMMAND``
``--check-hash-based-pycs=MODE``  ``_check_hash_pycs_mode = MODE``
``-d``                            ``parser_debug++``
``-E``                            ``use_environment = 0``
``-i``                            ``inspect++`` and ``interactive++``
``-I``                            ``isolated = 1``
``-m MODULE``                     ``run_module = MODULE``
``-O``                            ``optimization_level++``
``-q``                            ``quiet++``
``-R``                            ``use_hash_seed = 0``
``-s``                            ``user_site_directory = 0``
``-S``                            ``site_import``
``-t``                            ignored (kept for backwards compatibility)
``-u``                            ``buffered_stdio = 0``
``-v``                            ``verbose++``
``-W WARNING``                    add ``WARNING`` to ``warnoptions``
``-x``                            ``skip_source_first_line = 1``
``-X XOPTION``                    add ``XOPTION`` to ``xoptions``
================================  ================================

``-h``, ``-?`` and ``-V`` options are handled outside ``PyConfig``.

Environment Variables
---------------------

Environment variables mapped to ``PyPreConfig`` fields:

=================================  =============================================
Variable                           ``PyPreConfig`` field
=================================  =============================================
``PYTHONCOERCECLOCALE``            ``coerce_c_locale``, ``coerce_c_locale_warn``
``PYTHONDEVMODE``                  ``dev_mode``
``PYTHONLEGACYWINDOWSFSENCODING``  ``legacy_windows_fs_encoding``
``PYTHONMALLOC``                   ``allocator``
``PYTHONUTF8``                     ``utf8_mode``
=================================  =============================================

Environment variables mapped to ``PyConfig`` fields:

=================================  ====================================
Variable                           ``PyConfig`` field
=================================  ====================================
``PYTHONDEBUG``                    ``parser_debug``
``PYTHONDEVMODE``                  ``dev_mode``
``PYTHONDONTWRITEBYTECODE``        ``write_bytecode``
``PYTHONDUMPREFS``                 ``dump_refs``
``PYTHONEXECUTABLE``               ``program_name``
``PYTHONFAULTHANDLER``             ``faulthandler``
``PYTHONHASHSEED``                 ``use_hash_seed``, ``hash_seed``
``PYTHONHOME``                     ``home``
``PYTHONINSPECT``                  ``inspect``
``PYTHONIOENCODING``               ``stdio_encoding``, ``stdio_errors``
``PYTHONLEGACYWINDOWSSTDIO``       ``legacy_windows_stdio``
``PYTHONMALLOCSTATS``              ``malloc_stats``
``PYTHONNOUSERSITE``               ``user_site_directory``
``PYTHONOPTIMIZE``                 ``optimization_level``
``PYTHONPATH``                     ``module_search_path_env``
``PYTHONPROFILEIMPORTTIME``        ``import_time``
``PYTHONPYCACHEPREFIX,``           ``pycache_prefix``
``PYTHONTRACEMALLOC``              ``tracemalloc``
``PYTHONUNBUFFERED``               ``buffered_stdio``
``PYTHONVERBOSE``                  ``verbose``
``PYTHONWARNINGS``                 ``warnoptions``
=================================  ====================================

``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are
specific to Windows.

``PYTHONDEVMODE`` is mapped to ``PyPreConfig.dev_mode`` and
``PyConfig.dev_mode``.


Annex: Python 3.7 API
=====================

Python 3.7 has 4 functions in its C API to initialize and finalize
Python:

* ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python
* ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python

Python can be configured using scattered global configuration variables
(like ``Py_IgnoreEnvironmentFlag``) and using the following functions:

* ``PyImport_AppendInittab()``
* ``PyImport_ExtendInittab()``
* ``PyMem_SetAllocator()``
* ``PyMem_SetupDebugHooks()``
* ``PyObject_SetArenaAllocator()``
* ``Py_SetPath()``
* ``Py_SetProgramName()``
* ``Py_SetPythonHome()``
* ``Py_SetStandardStreamEncoding()``
* ``PySys_AddWarnOption()``
* ``PySys_AddXOption()``
* ``PySys_ResetWarnOptions()``

There is also a high-level ``Py_Main()`` function.


Copyright
=========

This document has been placed in the public domain.

From vstinner at redhat.com  Wed Mar 27 21:34:39 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 28 Mar 2019 02:34:39 +0100
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <CAP7h-xZANmsqjbdOY985YWwYZCMWgZ25qmg0GuTd-_4+WYK9Mg@mail.gmail.com>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <CAP7h-xZANmsqjbdOY985YWwYZCMWgZ25qmg0GuTd-_4+WYK9Mg@mail.gmail.com>
Message-ID: <CA+3bQGEtmE2vB-bEbrHCDgKCJ=YRPPedwCYANgE4fshBzSQ1Dw@mail.gmail.com>

Le mer. 27 mars 2019 ? 19:35, Alexander Belopolsky
<alexander.belopolsky at gmail.com> a ?crit :
> Would you consider making _Py_UnixMain public as well?
>
> It is useful for high level embedding and not trivial for 3rd parties to reimplement.

Right, Py_Main() is causing a lot of practice issues, especially
mojibake because of the C locale coercion (PEP 538) and UTF-8 Mode
(PEP 540): both added in Python 3.7. I added that to the Rationale of
my PEP 587.

I just fixed the mojibake issue in Python 3.8 by disabling C locale
coercion and UTF-8 Mode by default. I'm not sure if nor how Python 3.7
should be fixed in a minor 3.7.x release.

Making _Py_UnixMain() public has already been discussed here:
https://discuss.python.org/t/adding-char-based-apis-for-unix/916

My PEP 587 allows to pass command line arguments as bytes (char*) or
Unicode (wchar_t*).

Ok, I just added Py_UnixMain() to the PEP (just make it part of the public API).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From vstinner at redhat.com  Wed Mar 27 21:41:43 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 28 Mar 2019 02:41:43 +0100
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
Message-ID: <CA+3bQGGSo=jxxaHFHHrnifiOWydPJiNCLavcPy8upN9PcKtTeA@mail.gmail.com>

Ah, I forgot to say that a major enhancement for the implementation is
that I wrote a lot of new unit tests for the existing Python
Initialization API. Python 3.7 has most of these tests. I wrote even
more tests for my new private initialization API ;-)

I wrote these tests to specify and validate the priority between the
different ways to configuration Python and "rules" (a parameter
setting other parameters):
https://github.com/python/peps/blob/master/pep-0587.rst#priority-and-rules

Victor

Le mer. 27 mars 2019 ? 18:48, Victor Stinner <vstinner at redhat.com> a ?crit :
>
> Hi,
>
> I would like to add a new C API to initialize Python. I would like
> your opinion on the whole API before making it public. The code is
> already implemented. Doc of the new API:
>
>    https://pythondev.readthedocs.io/init_config.html
>
>
> To make the API public, _PyWstrList, _PyInitError, _PyPreConfig,
> _PyCoreConfig and related functions should be made public.
>
> By the way, I would suggest to rename "_PyCoreConfig" to just
> "PyConfig" :-) I don't think that "core init" vs "main init" is really
> relevant: more about that below.
>
>
> Let's start with two examples using the new API.
>
> Example of simple initialization to enable isolated mode:
>
>     _PyCoreConfig config = _PyCoreConfig_INIT;
>     config.isolated = 1;
>
>     _PyInitError err = _Py_InitializeFromConfig(&config);
>     if (_Py_INIT_FAILED(err)) {
>         _Py_ExitInitError(err);
>     }
>     /* ... use Python API here ... */
>     Py_Finalize();
>
> Example using the pre-initialization to enable the UTF-8 Mode (and use the
> "legacy" Py_Initialize() function):
>
>     _PyPreConfig preconfig = _PyPreConfig_INIT;
>     preconfig.utf8_mode = 1;
>
>     _PyInitError err = _Py_PreInitialize(&preconfig);
>     if (_Py_INIT_FAILED(err)) {
>         _Py_ExitInitError(err);
>     }
>
>     /* at this point, Python will only speak UTF-8 */
>
>     Py_Initialize();
>     /* ... use Python API here ... */
>     Py_Finalize();
>
> Since November 2017, I'm refactoring the Python Initialization code to
> cleanup the code and prepare a new ("better") API to configure Python
> Initialization. I just fixed the last issues that Nick Coghlan asked
> me to fix (add a pre-initialization step: done, fix mojibake: done).
> My work is inspired by Nick Coghlan's PEP 432, but it is not
> implementing it directly. I had other motivations than Nick even if we
> are somehow going towards the same direction.
>
> Nick wants to get a half-initialized Python ("core init"), configure
> Python using the Python API and Python objects, and then finish the
> implementation ("main init").
>
> I chose a different approach: put *everything* into a single C
> structure (_PyCoreConfig) using C types. Using the structure, you
> should be able to do what Nick wanted to do, but with C rather than
> Python. Nick: please tell me if I'm wrong :-)
>
> This work is also connected to Eric Snow's work on sub-interpreters
> (PEP 554) and moving global variables into structures. For example,
> I'm using his _PyRuntime structure to store a new "preconfig" state
> (pre-initialization configuration, more about that below).
>
> In November 2017, when I started to work on the Python Initialization
> (bpo-32030), I identified the following problems:
>
> * Many parts of the code were interdependent
> * Code executed early in Py_Main() used the Python API before the Python API
>   was fully initialized. Like code parsing -W command line option which
>   used PyUnicode_FromWideChar() and PyList_Append().
> * Error handling used Py_FatalError() which didn't let the caller to decide
>   how to handle the error. Moreover, exit() was used to exit Python,
> whereas libpython shouldn't do that: a library should not exit the
> whole process! (imagine when Python is embedded inside an application)
>
> One year and a half later, I implemented the following solutions:
>
> * Py_Main() and Py_Initialize() code has been reorganized to respect
>   priorities between global configuration variables (ex:
>   Py_IgnoreEnvironmentFlag), environment variables (ex: PYTHONPATH), command
>   line arguments (ex: -X utf8), configuration files (ex: pyenv.cfg), and the
>   new _PyPreConfig and _PyCoreConfig structures which store the whole
>   configuration.
> * Python Initialization no longer uses the Python API but only C types
>   like wchar_t* strings, a new _PyWstrList structure and PyMem_RawMalloc()
>   memory allocator (PyMem_Malloc() is no longer used during init).
> * The code has been modified to use a new _PyInitError structure. The caller
>   of the top function gets control to cleanup everything before handling the
>   error (display a fatal error message or simply exit Python).
>
> The new _PyCoreConfig structure has the top-priority and provides a single
> structure for all configuration parameters.
>
> It becomes possible to override the code computing the "path configuration"
> like sys.path to fully control where Python looks to import modules. It
> becomes possible to use an empty list of paths to only allow builtin modules.
>
> A new "pre-initialization" steps is responsible to configure the bare minimum
> before the Python initialization: memory allocators and encodings
> (LC_CTYPE locale
> and the UTF-8 mode). The LC_CTYPE is no longer coerced and the UTF-8 Mode is
> no longer enabled automatically depending on the user configuration to prevent
> mojibake. Previously, calling Py_DecodeLocale() to get a Unicode wchar_t*
> string from a bytes wchar* string created mojibake when called before
> Py_Initialize() if the LC_CTYPE locale was coerced and/or if the UTF-8 Mode was
> enabled.
>
> The pre-initialization step ensures that the encodings and memory allocators
> are well defined *before* Py_Initialize() is called.
>
> Since the new API is currently private, I didn't document it in
> Python. Moreover, the code changed a lot last year :-) But it should
> now be way more stable. I started to document it in a separated
> webpage:
>
>    https://pythondev.readthedocs.io/init_config.html
>
> The plan is to put it in the Python documentation once it becomes public.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From dimaqq at gmail.com  Wed Mar 27 21:59:43 2019
From: dimaqq at gmail.com (Dima Tisnek)
Date: Thu, 28 Mar 2019 10:59:43 +0900
Subject: [Python-Dev] Mistake in 3.7.3 Changelog
Message-ID: <CAGGBzXJQSemK7n68XPkLpP3cEv+61knbPmoBGqe_Kuui0Eb2xQ@mail.gmail.com>

Hi list,

Sorry, I'm not sure where to post this.

The Changelog reads:

bpo-12477: Fix bug in parsermodule when parsing a state in a DFA that
has two or more arcs with labels of the same type. Patch by Pablo
Galindo.

The `12477` is actually a master github pull request number, not a
Python bug number.

Correct BPO is https://bugs.python.org/issue36256

Cheers,
d

From songofacandy at gmail.com  Wed Mar 27 22:27:49 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Thu, 28 Mar 2019 11:27:49 +0900
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <18ccdacf-8fc8-5130-b4ba-89df84e02987@python.org>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
 <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
 <CAEfz+Tzj5uh3K9yOF-k+1S1_s_32CSz2_Ad4kB6WH+Y7m_c1BA@mail.gmail.com>
 <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>
 <18ccdacf-8fc8-5130-b4ba-89df84e02987@python.org>
Message-ID: <CAEfz+TytJj=oMXKUE5T8dJj0Jq6ZRp6Uex-JSwmpSaOt39Mx2Q@mail.gmail.com>

On Thu, Mar 28, 2019 at 12:26 AM Steve Dower <steve.dower at python.org> wrote:
>
[snip]
> 2. Do we want everyone else to stop using it for their own purposes?
[snip]
>
> The second question is harder to answer, and in the absence of input
> from those who are already using it (or the absence of evidence that
> nobody else is using it), the best we can do is evaluate how much of a
> maintenance burden the class is.
>
> In my opinion, it is a very low maintenance burden, and a very low
> cognitive burden on users, and so the cost of deprecating it on
> third-parties who are using it vastly outweighs the cost of leaving it
> alone.

As of my personal experience,  I used PendingDeprecationWarning
because it was convention, not because it is useful.

I didn't know "right way" to chose DeprecationWarning or
PendingDeprecationWarning.
So I need to survey convention in CPython.

I think I'm not only alone.  Many developers may pay costs:

1. Confused by two warning class.
2. Survey convention
3. Follow the convention (replace PendingDeprecationWarning to
    DeprecationWarning in N-1 release)

>
> If someone can show that either no third-parties are using it, or all
> those that do will probably never explicitly support Python 3.8 or
> later, or all those that do would prefer to stop using it, then I'll
> happily change my mind here. But right now, it seems like deprecating it
> will cause an unknown amount of churn for minimal benefit.
>

Even though "document only" deprecation?
I don't propose raising DeprecationWarning for use of PendingDeprecationWarning.

If you dislike document it as ".. deprecated:: 3.8", I'm OK to use "..
note::" directive.


> At most, I'd document it as "this will probably not survive to the next
> major change release, even though we aren't planning to do one yet"
> (much like the Py_UNICODE APIs). Ironically, PendingDeprecationWarning
> seems a pretty accurate way of indicating this state.
>
> Cheers,
> Steve

I think it is still confusing.  In case of Py_UNICODE, there are 10+ years until
"next major change release."  But it's not true for everytime.  If
there are only
two years until "next major version", we should absolutely use
DeprecationWarning.

And we used "document only deprecation" instead of PendingDeprecationWarning.
For example, array('u') have not raised PendingDeprecationWarning for long time.
Only document says "Deprecated since version 3.3, will be removed in
version 4.0." [1]

[1] https://docs.python.org/3/library/array.html

I prefer document only deprecation to PendingDeprecationWarning for somehting
"It will not removed in foreseeable future.  But it will probably
removed in the future."

Note that -Wd and testing tool enable both of PendingDeprecationWarning and
DeprecationWarning.  If we use PendingDeprecationWarning for them, it
will be too noisy.


I don't think it's worth enough to try "Make PendingDeprecationWarning
useful again!".
I want to document "PendingDeprecationWarning is not useful anymore".

-- 
Inada Naoki  <songofacandy at gmail.com>

From nad at python.org  Wed Mar 27 22:37:08 2019
From: nad at python.org (Ned Deily)
Date: Wed, 27 Mar 2019 22:37:08 -0400
Subject: [Python-Dev] Mistake in 3.7.3 Changelog
In-Reply-To: <CAGGBzXJQSemK7n68XPkLpP3cEv+61knbPmoBGqe_Kuui0Eb2xQ@mail.gmail.com>
References: <CAGGBzXJQSemK7n68XPkLpP3cEv+61knbPmoBGqe_Kuui0Eb2xQ@mail.gmail.com>
Message-ID: <EE2FD92E-030E-4CF7-9E62-E1A323093081@python.org>

On Mar 27, 2019, at 21:59, Dima Tisnek <dimaqq at gmail.com> wrote:
> The Changelog reads:
> 
> bpo-12477: Fix bug in parsermodule when parsing a state in a DFA that
> has two or more arcs with labels of the same type. Patch by Pablo
> Galindo.
> 
> The `12477` is actually a master github pull request number, not a
> Python bug number.
> 
> Correct BPO is https://bugs.python.org/issue36256

Thanks for the report!  I'll see that it gets fixed.

--
  Ned Deily
  nad at python.org -- []


From nad at python.org  Thu Mar 28 00:24:45 2019
From: nad at python.org (Ned Deily)
Date: Thu, 28 Mar 2019 00:24:45 -0400
Subject: [Python-Dev] Mistake in 3.7.3 Changelog
In-Reply-To: <EE2FD92E-030E-4CF7-9E62-E1A323093081@python.org>
References: <CAGGBzXJQSemK7n68XPkLpP3cEv+61knbPmoBGqe_Kuui0Eb2xQ@mail.gmail.com>
 <EE2FD92E-030E-4CF7-9E62-E1A323093081@python.org>
Message-ID: <4A10E6CB-2136-4162-833A-0CD4262A5E3B@python.org>

On Mar 27, 2019, at 22:37, Ned Deily <nad at python.org> wrote:
> On Mar 27, 2019, at 21:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>> Correct BPO is https://bugs.python.org/issue36256
> Thanks for the report!  I'll see that it gets fixed.

    https://github.com/python/cpython/pull/12600

There were also a couple more in master/3.8:
    https://github.com/python/cpython/pull/12599

The on-line docs for 3.7 and dev should reflect the changes within a few hours.

--
  Ned Deily
  nad at python.org -- []


From turnbull.stephen.fw at u.tsukuba.ac.jp  Thu Mar 28 00:27:07 2019
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Thu, 28 Mar 2019 13:27:07 +0900
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <CA+3bQGEtmE2vB-bEbrHCDgKCJ=YRPPedwCYANgE4fshBzSQ1Dw@mail.gmail.com>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <CAP7h-xZANmsqjbdOY985YWwYZCMWgZ25qmg0GuTd-_4+WYK9Mg@mail.gmail.com>
 <CA+3bQGEtmE2vB-bEbrHCDgKCJ=YRPPedwCYANgE4fshBzSQ1Dw@mail.gmail.com>
Message-ID: <23708.19739.463707.444487@turnbull.sk.tsukuba.ac.jp>

Victor Stinner writes:

 > I just fixed the mojibake issue in Python 3.8 by disabling C locale
 > coercion and UTF-8 Mode by default. I'm not sure if nor how Python 3.7
 > should be fixed in a minor 3.7.x release.

That sounds like a potential regression.  Those two features were
added *and turned on by default* (which really means "if you detect
LC_TYPE=C, coerce") to relieve previously existing mojibake/
UnicodeError issues due to ASCII-only environments that are difficult
to configure (such as containers).  Turning them on by default was the
controversial part -- it was known that on or off, some environments
would have problems, and that's why they needed PEPs.  Do those issues
return now?  If so, where is the PEP rationale for defaulting to "on"
faulty?





From greg at krypto.org  Thu Mar 28 01:23:41 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 27 Mar 2019 22:23:41 -0700
Subject: [Python-Dev] PEP 556 threaded garbage collection & linear
 recursion in gc
In-Reply-To: <CAExdVN=JCWJQOPUxd=ApSUDzBMm58FzKBeS0ELLBuEmvwVvdFQ@mail.gmail.com>
References: <CAGE7PN+8o8AVMpXh84ZqH7cN69r9rr3yq=-enE5JaF6eXzi2aQ@mail.gmail.com>
 <CAExdVN=JCWJQOPUxd=ApSUDzBMm58FzKBeS0ELLBuEmvwVvdFQ@mail.gmail.com>
Message-ID: <CAGE7PNKWeK8AuuH3QDeUTLwLu8a00wZx3EvnBnf9S=TYsA+Kug@mail.gmail.com>

Good point, I hadn't considered that it was regular common ref count 0
dealloc chaining.  The processes unfortunately didn't have faulthandler
enabled so there wasn't much info from where in the python code it happened
(now fixed).  I'll see if anything looks particularly unusual next time I
hear of such a report.

-G

On Wed, Mar 27, 2019, 5:38 PM Tim Peters <tim.peters at gmail.com> wrote:

> [Gregory P. Smith <greg at krypto.org>]
> > ...
> > A situation came up the other day where I believe this could've helped.
> >
> > Scenario (admittedly not one most environments run into): A Python
> process
> > with a C++ extension module implementing a threaded server (threads
> > spawned by C++) that could call back into CPython within server request
> > handlers.  (ie: how all threaded servers tend to work regardless of core
> > loop implementation language)
> >
> > Python code in the application had done something (unknown to me, I
> > didn't dive into their code) that built up large enough presumably nested
> > or recursive data structures that the garbage collector, when triggered,
> > would wind up in very deep recursion.  This caused a stack overflow
> >  as the C++ spawned threads were only being given a 256k stack (to
> > conserve virtual address space - there can potentially be a _ton_ of
> > threads in this code).
> >
> > That had a C++ stack trace 1000+ levels deep repeating the pattern of
> >
> > ...
> >     @     0x564d59bd21de         32  func_dealloc
> >     @     0x564d59bce0c1         32  cell_dealloc
> >     @     0x564d5839db41         48  tupledealloc
> >     @     0x564d59bd21de         32  func_dealloc
> >     @     0x564d59bce0c1         32  cell_dealloc
> >     @     0x564d5839db41         48  tupledealloc
> > ...
> >
> > If our gc were done on a thread of its own spawned by Python, with a
> typical
> > normal larger default stack size (8mb) this would've been less likely
> > to trigger a crash (though obviously still possible if the recursion is
> linear).
>
> Just noting that gc is probably a red herring in that scenario.  gc
> (cleverly!) determines the set of trash objects without needing any
> recursive graph crawls.  And gc doesn't deallocate anything - not
> directly.  It simply applies Py_DECREF to trash objects, once for each
> PyObject* found pointing to them.  _If_ deallocation occurs as a
> result, it's the same as if the user had done `del` on the appropriate
> object manually.  The recursion then occurs in the chain of
> XXX_dealloc calls, which in turn do more Py_DECREFs of their own, and
> which have nothing in particular to do with gc.  Approximately the
> same stack depth would be hit in a Python built without gc if the user
> did the same sequence of `del`s by hand to break trash cycles.
>
> The good news is that the "trashcan" mechanism - which predates gc -
> was introduced to limit call depth in the presence of some potentially
> unbounded chains of dealloc calls.  So teach trashcan about the
> pattern here, and the stack depth problem goes away, with or without
> gc.
>
> The bad news is that the traschcan mechanism is excruciating, a
> long-time source of subtle bugs of its own :-(
>
> Note:  without actual code to run, it's possible that trashcan is
> _already_ trying to limit stack depth in this specific case, but the
> stack size is too small to accommodate the largest depth of recursive
> calls trashcan is willing to allow before interfering.  Or some bug
> snuck into trashcan, or its invoking code, that causes it to fail to
> work as intended due to some unknown detail of the code above.
> There's just no way to guess without code to provoke it.
>
>
> > Another take away from this is that it appears possible to cause our gc
> > to go into linear recursion.
>
> As above, it's probably not gc, but deallocation as a side effect of
> Py_DECREF dropping a refcount to 0.  gc is just one way Py_DECREF can
> get invoked.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190327/aa3db392/attachment.html>

From tim.peters at gmail.com  Thu Mar 28 02:33:29 2019
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 28 Mar 2019 01:33:29 -0500
Subject: [Python-Dev] PEP 556 threaded garbage collection & linear
 recursion in gc
In-Reply-To: <CAGE7PNKWeK8AuuH3QDeUTLwLu8a00wZx3EvnBnf9S=TYsA+Kug@mail.gmail.com>
References: <CAGE7PN+8o8AVMpXh84ZqH7cN69r9rr3yq=-enE5JaF6eXzi2aQ@mail.gmail.com>
 <CAExdVN=JCWJQOPUxd=ApSUDzBMm58FzKBeS0ELLBuEmvwVvdFQ@mail.gmail.com>
 <CAGE7PNKWeK8AuuH3QDeUTLwLu8a00wZx3EvnBnf9S=TYsA+Kug@mail.gmail.com>
Message-ID: <CAExdVNmD6ycgpVFBtQ_O-1=_QgTo9KutKQV7MxihnsF9Zq0Fqw@mail.gmail.com>

[Gregory P. Smith <greg at krypto.org>]
> Good point, I hadn't considered that it was regular common ref
> count 0 dealloc chaining.

It pretty much has to be whenever you see a chain of XXX_dealloc
routines in a stack trace.  gcmodule.c never even looks at a
tp_dealloc slot directly, let alone directly invoke a deallocation
method.  That all happens indirectly, as a result of what Py_DECREF
does.  Then once you're first inside one tp_dealloc method, gc is
completely irrelevant - it's that tp_dealloc for the top-level
container does its own Py_DECREF on a contained container, which in
turn can do _its_ own Py_DECREF on one of its contained containers
.... etc.  You can get an arbitrarily deep stack of XXX_dealloc calls
then, and there's really no other way to get that.

BTW,  "container" here is used in a very broad C-level sense, not a
high-level Python sense:  any PyObject that contains a pointer to a
PyObject is "a container" in the intended sense.

> The processes unfortunately didn't have faulthandler enabled so there wasn't
> much info from where in the python code it happened (now fixed).

It's quite possible that the top-level container was Py_DECREF'ed by
code in gcmodule.c.  But gc gets blamed at first for a lot of stuff
that's not actually its fault ;-)

>  I'll see if anything looks particularly unusual next time I hear of such a report.

The trashcan mechanism is the one and only hack in the code intended
to stop unbounded XXX_dealloc stacks, so that's what needs looking at.
Alas, it's hard to work with because it's so very low-level, and
there's nothing portable that can be relied on about stack sizes or
requirements across platforms or compilers.

Some possibilities:

- The trashcan code is buggy.

- The  maximum container dealloc stack depth trashcan intends to allow
(PyTrash_UNWIND_LEVEL = 50) is too large for the C stack a thread gets
under this app on this platform using this compiler.

- One or more of the specific container types involved in this app's
dealloc chain doesn't use the trashcan gimmick at all, so is invisible
to trashcan's count of how deep the call stack has gotten.

For example, cell_dealloc was in your stack trace, but I see no use of
trashcan code in that function (Py_TRASHCAN_SAFE_BEGIN /
Py_TRASHCAN_SAFE_END).  So the trashcan hack has no idea that
cell_dealloc calls are on the stack.

And likewise for func_dealloc.- looks like calls to that are also
invisible to the trashcan.

tupledealloc is cool, though.

IIRC, Christian Tismer introduced the trashcan because code only he
wrote ;-) was blowing the stack when very deeply nested lists and/or
tuples became trash.

>From a quick scan of the current code, looks like it was later added
to only a few types that aren't container types in the Python sense.

Which may or may not matter here.  Your stack trace showed a
tupledealloc in one of every three slots, so even if two of every
three slots were invisible to the traschcan, the call stack "should
have been" limited to a maximum of about PyTrash_UNWIND_LEVEL  * 3 =
150 XXX_dealloc functions.  But you saw a stack 1000+ levels deep.  So
something else that isn't immediately apparent is also going on.

From vstinner at redhat.com  Thu Mar 28 04:48:22 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 28 Mar 2019 09:48:22 +0100
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <23708.19739.463707.444487@turnbull.sk.tsukuba.ac.jp>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <CAP7h-xZANmsqjbdOY985YWwYZCMWgZ25qmg0GuTd-_4+WYK9Mg@mail.gmail.com>
 <CA+3bQGEtmE2vB-bEbrHCDgKCJ=YRPPedwCYANgE4fshBzSQ1Dw@mail.gmail.com>
 <23708.19739.463707.444487@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CA+3bQGHtnLr14hwH7urMDje0q63S4O8Ga_SBPzJ3eywGy-VwCg@mail.gmail.com>

Le jeu. 28 mars 2019 ? 05:27, Stephen J. Turnbull
<turnbull.stephen.fw at u.tsukuba.ac.jp> a ?crit :
> Victor Stinner writes:
>
>  > I just fixed the mojibake issue in Python 3.8 by disabling C locale
>  > coercion and UTF-8 Mode by default. I'm not sure if nor how Python 3.7
>  > should be fixed in a minor 3.7.x release.
>
> That sounds like a potential regression.  Those two features were
> added *and turned on by default* (which really means "if you detect
> LC_TYPE=C, coerce") to relieve previously existing mojibake/
> UnicodeError issues due to ASCII-only environments that are difficult
> to configure (such as containers).  Turning them on by default was the
> controversial part -- it was known that on or off, some environments
> would have problems, and that's why they needed PEPs.  Do those issues
> return now?  If so, where is the PEP rationale for defaulting to "on"
> faulty?

If you use "python3.8", there is no change. I'm only talking about the
specific case of Python embedded in an application: when you use the C
API.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From solipsis at pitrou.net  Thu Mar 28 05:27:23 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Mar 2019 10:27:23 +0100
Subject: [Python-Dev] PEP 556 threaded garbage collection & linear
 recursion in gc
In-Reply-To: <CAGE7PN+8o8AVMpXh84ZqH7cN69r9rr3yq=-enE5JaF6eXzi2aQ@mail.gmail.com>
References: <CAGE7PN+8o8AVMpXh84ZqH7cN69r9rr3yq=-enE5JaF6eXzi2aQ@mail.gmail.com>
Message-ID: <20190328102723.25bed461@fsol>

On Wed, 27 Mar 2019 15:59:25 -0700
"Gregory P. Smith" <greg at krypto.org> wrote:
> 
> That had a C++ stack trace 1000+ levels deep repeating the pattern of
> 
> ...
>     @     0x564d59bd21de         32  func_dealloc
>     @     0x564d59bce0c1         32  cell_dealloc
>     @     0x564d5839db41         48  tupledealloc
>     @     0x564d59bd21de         32  func_dealloc
>     @     0x564d59bce0c1         32  cell_dealloc
>     @     0x564d5839db41         48  tupledealloc
> ...

As Tim said, if you still have a core dump somewhere (or can reproduce
the issue) it would be nice to know why the "trashcan" mechanism didn't
trigger.

Regards

Antoine.

From J.Demeyer at UGent.be  Thu Mar 28 06:06:00 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Thu, 28 Mar 2019 11:06:00 +0100
Subject: [Python-Dev] PEP 556 threaded garbage collection & linear
 recursion in gc
In-Reply-To: <CAExdVN=JCWJQOPUxd=ApSUDzBMm58FzKBeS0ELLBuEmvwVvdFQ@mail.gmail.com>
References: <CAGE7PN+8o8AVMpXh84ZqH7cN69r9rr3yq=-enE5JaF6eXzi2aQ@mail.gmail.com>
 <CAExdVN=JCWJQOPUxd=ApSUDzBMm58FzKBeS0ELLBuEmvwVvdFQ@mail.gmail.com>
Message-ID: <5C9C9C88.7020904@UGent.be>

On 2019-03-28 01:38, Tim Peters wrote:
> The bad news is that the traschcan mechanism is excruciating, a
> long-time source of subtle bugs of its own :-(

It just happens that I created a PR to fix some of the trashcan 
problems: see https://bugs.python.org/issue35983 and 
https://github.com/python/cpython/pull/11841

From vano at mail.mipt.ru  Thu Mar 28 06:24:10 2019
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Thu, 28 Mar 2019 13:24:10 +0300
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
Message-ID: <f8b4ebc4-ee9b-2557-cf74-76babd717da3@mail.mipt.ru>


On 27.03.2019 20:48, Victor Stinner wrote:
> Hi,
>
> I would like to add a new C API to initialize Python. I would like
> your opinion on the whole API before making it public. The code is
> already implemented. Doc of the new API:
>
>     https://pythondev.readthedocs.io/init_config.html
>
>
> To make the API public, _PyWstrList, _PyInitError, _PyPreConfig,
> _PyCoreConfig and related functions should be made public.
>
> By the way, I would suggest to rename "_PyCoreConfig" to just
> "PyConfig" :-) I don't think that "core init" vs "main init" is really
> relevant: more about that below.
>
>
> Let's start with two examples using the new API.
>
> Example of simple initialization to enable isolated mode:
>
>      _PyCoreConfig config = _PyCoreConfig_INIT;
>      config.isolated = 1;
>
>      _PyInitError err = _Py_InitializeFromConfig(&config);
By my outsider observation, the `config' argument and return code are asking to be added to Py_Initialize instead, 
`_Py_InitializeFromConfig` and `_Py_PreInitialize` look redundant.
>      if (_Py_INIT_FAILED(err)) {
>          _Py_ExitInitError(err);
>      }
>      /* ... use Python API here ... */
>      Py_Finalize();
>
> Example using the pre-initialization to enable the UTF-8 Mode (and use the
> "legacy" Py_Initialize() function):
>
>      _PyPreConfig preconfig = _PyPreConfig_INIT;
>      preconfig.utf8_mode = 1;
>
>      _PyInitError err = _Py_PreInitialize(&preconfig);
>      if (_Py_INIT_FAILED(err)) {
>          _Py_ExitInitError(err);
>      }
>
>      /* at this point, Python will only speak UTF-8 */
>
>      Py_Initialize();
>      /* ... use Python API here ... */
>      Py_Finalize();
>
> Since November 2017, I'm refactoring the Python Initialization code to
> cleanup the code and prepare a new ("better") API to configure Python
> Initialization. I just fixed the last issues that Nick Coghlan asked
> me to fix (add a pre-initialization step: done, fix mojibake: done).
> My work is inspired by Nick Coghlan's PEP 432, but it is not
> implementing it directly. I had other motivations than Nick even if we
> are somehow going towards the same direction.
>
> Nick wants to get a half-initialized Python ("core init"), configure
> Python using the Python API and Python objects, and then finish the
> implementation ("main init").
>
> I chose a different approach: put *everything* into a single C
> structure (_PyCoreConfig) using C types. Using the structure, you
> should be able to do what Nick wanted to do, but with C rather than
> Python. Nick: please tell me if I'm wrong :-)
>
> This work is also connected to Eric Snow's work on sub-interpreters
> (PEP 554) and moving global variables into structures. For example,
> I'm using his _PyRuntime structure to store a new "preconfig" state
> (pre-initialization configuration, more about that below).
>
> In November 2017, when I started to work on the Python Initialization
> (bpo-32030), I identified the following problems:
>
> * Many parts of the code were interdependent
> * Code executed early in Py_Main() used the Python API before the Python API
>    was fully initialized. Like code parsing -W command line option which
>    used PyUnicode_FromWideChar() and PyList_Append().
> * Error handling used Py_FatalError() which didn't let the caller to decide
>    how to handle the error. Moreover, exit() was used to exit Python,
> whereas libpython shouldn't do that: a library should not exit the
> whole process! (imagine when Python is embedded inside an application)
>
> One year and a half later, I implemented the following solutions:
>
> * Py_Main() and Py_Initialize() code has been reorganized to respect
>    priorities between global configuration variables (ex:
>    Py_IgnoreEnvironmentFlag), environment variables (ex: PYTHONPATH), command
>    line arguments (ex: -X utf8), configuration files (ex: pyenv.cfg), and the
>    new _PyPreConfig and _PyCoreConfig structures which store the whole
>    configuration.
> * Python Initialization no longer uses the Python API but only C types
>    like wchar_t* strings, a new _PyWstrList structure and PyMem_RawMalloc()
>    memory allocator (PyMem_Malloc() is no longer used during init).
> * The code has been modified to use a new _PyInitError structure. The caller
>    of the top function gets control to cleanup everything before handling the
>    error (display a fatal error message or simply exit Python).
>
> The new _PyCoreConfig structure has the top-priority and provides a single
> structure for all configuration parameters.
>
> It becomes possible to override the code computing the "path configuration"
> like sys.path to fully control where Python looks to import modules. It
> becomes possible to use an empty list of paths to only allow builtin modules.
>
> A new "pre-initialization" steps is responsible to configure the bare minimum
> before the Python initialization: memory allocators and encodings
> (LC_CTYPE locale
> and the UTF-8 mode). The LC_CTYPE is no longer coerced and the UTF-8 Mode is
> no longer enabled automatically depending on the user configuration to prevent
> mojibake. Previously, calling Py_DecodeLocale() to get a Unicode wchar_t*
> string from a bytes wchar* string created mojibake when called before
> Py_Initialize() if the LC_CTYPE locale was coerced and/or if the UTF-8 Mode was
> enabled.
>
> The pre-initialization step ensures that the encodings and memory allocators
> are well defined *before* Py_Initialize() is called.
>
> Since the new API is currently private, I didn't document it in
> Python. Moreover, the code changed a lot last year :-) But it should
> now be way more stable. I started to document it in a separated
> webpage:
>
>     https://pythondev.readthedocs.io/init_config.html
>
> The plan is to put it in the Python documentation once it becomes public.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- 
Regards,
Ivan


From vstinner at redhat.com  Thu Mar 28 10:03:26 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 28 Mar 2019 15:03:26 +0100
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <CAP1=2W6+udqgYbXCvGQAg4tbuKdU8LfrDeCrrvwMoxtZZ7UN1g@mail.gmail.com>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <70c245c7-8f40-a51b-934b-af958f7cd849@python.org>
 <CAP1=2W6+udqgYbXCvGQAg4tbuKdU8LfrDeCrrvwMoxtZZ7UN1g@mail.gmail.com>
Message-ID: <CA+3bQGG3BAp_4pEqkY9JiJOEStkX4KA=cdaYbTv30FospKC-7w@mail.gmail.com>

Le mer. 27 mars 2019 ? 21:26, Brett Cannon <brett at python.org> a ?crit :
> On Wed, Mar 27, 2019 at 12:39 PM Steve Dower <steve.dower at python.org> wrote:
>> I this this should be its own PEP, since as you say it is not
>> implementing the only PEP we have (or alternatively, maybe you should
>> collaborate to update PEP 432 so that it reflects what you think we
>> ought to be implementing).
>
> I agree that if this isn't doing what PEP 432 set out but going its own way we should probably discuss in regards to 432.

I'm sorry, I was in a hurry when I wrote the new PEP 587 and it seems
like I created some confusion. My PEP 587 is very similar to the PEP
432, because it is basically an implementation of the PEP 432 design.
I am collaborating closely with Nick Coghlan and Eric Snow on the
Python Initialization for 1 year and a half, and I just continued the
work they started.

The PEP 432 has been written in 2012 and has a "broader scope". Since
the PEP has been written, the code has been modified slowly towards
PEP 432 design, but not "exactly" the expected design, because of
concrete practical issues of the implementation.

The PEP 587 is the updated *implementation* of the PEP 432 which can
be seen as the overall *design*.

The PEP 587 is only a subset of the PEP 432: C API to initialize
Python, whereas PEP 432 goes further by introducing the concepts of
"core" and "main" initialization. The "core initialization" is a bare
minimum working Python only with builtin types, partial sys module and
no importlib. "Main initialization" is a fully working Python. This
part is out of the scope of the PEP 587, but the PEP 587 should be
flexible enough to allow to implement it later.

In fact, there is already a PyConfig._init_config flag (currently
named _PyCoreConfig._init_main) which only initializes Python up to
the "core initialization" if set to 0. This parameter is private since
it's unclear to me what should be the exact scope of "core" vs "main"
init.

I wrote a PR to clarify the relationship between the PEP 587 and the PEP 432:
https://github.com/python/peps/pull/955/files

Victor

From steve.dower at python.org  Thu Mar 28 11:32:57 2019
From: steve.dower at python.org (Steve Dower)
Date: Thu, 28 Mar 2019 08:32:57 -0700
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <CA+3bQGG3BAp_4pEqkY9JiJOEStkX4KA=cdaYbTv30FospKC-7w@mail.gmail.com>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <70c245c7-8f40-a51b-934b-af958f7cd849@python.org>
 <CAP1=2W6+udqgYbXCvGQAg4tbuKdU8LfrDeCrrvwMoxtZZ7UN1g@mail.gmail.com>
 <CA+3bQGG3BAp_4pEqkY9JiJOEStkX4KA=cdaYbTv30FospKC-7w@mail.gmail.com>
Message-ID: <119e26f0-d826-7e13-aa4a-e0a67bea3265@python.org>

On 28Mar2019 0703, Victor Stinner wrote:
> In fact, there is already a PyConfig._init_config flag (currently
> named _PyCoreConfig._init_main) which only initializes Python up to
> the "core initialization" if set to 0. This parameter is private since
> it's unclear to me what should be the exact scope of "core" vs "main"
> init.

We tried to set up a video call between the interested people (Eric, 
Nick, myself, yourself, couple of others) to clarify this point, and you 
refused to join ;)

That said, the call never happened (honestly, there's not a lot of point 
in doing it without you being part of it), so we still don't have a 
clear idea of where the line should be drawn. But there are enough of us 
with fuzzy but valid ideas in our heads that we really need that 
brainstorming session to mix them together and find something feasible. 
Maybe we're best to put it off until PyCon at this point.

Cheers,
Steve

From pablogsal at gmail.com  Thu Mar 28 12:37:08 2019
From: pablogsal at gmail.com (Pablo Galindo Salgado)
Date: Thu, 28 Mar 2019 16:37:08 +0000
Subject: [Python-Dev] PEP 570
Message-ID: <CAFjbc8FG2ihd96pQXF_i9HXb2VO=0cMFZjG4_5AJReoJS4uc+w@mail.gmail.com>

Hi everyone,

We have submitted PEP 570 for consideration to the steering council:

https://github.com/python/steering-council/issues/4

The discussion is happening on discourse:

https://discuss.python.org/t/pep-570-python-positional-only-parameters/1078

To eliminate splitting the discussion into two forums (discourse and
python-dev),
we kindly ask you to go to discourse if you want to participate in the
debate :)

Here is the full document of the PEP:

https://www.python.org/dev/peps/pep-0570/

Pablo Galindo Salgado
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190328/fbf4aafa/attachment.html>

From brian.kuhl at windriver.com  Thu Mar 28 13:15:46 2019
From: brian.kuhl at windriver.com (Kuhl, Brian)
Date: Thu, 28 Mar 2019 17:15:46 +0000
Subject: [Python-Dev] VxWorks and Python
Message-ID: <6C486AB80682E24095E3E77A408D68F20149A5DF1E@ALA-MBD.corp.ad.wrs.com>

Hi Python developers,
Victor suggested I post a little background to help the maintainers understand VxWorks a little better.
It can be found here
https://github.com/Wind-River/cpython/wiki/Python-on-VxWorks
If you'd like more detail in a particular area, please feel free to reply on or off list.

I'd also like to mention again, that Wind River is interested in setting up a short term consulting arrangement, with any core dev that's available.
To help accelerate Peixing's PRs acceptance
https://github.com/python/cpython/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+bpo-31904
and generally improve Cpython.

Wind River operates globally so don't let you location dissuade you.

Many thanks for all your help,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190328/a748a233/attachment.html>

From nas-python at arctrix.com  Thu Mar 28 15:39:28 2019
From: nas-python at arctrix.com (Neil Schemenauer)
Date: Thu, 28 Mar 2019 13:39:28 -0600
Subject: [Python-Dev] PEP 556 threaded garbage collection & linear
 recursion in gc
In-Reply-To: <20190328102723.25bed461@fsol>
References: <CAGE7PN+8o8AVMpXh84ZqH7cN69r9rr3yq=-enE5JaF6eXzi2aQ@mail.gmail.com>
 <20190328102723.25bed461@fsol>
Message-ID: <20190328193928.pg2g3skia5ggpkjx@python.ca>

On 2019-03-28, Antoine Pitrou wrote:
> On Wed, 27 Mar 2019 15:59:25 -0700
> "Gregory P. Smith" <greg at krypto.org> wrote:
> > 
> > That had a C++ stack trace 1000+ levels deep repeating the pattern of
> > 
> > ...
> >     @     0x564d59bd21de         32  func_dealloc
> >     @     0x564d59bce0c1         32  cell_dealloc
> >     @     0x564d5839db41         48  tupledealloc
> >     @     0x564d59bd21de         32  func_dealloc
> >     @     0x564d59bce0c1         32  cell_dealloc
> >     @     0x564d5839db41         48  tupledealloc
> > ...
> 
> As Tim said, if you still have a core dump somewhere (or can reproduce
> the issue) it would be nice to know why the "trashcan" mechanism didn't
> trigger.

To expand on this, every time tupledealloc gets called,
Py_TRASHCAN_SAFE_BEGIN also gets invoked.  It increments
tstate->trash_delete_nesting.  As Tim suggests, maybe
PyTrash_UNWIND_LEVEL is too large given the size of the C stack
frames from func_dealloc + cell_dealloc + tupledealloc.

That theory seems hard to believe though, unless the C stack is
quite small.  I see PyTrash_UNWIND_LEVEL = 50.  Perhaps the stack
could have been mostly used up before the dealloc sequence started.

The other option is that there is some bug in the trashcan
mechanism.  It certainly is some very tricky code.

Regards,

  Neil

From steve.dower at python.org  Thu Mar 28 18:35:46 2019
From: steve.dower at python.org (Steve Dower)
Date: Thu, 28 Mar 2019 15:35:46 -0700
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
Message-ID: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>

Hi all

Time is short, but I'm hoping to get PEP 578 (formerly PEP 551) into 
Python 3.8. Here's the current text for review and comment before I 
submit to the Steering Council.

The formatted text is at https://www.python.org/dev/peps/pep-0578/ 
(update just pushed, so give it an hour or so, but it's fundamentally 
the same as what's there)

No Discourse post, because we don't have a python-dev equivalent there 
yet, so please reply here for this one.

Implementation is at https://github.com/zooba/cpython/tree/pep-578/ and 
my backport to 3.7 (https://github.com/zooba/cpython/tree/pep-578-3.7/) 
is already getting some real use (though this will not be added to 3.7, 
unless people *really* want it, so the backport is just for reference).

Cheers,
Steve

=====

PEP: 578
Title: Python Runtime Audit Hooks
Version: $Revision$
Last-Modified: $Date$
Author: Steve Dower <steve.dower at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 16-Jun-2018
Python-Version: 3.8
Post-History:

Abstract
========

This PEP describes additions to the Python API and specific behaviors
for the CPython implementation that make actions taken by the Python
runtime visible to auditing tools. Visibility into these actions
provides opportunities for test frameworks, logging frameworks, and
security tools to monitor and optionally limit actions taken by the
runtime.

This PEP proposes adding two APIs to provide insights into a running
Python application: one for arbitrary events, and another specific to
the module import system. The APIs are intended to be available in all
Python implementations, though the specific messages and values used
are unspecified here to allow implementations the freedom to determine
how best to provide information to their users. Some examples likely
to be used in CPython are provided for explanatory purposes.

See PEP 551 for discussion and recommendations on enhancing the
security of a Python runtime making use of these auditing APIs.

Background
==========

Python provides access to a wide range of low-level functionality on
many common operating systems. While this is incredibly useful for
"write-once, run-anywhere" scripting, it also makes monitoring of
software written in Python difficult. Because Python uses native system
APIs directly, existing monitoring tools either suffer from limited
context or auditing bypass.

Limited context occurs when system monitoring can report that an
action occurred, but cannot explain the sequence of events leading to
it. For example, network monitoring at the OS level may be able to
report "listening started on port 5678", but may not be able to
provide the process ID, command line, parent process, or the local
state in the program at the point that triggered the action. Firewall
controls to prevent such an action are similarly limited, typically
to process names or some global state such as the current user, and
in any case rarely provide a useful log file correlated with other
application messages.

Auditing bypass can occur when the typical system tool used for an
action would ordinarily report its use, but accessing the APIs via
Python do not trigger this. For example, invoking "curl" to make HTTP
requests may be specifically monitored in an audited system, but
Python's "urlretrieve" function is not.

Within a long-running Python application, particularly one that
processes user-provided information such as a web app, there is a risk
of unexpected behavior. This may be due to bugs in the code, or
deliberately induced by a malicious user. In both cases, normal
application logging may be bypassed resulting in no indication that
anything out of the ordinary has occurred.

Additionally, and somewhat unique to Python, it is very easy to affect
the code that is run in an application by manipulating either the
import system's search path or placing files earlier on the path than
intended. This is often seen when developers create a script with the
same name as the module they intend to use - for example, a
``random.py`` file that attempts to import the standard library
``random`` module.

This is not sandboxing, as this proposal does not attempt to prevent
malicious behavior (though it enables some new options to do so).
See the `Why Not A Sandbox`_ section below for further discussion.

Overview of Changes
===================

The aim of these changes is to enable both application developers and
system administrators to integrate Python into their existing
monitoring systems without dictating how those systems look or behave.

We propose two API changes to enable this: an Audit Hook?and Verified
Open Hook. Both are available from Python and native code, allowing
applications and frameworks written in pure Python code to take
advantage of the extra messages, while also allowing embedders or
system administrators to deploy builds of Python where auditing is
always enabled.

Only CPython is bound to provide the native APIs as described here.
Other implementations should provide the pure Python APIs, and
may provide native versions as appropriate for their underlying
runtimes. Auditing events are likewise considered implementation
specific, but are bound by normal feature compatibility guarantees.

Audit Hook
----------

In order to observe actions taken by the runtime (on behalf of the
caller), an API is required to raise messages from within certain
operations. These operations are typically deep within the Python
runtime or standard library, such as dynamic code compilation, module
imports, DNS resolution, or use of certain modules such as ``ctypes``.

The following new C APIs allow embedders and CPython implementors to
send and receive audit hook messages::

    # Add an auditing hook
    typedef int (*hook_func)(const char *event, PyObject *args,
                             void *userData);
    int PySys_AddAuditHook(hook_func hook, void *userData);

    # Raise an event with all auditing hooks
    int PySys_Audit(const char *event, PyObject *args);

    # Internal API used during Py_Finalize() - not publicly accessible
    void _Py_ClearAuditHooks(void);

The new Python APIs for receiving and raising audit hooks are::

    # Add an auditing hook
    sys.addaudithook(hook: Callable[[str, tuple]])

    # Raise an event with all auditing hooks
    sys.audit(str, *args)


Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time,
including before ``Py_Initialize()``, or by calling
``sys.addaudithook()`` from Python code. Hooks cannot be removed or
replaced.

When events of interest are occurring, code can either call
``PySys_Audit()`` from C (while the GIL is held) or ``sys.audit()``. The
string argument is the name of the event, and the tuple contains
arguments. A given event name should have a fixed schema for arguments,
which should be considered a public API (for each x.y version release),
and thus should only change between feature releases with updated
documentation.

For maximum compatibility, events using the same name as an event in
the reference interpreter CPython should make every attempt to use
compatible arguments. Including the name or an abbreviation of the
implementation in implementation-specific event names will also help
prevent collisions. For example, a ``pypy.jit_invoked`` event is clearly
distinguised from an ``ipy.jit_invoked`` event.

When an event is audited, each hook is called in the order it was added
with the event name and tuple. If any hook returns with an exception
set, later hooks are ignored and *in general* the Python runtime should
terminate. This is intentional to allow hook implementations to decide
how to respond to any particular event. The typical responses will be to
log the event, abort the operation with an exception, or to immediately
terminate the process with an operating system exit call.

When an event is audited but no hooks have been set, the ``audit()``
function should impose minimal overhead. Ideally, each argument is a
reference to existing data?rather than a value calculated just for the
auditing call.

As hooks may be Python objects, they need to be freed during
``Py_Finalize()``. To do this, we add an internal API
``_Py_ClearAuditHooks()`` that releases any Python hooks and any
memory held. This is an internal function with no public export, and
we recommend it raise its own audit event for all current hooks to
ensure that unexpected calls are observed.

Below in `Suggested Audit Hook Locations`_, we recommend some important
operations that should raise audit events.

Python implementations should document which operations will raise
audit events, along with the event schema. It is intentional that
``sys.addaudithook(print)`` be a trivial way to display all messages.

Verified Open Hook
------------------

Most operating systems have a mechanism to distinguish between files
that can be executed and those that can not. For example, this may be an
execute bit in the permissions field, a verified hash of the file
contents to detect potential code tampering, or file system path
restrictions. These are an important security mechanism for preventing
execution of data or code that is not approved for a given environment.
Currently, Python has no way to integrate with these when launching
scripts or importing modules.

The new public C API for the verified open hook is::

    # Set the handler
    typedef PyObject *(*hook_func)(PyObject *path, void *userData)
    int PyImport_SetOpenForImportHook(hook_func handler, void *userData)

    # Open a file using the handler
    PyObject *PyImport_OpenForImport(const char *path)

The new public Python API for the verified open hook is::

    # Open a file using the handler
    importlib.util.open_for_import(path : str) -> io.IOBase


The ``importlib.util.open_for_import()`` function is a drop-in
replacement?for ``open(str(pathlike), 'rb')``. Its default behaviour is
to open a file for raw, binary access. To change the behaviour a new
handler should be set. Handler functions only accept ``str`` arguments.
The C API ``PyImport_OpenForImport`` function assumes UTF-8 encoding.

A custom handler may be set by calling ``PyImport_SetOpenForImportHook()``
from C at any time, including before ``Py_Initialize()``. However, if a
hook has already been set then the call will fail. When
``open_for_import()`` is called with a hook set, the hook will be passed
the path and its return value will be returned directly. The returned
object should be an open file-like object that supports reading raw
bytes. This is explicitly intended to allow a ``BytesIO`` instance if
the open handler has already read the entire file into memory.

Note that these hooks can import and call the ``_io.open()`` function on
CPython without triggering themselves. They can also use ``_io.BytesIO``
to return a compatible result using an in-memory buffer.

If the hook determines that the file should not be loaded, it should
raise an exception of its choice, as well as performing any other
logging.

All import and execution functionality involving code from a file will
be changed to use ``open_for_import()`` unconditionally. It is important
to note that calls to ``compile()``, ``exec()`` and ``eval()`` do not go
through this function - an audit hook that includes the code from these
calls is the best opportunity to validate code that is read from the
file. Given the current decoupling between import and execution in
Python, most imported code will go through both ``open_for_import()``
and the log hook for ``compile``, and so care should be taken to avoid
repeating verification steps.

There is no Python API provided for changing the open hook. To modify
import behavior from Python code, use the existing functionality
provided by ``importlib``.

API Availability
----------------

While all the functions added here are considered public and stable API,
the behavior of the functions is implementation specific. Most
descriptions here refer?to the CPython?implementation, and while other
implementations should provide the functions, there is no requirement
that they behave the same.

For example, ``sys.addaudithook()`` and ``sys.audit()`` should exist but
may do nothing. This allows code to make calls to ``sys.audit()``
without having to test for existence, but it should not assume that its
call will have any effect. (Including existence tests in
security-critical code allows another vector to bypass auditing, so it
is preferable that the function always exist.)

``importlib.util.open_for_import(path)`` should at a minimum always
return ``_io.open(path, 'rb')``. Code using the function should make no
further assumptions about what may occur, and implementations other than
CPython are not required to let developers override the behavior of this
function with a hook.

Suggested Audit Hook Locations
==============================

The locations and parameters in calls to ``sys.audit()`` or
``PySys_Audit()`` are to be determined by individual Python
implementations. This is to allow maximum freedom for implementations
to expose the operations that are most relevant to their platform,
and to avoid or ignore potentially expensive or noisy events.

Table 1 acts as both suggestions of operations that should trigger
audit events on all implementations, and examples of event schemas.

Table 2 provides further examples that are not required, but are
likely to be available in CPython.

Refer to the documentation associated with your version of Python to
see which operations provide audit events.

.. csv-table:: Table 1: Suggested Audit Hooks
    :header: "API Function", "Event Name", "Arguments", "Rationale"
    :widths: 2, 2, 3, 6

    ``PySys_AddAuditHook``, ``sys.addaudithook``, "", "Detect when new
    audit hooks are being added.
    "
    ``PyImport_SetOpenForImportHook``, ``setopenforimporthook``, "", "
    Detects any attempt to set the ``open_for_import`` hook.
    "
    "``compile``, ``exec``, ``eval``, ``PyAst_CompileString``,
    ``PyAST_obj2mod``", ``compile``, "``(code, filename_or_none)``", "
    Detect dynamic code compilation, where ``code`` could be a string or
    AST. Note that this will be called for regular imports of source
    code, including those that were opened with ``open_for_import``.
    "
    "``exec``, ``eval``, ``run_mod``", ``exec``, "``(code_object,)``", "
    Detect dynamic execution of code objects. This only occurs for
    explicit calls, and is not raised for normal function invocation.
    "
    ``import``, ``import``, "``(module, filename, sys.path,
    sys.meta_path, sys.path_hooks)``", "Detect when modules are
    imported. This is raised before the module name is resolved to a
    file. All arguments other than the module name may be ``None`` if
    they are not used or available.
    "
    "``open``", ``open``, "``(path, mode, flags)``", "Detect when a file
    is about to be opened. *path* and *mode* are the usual parameters to
    ``open`` if available, while *flags* is provided instead of *mode*
    in some cases.
    "
    ``PyEval_SetProfile``, ``sys.setprofile``, "", "Detect when code is
    injecting trace functions. Because of the implementation, exceptions
    raised from the hook will abort the operation, but will not be
    raised in Python code. Note that ``threading.setprofile`` eventually
    calls this function, so the event will be audited for each thread.
    "
    ``PyEval_SetTrace``, ``sys.settrace``, "", "Detect when code is
    injecting trace functions. Because of the implementation, exceptions
    raised from the hook will abort the operation, but will not be
    raised in Python code. Note that ``threading.settrace`` eventually
    calls this function, so the event will be audited for each thread.
    "
    "``_PyObject_GenericSetAttr``, ``check_set_special_type_attr``,
    ``object_set_class``, ``func_set_code``, ``func_set_[kw]defaults``","
    ``object.__setattr__``","``(object, attr, value)``","Detect monkey
    patching of types and objects. This event
    is raised for the ``__class__`` attribute and any attribute on
    ``type`` objects.
    "
    "``_PyObject_GenericSetAttr``",``object.__delattr__``,"``(object,
    attr)``","Detect deletion of object attributes. This event is raised
    for any attribute on ``type`` objects.
    "
    "``Unpickler.find_class``",``pickle.find_class``,"``(module_name,
    global_name)``","Detect imports and global name lookup when
    unpickling.
    "


.. csv-table:: Table 2: Potential CPython Audit Hooks
    :header: "API Function", "Event Name", "Arguments", "Rationale"
    :widths: 2, 2, 3, 6

    ``_PySys_ClearAuditHooks``, ``sys._clearaudithooks``, "", "Notifies
    hooks they are being cleaned up, mainly in case the event is
    triggered unexpectedly. This event cannot be aborted.
    "
    ``code_new``, ``code.__new__``, "``(bytecode, filename, name)``", "
    Detect dynamic creation of code objects. This only occurs for
    direct instantiation, and is not raised for normal compilation.
    "
    ``func_new_impl``, ``function.__new__``, "``(code,)``", "Detect
    dynamic creation of function objects. This only occurs for direct
    instantiation, and is not raised for normal compilation.
    "
    "``_ctypes.dlopen``, ``_ctypes.LoadLibrary``", ``ctypes.dlopen``, "
    ``(module_or_path,)``", "Detect when native modules are used.
    "
    ``_ctypes._FuncPtr``, ``ctypes.dlsym``, "``(lib_object, name)``", "
    Collect information about specific symbols retrieved from native
    modules.
    "
    ``_ctypes._CData``, ``ctypes.cdata``, "``(ptr_as_int,)``", "Detect
    when code is accessing arbitrary memory using ``ctypes``.
    "
    "``new_mmap_object``",``mmap.__new__``,"``(fileno, map_size, access,
    offset)``", "Detects creation of mmap objects. On POSIX, access may
    have been calculated from the ``prot`` and ``flags`` arguments.
    "
    ``sys._getframe``, ``sys._getframe``, "``(frame_object,)``", "Detect
    when code is accessing frames directly.
    "
    ``sys._current_frames``, ``sys._current_frames``, "", "Detect when
    code is accessing frames directly.
    "
    "``socket.bind``, ``socket.connect``, ``socket.connect_ex``,
    ``socket.getaddrinfo``, ``socket.getnameinfo``, ``socket.sendmsg``,
    ``socket.sendto``", ``socket.address``, "``(address,)``", "Detect
    access to network resources. The address is unmodified from the
    original call.
    "
    "``member_get``, ``func_get_code``, ``func_get_[kw]defaults``
    ",``object.__getattr__``,"``(object, attr)``","Detect access to
    restricted attributes. This event is raised for any built-in
    members that are marked as restricted, and members that may allow
    bypassing imports.
    "
    "``urllib.urlopen``",``urllib.Request``,"``(url, data, headers,
    method)``", "Detects URL requests.
    "

Performance Impact
==================

The important performance impact is the case where events are being
raised but there are no hooks attached. This is the unavoidable case -
once a developer has added audit hooks they have explicitly chosen to
trade performance for functionality. Performance impact with hooks added
are not of interest here, since this is opt-in functionality.

Analysis using the Python Performance Benchmark Suite [1]_ shows no
significant impact, with the vast majority of benchmarks showing
between 1.05x faster to 1.05x slower.

In our opinion, the performance impact of the set of auditing points
described in this PEP is negligible.

Rejected Ideas
==============

Separate module for audit hooks
-------------------------------

The proposal is to add a new module for audit hooks, hypothetically
``audit``. This would separate the API and implementation from the
``sys`` module, and allow naming the C functions ``PyAudit_AddHook`` and
``PyAudit_Audit`` rather than the current variations.

Any such module would need to be a built-in module that is guaranteed to
always be present. The nature of these hooks is that they must be
callable without condition, as any conditional imports or calls provide
opportunities to intercept and suppress or modify events.

Given it is one of the most core modules, the ``sys`` module is somewhat
protected against module shadowing attacks. Replacing ``sys`` with a
sufficiently functional module that the application can still run is a
much more complicated task than replacing a module with only one
function of interest. An attacker that has the ability to shadow the
``sys`` module is already capable of running arbitrary code from files,
whereas an ``audit`` module could be replaced with a single line in a
``.pth`` file anywhere on the search path::

     import sys; sys.modules['audit'] = type('audit', (object,),
         {'audit': lambda *a: None, 'addhook': lambda *a: None})

Multiple layers of protection already exist for monkey patching attacks
against either ``sys`` or ``audit``, but assignments or insertions to
``sys.modules`` are not audited.

This idea is rejected because it makes it trivial to suppress all calls
to ``audit``.

Flag in sys.flags to indicate "audited" mode
--------------------------------------------

The proposal is to add a value in ``sys.flags`` to indicate when Python
is running in a "secure" or "audited" mode. This would allow
applications to detect when some features are enabled or when hooks
have been added and modify their behaviour appropriately.

Currently, we are not aware of any legitimate reasons for a program to
behave differently in the presence of audit hooks.

Both application-level APIs ``sys.audit`` and
``importlib.util.open_for_import`` are always present and functional,
regardless of whether the regular ``python`` entry point or some
alternative entry point is used. Callers cannot determine whether any
hooks have been added (except by performing side-channel analysis), nor
do they need to. The calls should be fast enough that callers do not
need to avoid them, and the program is responsible for ensuring that
any added hooks are fast enough to not affect application performance.

The argument that this is "security by obscurity" is valid, but
irrelevant. Security by obscurity is only an issue when there are no
other protective mechanisms; obscurity as the first step in avoiding
attack is strongly recommended (see `this article
<https://danielmiessler.com/study/security-by-obscurity/>`_ for
discussion).

This idea is rejected because there are no appropriate reasons for an
application to change its behaviour based on whether these APIs are in
use.

Why Not A Sandbox
=================

Sandboxing CPython has been attempted many times in the past, and each
past attempt has failed. Fundamentally, the problem is that certain
functionality has to be restricted when executing the sandboxed code,
but otherwise needs to be available for normal operation of Python. For
example, completely removing the ability to compile strings into
bytecode also breaks the ability to import modules from source code, and
if it is not completely removed then there are too many ways to get
access to that functionality indirectly. There is not yet any feasible
way to generically determine whether a given operation is "safe" or not.
Further information and references available at [2]_.

This proposal does not attempt to restrict functionality, but simply
exposes the fact that the functionality is being used. Particularly for
intrusion scenarios, detection is significantly more important than
early prevention (as early prevention will generally drive attackers to
use an alternate, less-detectable, approach). The availability of audit
hooks alone does not change the attack surface of Python in any way, but
they enable defenders to integrate Python into their environment in ways
that are currently not possible.

Since audit hooks have the ability to safely prevent an operation
occuring, this feature does enable the ability to provide some level of
sandboxing. In most cases, however, the intention is to enable logging
rather than creating a sandbox.

Relationship to PEP 551
=======================

This API was originally presented as part of
`PEP 551 <https://www.python.org/dev/peps/pep-0551/>`_ Security
Transparency in the Python Runtime.

For simpler review purposes, and due to the broader applicability of
these APIs beyond security, the API design is now presented separately.

PEP 551 is an informational PEP discussing how to integrate Python into
a secure or audited environment.

References
==========

.. [1] Python Performance Benchmark Suite 
`<https://github.com/python/performance>`_

.. [2] Python Security model - Sandbox 
`<https://python-security.readthedocs.io/security.html#sandbox>`_

Copyright
=========

Copyright (c) 2019 by Microsoft Corporation. This material may be
distributed only subject to the terms and conditions set forth in the
Open Publication License, v1.0 or later (the latest version is presently
available at http://www.opencontent.org/openpub/).

From steve.dower at python.org  Thu Mar 28 18:55:21 2019
From: steve.dower at python.org (Steve Dower)
Date: Thu, 28 Mar 2019 15:55:21 -0700
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
Message-ID: <c71c319a-32aa-58d7-a34c-53856d90cf9d@python.org>

The implementation can be viewed as a PR at 
https://github.com/python/cpython/pull/12613

On 28Mar2019 1535, Steve Dower wrote:
> Hi all
> 
> Time is short, but I'm hoping to get PEP 578 (formerly PEP 551) into 
> Python 3.8. Here's the current text for review and comment before I 
> submit to the Steering Council.
> 
> The formatted text is at https://www.python.org/dev/peps/pep-0578/ 
> (update just pushed, so give it an hour or so, but it's fundamentally 
> the same as what's there)
> 
> No Discourse post, because we don't have a python-dev equivalent there 
> yet, so please reply here for this one.
> 
> Implementation is at https://github.com/zooba/cpython/tree/pep-578/ and 
> my backport to 3.7 (https://github.com/zooba/cpython/tree/pep-578-3.7/) 
> is already getting some real use (though this will not be added to 3.7, 
> unless people *really* want it, so the backport is just for reference).
> 
> Cheers,
> Steve
> 
> =====
> 
> PEP: 578
> Title: Python Runtime Audit Hooks
> Version: $Revision$
> Last-Modified: $Date$
> Author: Steve Dower <steve.dower at python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 16-Jun-2018
> Python-Version: 3.8
> Post-History:
> 
> Abstract
> ========
> 
> This PEP describes additions to the Python API and specific behaviors
> for the CPython implementation that make actions taken by the Python
> runtime visible to auditing tools. Visibility into these actions
> provides opportunities for test frameworks, logging frameworks, and
> security tools to monitor and optionally limit actions taken by the
> runtime.
> 
> This PEP proposes adding two APIs to provide insights into a running
> Python application: one for arbitrary events, and another specific to
> the module import system. The APIs are intended to be available in all
> Python implementations, though the specific messages and values used
> are unspecified here to allow implementations the freedom to determine
> how best to provide information to their users. Some examples likely
> to be used in CPython are provided for explanatory purposes.
> 
> See PEP 551 for discussion and recommendations on enhancing the
> security of a Python runtime making use of these auditing APIs.
> 
> Background
> ==========
> 
> Python provides access to a wide range of low-level functionality on
> many common operating systems. While this is incredibly useful for
> "write-once, run-anywhere" scripting, it also makes monitoring of
> software written in Python difficult. Because Python uses native system
> APIs directly, existing monitoring tools either suffer from limited
> context or auditing bypass.
> 
> Limited context occurs when system monitoring can report that an
> action occurred, but cannot explain the sequence of events leading to
> it. For example, network monitoring at the OS level may be able to
> report "listening started on port 5678", but may not be able to
> provide the process ID, command line, parent process, or the local
> state in the program at the point that triggered the action. Firewall
> controls to prevent such an action are similarly limited, typically
> to process names or some global state such as the current user, and
> in any case rarely provide a useful log file correlated with other
> application messages.
> 
> Auditing bypass can occur when the typical system tool used for an
> action would ordinarily report its use, but accessing the APIs via
> Python do not trigger this. For example, invoking "curl" to make HTTP
> requests may be specifically monitored in an audited system, but
> Python's "urlretrieve" function is not.
> 
> Within a long-running Python application, particularly one that
> processes user-provided information such as a web app, there is a risk
> of unexpected behavior. This may be due to bugs in the code, or
> deliberately induced by a malicious user. In both cases, normal
> application logging may be bypassed resulting in no indication that
> anything out of the ordinary has occurred.
> 
> Additionally, and somewhat unique to Python, it is very easy to affect
> the code that is run in an application by manipulating either the
> import system's search path or placing files earlier on the path than
> intended. This is often seen when developers create a script with the
> same name as the module they intend to use - for example, a
> ``random.py`` file that attempts to import the standard library
> ``random`` module.
> 
> This is not sandboxing, as this proposal does not attempt to prevent
> malicious behavior (though it enables some new options to do so).
> See the `Why Not A Sandbox`_ section below for further discussion.
> 
> Overview of Changes
> ===================
> 
> The aim of these changes is to enable both application developers and
> system administrators to integrate Python into their existing
> monitoring systems without dictating how those systems look or behave.
> 
> We propose two API changes to enable this: an Audit Hook?and Verified
> Open Hook. Both are available from Python and native code, allowing
> applications and frameworks written in pure Python code to take
> advantage of the extra messages, while also allowing embedders or
> system administrators to deploy builds of Python where auditing is
> always enabled.
> 
> Only CPython is bound to provide the native APIs as described here.
> Other implementations should provide the pure Python APIs, and
> may provide native versions as appropriate for their underlying
> runtimes. Auditing events are likewise considered implementation
> specific, but are bound by normal feature compatibility guarantees.
> 
> Audit Hook
> ----------
> 
> In order to observe actions taken by the runtime (on behalf of the
> caller), an API is required to raise messages from within certain
> operations. These operations are typically deep within the Python
> runtime or standard library, such as dynamic code compilation, module
> imports, DNS resolution, or use of certain modules such as ``ctypes``.
> 
> The following new C APIs allow embedders and CPython implementors to
> send and receive audit hook messages::
> 
>  ?? # Add an auditing hook
>  ?? typedef int (*hook_func)(const char *event, PyObject *args,
>  ??????????????????????????? void *userData);
>  ?? int PySys_AddAuditHook(hook_func hook, void *userData);
> 
>  ?? # Raise an event with all auditing hooks
>  ?? int PySys_Audit(const char *event, PyObject *args);
> 
>  ?? # Internal API used during Py_Finalize() - not publicly accessible
>  ?? void _Py_ClearAuditHooks(void);
> 
> The new Python APIs for receiving and raising audit hooks are::
> 
>  ?? # Add an auditing hook
>  ?? sys.addaudithook(hook: Callable[[str, tuple]])
> 
>  ?? # Raise an event with all auditing hooks
>  ?? sys.audit(str, *args)
> 
> 
> Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time,
> including before ``Py_Initialize()``, or by calling
> ``sys.addaudithook()`` from Python code. Hooks cannot be removed or
> replaced.
> 
> When events of interest are occurring, code can either call
> ``PySys_Audit()`` from C (while the GIL is held) or ``sys.audit()``. The
> string argument is the name of the event, and the tuple contains
> arguments. A given event name should have a fixed schema for arguments,
> which should be considered a public API (for each x.y version release),
> and thus should only change between feature releases with updated
> documentation.
> 
> For maximum compatibility, events using the same name as an event in
> the reference interpreter CPython should make every attempt to use
> compatible arguments. Including the name or an abbreviation of the
> implementation in implementation-specific event names will also help
> prevent collisions. For example, a ``pypy.jit_invoked`` event is clearly
> distinguised from an ``ipy.jit_invoked`` event.
> 
> When an event is audited, each hook is called in the order it was added
> with the event name and tuple. If any hook returns with an exception
> set, later hooks are ignored and *in general* the Python runtime should
> terminate. This is intentional to allow hook implementations to decide
> how to respond to any particular event. The typical responses will be to
> log the event, abort the operation with an exception, or to immediately
> terminate the process with an operating system exit call.
> 
> When an event is audited but no hooks have been set, the ``audit()``
> function should impose minimal overhead. Ideally, each argument is a
> reference to existing data?rather than a value calculated just for the
> auditing call.
> 
> As hooks may be Python objects, they need to be freed during
> ``Py_Finalize()``. To do this, we add an internal API
> ``_Py_ClearAuditHooks()`` that releases any Python hooks and any
> memory held. This is an internal function with no public export, and
> we recommend it raise its own audit event for all current hooks to
> ensure that unexpected calls are observed.
> 
> Below in `Suggested Audit Hook Locations`_, we recommend some important
> operations that should raise audit events.
> 
> Python implementations should document which operations will raise
> audit events, along with the event schema. It is intentional that
> ``sys.addaudithook(print)`` be a trivial way to display all messages.
> 
> Verified Open Hook
> ------------------
> 
> Most operating systems have a mechanism to distinguish between files
> that can be executed and those that can not. For example, this may be an
> execute bit in the permissions field, a verified hash of the file
> contents to detect potential code tampering, or file system path
> restrictions. These are an important security mechanism for preventing
> execution of data or code that is not approved for a given environment.
> Currently, Python has no way to integrate with these when launching
> scripts or importing modules.
> 
> The new public C API for the verified open hook is::
> 
>  ?? # Set the handler
>  ?? typedef PyObject *(*hook_func)(PyObject *path, void *userData)
>  ?? int PyImport_SetOpenForImportHook(hook_func handler, void *userData)
> 
>  ?? # Open a file using the handler
>  ?? PyObject *PyImport_OpenForImport(const char *path)
> 
> The new public Python API for the verified open hook is::
> 
>  ?? # Open a file using the handler
>  ?? importlib.util.open_for_import(path : str) -> io.IOBase
> 
> 
> The ``importlib.util.open_for_import()`` function is a drop-in
> replacement?for ``open(str(pathlike), 'rb')``. Its default behaviour is
> to open a file for raw, binary access. To change the behaviour a new
> handler should be set. Handler functions only accept ``str`` arguments.
> The C API ``PyImport_OpenForImport`` function assumes UTF-8 encoding.
> 
> A custom handler may be set by calling ``PyImport_SetOpenForImportHook()``
> from C at any time, including before ``Py_Initialize()``. However, if a
> hook has already been set then the call will fail. When
> ``open_for_import()`` is called with a hook set, the hook will be passed
> the path and its return value will be returned directly. The returned
> object should be an open file-like object that supports reading raw
> bytes. This is explicitly intended to allow a ``BytesIO`` instance if
> the open handler has already read the entire file into memory.
> 
> Note that these hooks can import and call the ``_io.open()`` function on
> CPython without triggering themselves. They can also use ``_io.BytesIO``
> to return a compatible result using an in-memory buffer.
> 
> If the hook determines that the file should not be loaded, it should
> raise an exception of its choice, as well as performing any other
> logging.
> 
> All import and execution functionality involving code from a file will
> be changed to use ``open_for_import()`` unconditionally. It is important
> to note that calls to ``compile()``, ``exec()`` and ``eval()`` do not go
> through this function - an audit hook that includes the code from these
> calls is the best opportunity to validate code that is read from the
> file. Given the current decoupling between import and execution in
> Python, most imported code will go through both ``open_for_import()``
> and the log hook for ``compile``, and so care should be taken to avoid
> repeating verification steps.
> 
> There is no Python API provided for changing the open hook. To modify
> import behavior from Python code, use the existing functionality
> provided by ``importlib``.
> 
> API Availability
> ----------------
> 
> While all the functions added here are considered public and stable API,
> the behavior of the functions is implementation specific. Most
> descriptions here refer?to the CPython?implementation, and while other
> implementations should provide the functions, there is no requirement
> that they behave the same.
> 
> For example, ``sys.addaudithook()`` and ``sys.audit()`` should exist but
> may do nothing. This allows code to make calls to ``sys.audit()``
> without having to test for existence, but it should not assume that its
> call will have any effect. (Including existence tests in
> security-critical code allows another vector to bypass auditing, so it
> is preferable that the function always exist.)
> 
> ``importlib.util.open_for_import(path)`` should at a minimum always
> return ``_io.open(path, 'rb')``. Code using the function should make no
> further assumptions about what may occur, and implementations other than
> CPython are not required to let developers override the behavior of this
> function with a hook.
> 
> Suggested Audit Hook Locations
> ==============================
> 
> The locations and parameters in calls to ``sys.audit()`` or
> ``PySys_Audit()`` are to be determined by individual Python
> implementations. This is to allow maximum freedom for implementations
> to expose the operations that are most relevant to their platform,
> and to avoid or ignore potentially expensive or noisy events.
> 
> Table 1 acts as both suggestions of operations that should trigger
> audit events on all implementations, and examples of event schemas.
> 
> Table 2 provides further examples that are not required, but are
> likely to be available in CPython.
> 
> Refer to the documentation associated with your version of Python to
> see which operations provide audit events.
> 
> .. csv-table:: Table 1: Suggested Audit Hooks
>  ?? :header: "API Function", "Event Name", "Arguments", "Rationale"
>  ?? :widths: 2, 2, 3, 6
> 
>  ?? ``PySys_AddAuditHook``, ``sys.addaudithook``, "", "Detect when new
>  ?? audit hooks are being added.
>  ?? "
>  ?? ``PyImport_SetOpenForImportHook``, ``setopenforimporthook``, "", "
>  ?? Detects any attempt to set the ``open_for_import`` hook.
>  ?? "
>  ?? "``compile``, ``exec``, ``eval``, ``PyAst_CompileString``,
>  ?? ``PyAST_obj2mod``", ``compile``, "``(code, filename_or_none)``", "
>  ?? Detect dynamic code compilation, where ``code`` could be a string or
>  ?? AST. Note that this will be called for regular imports of source
>  ?? code, including those that were opened with ``open_for_import``.
>  ?? "
>  ?? "``exec``, ``eval``, ``run_mod``", ``exec``, "``(code_object,)``", "
>  ?? Detect dynamic execution of code objects. This only occurs for
>  ?? explicit calls, and is not raised for normal function invocation.
>  ?? "
>  ?? ``import``, ``import``, "``(module, filename, sys.path,
>  ?? sys.meta_path, sys.path_hooks)``", "Detect when modules are
>  ?? imported. This is raised before the module name is resolved to a
>  ?? file. All arguments other than the module name may be ``None`` if
>  ?? they are not used or available.
>  ?? "
>  ?? "``open``", ``open``, "``(path, mode, flags)``", "Detect when a file
>  ?? is about to be opened. *path* and *mode* are the usual parameters to
>  ?? ``open`` if available, while *flags* is provided instead of *mode*
>  ?? in some cases.
>  ?? "
>  ?? ``PyEval_SetProfile``, ``sys.setprofile``, "", "Detect when code is
>  ?? injecting trace functions. Because of the implementation, exceptions
>  ?? raised from the hook will abort the operation, but will not be
>  ?? raised in Python code. Note that ``threading.setprofile`` eventually
>  ?? calls this function, so the event will be audited for each thread.
>  ?? "
>  ?? ``PyEval_SetTrace``, ``sys.settrace``, "", "Detect when code is
>  ?? injecting trace functions. Because of the implementation, exceptions
>  ?? raised from the hook will abort the operation, but will not be
>  ?? raised in Python code. Note that ``threading.settrace`` eventually
>  ?? calls this function, so the event will be audited for each thread.
>  ?? "
>  ?? "``_PyObject_GenericSetAttr``, ``check_set_special_type_attr``,
>  ?? ``object_set_class``, ``func_set_code``, ``func_set_[kw]defaults``","
>  ?? ``object.__setattr__``","``(object, attr, value)``","Detect monkey
>  ?? patching of types and objects. This event
>  ?? is raised for the ``__class__`` attribute and any attribute on
>  ?? ``type`` objects.
>  ?? "
>  ?? "``_PyObject_GenericSetAttr``",``object.__delattr__``,"``(object,
>  ?? attr)``","Detect deletion of object attributes. This event is raised
>  ?? for any attribute on ``type`` objects.
>  ?? "
>  ?? "``Unpickler.find_class``",``pickle.find_class``,"``(module_name,
>  ?? global_name)``","Detect imports and global name lookup when
>  ?? unpickling.
>  ?? "
> 
> 
> .. csv-table:: Table 2: Potential CPython Audit Hooks
>  ?? :header: "API Function", "Event Name", "Arguments", "Rationale"
>  ?? :widths: 2, 2, 3, 6
> 
>  ?? ``_PySys_ClearAuditHooks``, ``sys._clearaudithooks``, "", "Notifies
>  ?? hooks they are being cleaned up, mainly in case the event is
>  ?? triggered unexpectedly. This event cannot be aborted.
>  ?? "
>  ?? ``code_new``, ``code.__new__``, "``(bytecode, filename, name)``", "
>  ?? Detect dynamic creation of code objects. This only occurs for
>  ?? direct instantiation, and is not raised for normal compilation.
>  ?? "
>  ?? ``func_new_impl``, ``function.__new__``, "``(code,)``", "Detect
>  ?? dynamic creation of function objects. This only occurs for direct
>  ?? instantiation, and is not raised for normal compilation.
>  ?? "
>  ?? "``_ctypes.dlopen``, ``_ctypes.LoadLibrary``", ``ctypes.dlopen``, "
>  ?? ``(module_or_path,)``", "Detect when native modules are used.
>  ?? "
>  ?? ``_ctypes._FuncPtr``, ``ctypes.dlsym``, "``(lib_object, name)``", "
>  ?? Collect information about specific symbols retrieved from native
>  ?? modules.
>  ?? "
>  ?? ``_ctypes._CData``, ``ctypes.cdata``, "``(ptr_as_int,)``", "Detect
>  ?? when code is accessing arbitrary memory using ``ctypes``.
>  ?? "
>  ?? "``new_mmap_object``",``mmap.__new__``,"``(fileno, map_size, access,
>  ?? offset)``", "Detects creation of mmap objects. On POSIX, access may
>  ?? have been calculated from the ``prot`` and ``flags`` arguments.
>  ?? "
>  ?? ``sys._getframe``, ``sys._getframe``, "``(frame_object,)``", "Detect
>  ?? when code is accessing frames directly.
>  ?? "
>  ?? ``sys._current_frames``, ``sys._current_frames``, "", "Detect when
>  ?? code is accessing frames directly.
>  ?? "
>  ?? "``socket.bind``, ``socket.connect``, ``socket.connect_ex``,
>  ?? ``socket.getaddrinfo``, ``socket.getnameinfo``, ``socket.sendmsg``,
>  ?? ``socket.sendto``", ``socket.address``, "``(address,)``", "Detect
>  ?? access to network resources. The address is unmodified from the
>  ?? original call.
>  ?? "
>  ?? "``member_get``, ``func_get_code``, ``func_get_[kw]defaults``
>  ?? ",``object.__getattr__``,"``(object, attr)``","Detect access to
>  ?? restricted attributes. This event is raised for any built-in
>  ?? members that are marked as restricted, and members that may allow
>  ?? bypassing imports.
>  ?? "
>  ?? "``urllib.urlopen``",``urllib.Request``,"``(url, data, headers,
>  ?? method)``", "Detects URL requests.
>  ?? "
> 
> Performance Impact
> ==================
> 
> The important performance impact is the case where events are being
> raised but there are no hooks attached. This is the unavoidable case -
> once a developer has added audit hooks they have explicitly chosen to
> trade performance for functionality. Performance impact with hooks added
> are not of interest here, since this is opt-in functionality.
> 
> Analysis using the Python Performance Benchmark Suite [1]_ shows no
> significant impact, with the vast majority of benchmarks showing
> between 1.05x faster to 1.05x slower.
> 
> In our opinion, the performance impact of the set of auditing points
> described in this PEP is negligible.
> 
> Rejected Ideas
> ==============
> 
> Separate module for audit hooks
> -------------------------------
> 
> The proposal is to add a new module for audit hooks, hypothetically
> ``audit``. This would separate the API and implementation from the
> ``sys`` module, and allow naming the C functions ``PyAudit_AddHook`` and
> ``PyAudit_Audit`` rather than the current variations.
> 
> Any such module would need to be a built-in module that is guaranteed to
> always be present. The nature of these hooks is that they must be
> callable without condition, as any conditional imports or calls provide
> opportunities to intercept and suppress or modify events.
> 
> Given it is one of the most core modules, the ``sys`` module is somewhat
> protected against module shadowing attacks. Replacing ``sys`` with a
> sufficiently functional module that the application can still run is a
> much more complicated task than replacing a module with only one
> function of interest. An attacker that has the ability to shadow the
> ``sys`` module is already capable of running arbitrary code from files,
> whereas an ``audit`` module could be replaced with a single line in a
> ``.pth`` file anywhere on the search path::
> 
>  ??? import sys; sys.modules['audit'] = type('audit', (object,),
>  ??????? {'audit': lambda *a: None, 'addhook': lambda *a: None})
> 
> Multiple layers of protection already exist for monkey patching attacks
> against either ``sys`` or ``audit``, but assignments or insertions to
> ``sys.modules`` are not audited.
> 
> This idea is rejected because it makes it trivial to suppress all calls
> to ``audit``.
> 
> Flag in sys.flags to indicate "audited" mode
> --------------------------------------------
> 
> The proposal is to add a value in ``sys.flags`` to indicate when Python
> is running in a "secure" or "audited" mode. This would allow
> applications to detect when some features are enabled or when hooks
> have been added and modify their behaviour appropriately.
> 
> Currently, we are not aware of any legitimate reasons for a program to
> behave differently in the presence of audit hooks.
> 
> Both application-level APIs ``sys.audit`` and
> ``importlib.util.open_for_import`` are always present and functional,
> regardless of whether the regular ``python`` entry point or some
> alternative entry point is used. Callers cannot determine whether any
> hooks have been added (except by performing side-channel analysis), nor
> do they need to. The calls should be fast enough that callers do not
> need to avoid them, and the program is responsible for ensuring that
> any added hooks are fast enough to not affect application performance.
> 
> The argument that this is "security by obscurity" is valid, but
> irrelevant. Security by obscurity is only an issue when there are no
> other protective mechanisms; obscurity as the first step in avoiding
> attack is strongly recommended (see `this article
> <https://danielmiessler.com/study/security-by-obscurity/>`_ for
> discussion).
> 
> This idea is rejected because there are no appropriate reasons for an
> application to change its behaviour based on whether these APIs are in
> use.
> 
> Why Not A Sandbox
> =================
> 
> Sandboxing CPython has been attempted many times in the past, and each
> past attempt has failed. Fundamentally, the problem is that certain
> functionality has to be restricted when executing the sandboxed code,
> but otherwise needs to be available for normal operation of Python. For
> example, completely removing the ability to compile strings into
> bytecode also breaks the ability to import modules from source code, and
> if it is not completely removed then there are too many ways to get
> access to that functionality indirectly. There is not yet any feasible
> way to generically determine whether a given operation is "safe" or not.
> Further information and references available at [2]_.
> 
> This proposal does not attempt to restrict functionality, but simply
> exposes the fact that the functionality is being used. Particularly for
> intrusion scenarios, detection is significantly more important than
> early prevention (as early prevention will generally drive attackers to
> use an alternate, less-detectable, approach). The availability of audit
> hooks alone does not change the attack surface of Python in any way, but
> they enable defenders to integrate Python into their environment in ways
> that are currently not possible.
> 
> Since audit hooks have the ability to safely prevent an operation
> occuring, this feature does enable the ability to provide some level of
> sandboxing. In most cases, however, the intention is to enable logging
> rather than creating a sandbox.
> 
> Relationship to PEP 551
> =======================
> 
> This API was originally presented as part of
> `PEP 551 <https://www.python.org/dev/peps/pep-0551/>`_ Security
> Transparency in the Python Runtime.
> 
> For simpler review purposes, and due to the broader applicability of
> these APIs beyond security, the API design is now presented separately.
> 
> PEP 551 is an informational PEP discussing how to integrate Python into
> a secure or audited environment.
> 
> References
> ==========
> 
> .. [1] Python Performance Benchmark Suite 
> `<https://github.com/python/performance>`_
> 
> .. [2] Python Security model - Sandbox 
> `<https://python-security.readthedocs.io/security.html#sandbox>`_
> 
> Copyright
> =========
> 
> Copyright (c) 2019 by Microsoft Corporation. This material may be
> distributed only subject to the terms and conditions set forth in the
> Open Publication License, v1.0 or later (the latest version is presently
> available at http://www.opencontent.org/openpub/).


From vstinner at redhat.com  Thu Mar 28 20:02:16 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Fri, 29 Mar 2019 01:02:16 +0100
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
Message-ID: <CA+3bQGEodqbHk4A4fPAzRYeDmX0OG51DWN_Rq28aJfHt15UWdw@mail.gmail.com>

Hi,

I read quickly the PEP, I'm not sure that I understood it correctly,
so here are some early questions more about the usage of the PEP, than
its implementation.

> This is not sandboxing, as this proposal does not attempt to prevent
> malicious behavior (though it enables some new options to do so).
> See the `Why Not A Sandbox`_ section below for further discussion.

I don't understand well the overall security model. If malicious
behaviors can still occur, what is the the purpose of auditing? For
example, if an audit hook writes events into a local log file, the
attacker can easily remove this log file, no?

While you say that it's not a sandbox, you designed multiple
protections to protect auditing, and the design is very close to a
sandbox.

Example:

    "``_PyObject_GenericSetAttr``, ``check_set_special_type_attr``,
    ``object_set_class``, ``func_set_code``, ``func_set_[kw]defaults``","
    ``object.__setattr__``","``(object, attr, value)``","Detect monkey
    patching of types and objects. This event
    is raised for the ``__class__`` attribute and any attribute on
    ``type`` objects.

It reminds me Python 2 Bastion module which has simply been removed
because it was unsafe:
https://docs.python.org/2/library/bastion.html

For example, using ctypes, you can access directly the underlying
dictionary of a type and modify "private" attributes. It's just an
example.


I wrote pysandbox in the past, and it was broken by default, as you
wrote in the PEP :-)

   The failure of pysandbox (2013)
   https://lwn.net/Articles/574215/

If you want to secure Python, run Python in a sandbox, don't try to
"add" a sandbox "on top" of Python (I know that it's more complicated
in practice).

Or use PyPy sandbox which has a very different design.


Le jeu. 28 mars 2019 ? 23:39, Steve Dower <steve.dower at python.org> a ?crit :
> This PEP describes additions to the Python API and specific behaviors
> for the CPython implementation that make actions taken by the Python
> runtime visible to auditing tools. Visibility into these actions
> provides opportunities for test frameworks, logging frameworks, and
> security tools to monitor and optionally limit actions taken by the
> runtime.

Is it possible to implement these features without adding a new API or
modifying Python?

Short example adding logs to open():
---
import builtins
import functools

def add_log(name, func):
    @functools.wraps(func)
    def wrapper(*args):
        print(name, args)
        return func(*args)
    return wrapper

builtins.open = add_log("open", builtins.open)

open(__file__).close()
---


> Verified Open Hook
> ------------------
>
> Most operating systems have a mechanism to distinguish between files
> that can be executed and those that can not. For example, this may be an
> execute bit in the permissions field, a verified hash of the file
> contents to detect potential code tampering, or file system path
> restrictions. These are an important security mechanism for preventing
> execution of data or code that is not approved for a given environment.
> Currently, Python has no way to integrate with these when launching
> scripts or importing modules.

In my experience, it doesn't work just because Python has too many
functions opening files indirectly or call external C libraries which
open files.

I vaguely recall an exploit in my pysandbox project which uses the
internal code of Python which displays a traceback... to read the
content of an arbitrary file on the disk :-( Game over. I would never
expect that there are so many ways to read a file in Python...

Even when I restricted pysandbox to the bare minimum of the Python
language (with no import), multiple exploits have been found.
Moreover, at the end, Python just became useful.

More generally, there are a lot of codes in Python which allow
arbitrary code injection :-( (Most of them are now fixed, hopefully!)

I did my best to modify as much functions as possible to implement the
PEP 446 "Make newly created file descriptors non-inheritable", but I
know that *many* functions call directly open() or fopen() and so
create inheritable file descriptors. For example, the Python ssl
module takes directly filenames and OpenSSL open directly files. It's
just one example.

You will never be able to cover all cases.

Having a single function which allows to open an arbitrary file
without triggering an audit event would defeat the whole purpose of
auditing, no? Again, maybe I didn't understand well the overall
purpose of the PEP, sorry.


> Table 2 provides further examples that are not required, but are
> likely to be available in CPython.
> (...)
> Performance Impact
> ==================
>
> The important performance impact is the case where events are being
> raised but there are no hooks attached. This is the unavoidable case -
> once a developer has added audit hooks they have explicitly chosen to
> trade performance for functionality.

(The Linux kernel uses advance tooling to inject hooks: it has no
impact on performances when no hook is used. Machine code of functions
is patched to inject a hook. Impressive stuff :-))

Here I expect a small overhead. But the global overhead will be
proportional to the number of hooks, no? Maybe it's not significant
with the proposed list of events, but it will be more significant with
100 or 1000 events?

I'm not saying that it's a blocker issue, I'm just thinking aloud to
make sure that I understood correctly :-)

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.

From turnbull.stephen.fw at u.tsukuba.ac.jp  Thu Mar 28 23:08:42 2019
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Fri, 29 Mar 2019 12:08:42 +0900
Subject: [Python-Dev] A request for PEP announcement format [was: PEP 570]
In-Reply-To: <CAFjbc8FG2ihd96pQXF_i9HXb2VO=0cMFZjG4_5AJReoJS4uc+w@mail.gmail.com>
References: <CAFjbc8FG2ihd96pQXF_i9HXb2VO=0cMFZjG4_5AJReoJS4uc+w@mail.gmail.com>
Message-ID: <23709.35898.906806.70203@turnbull.sk.tsukuba.ac.jp>

A proponent writes:

 > We have submitted PEP 570 for consideration to the steering
 > council:

I would like to ask *future* proponents to please include the title
explicitly in the first mention.  (This is not a criticism of the
previous post.  How could he know I would write this?)

In this case, it's here:

 > https://discuss.python.org/t/pep-570-Python-Positional-Only-Parameters/1078

(sorry, case changed for emphasis so clicking won't work) so no great
harm done!

Thanks in advance!

Steve

From jcgoble3 at gmail.com  Thu Mar 28 23:24:57 2019
From: jcgoble3 at gmail.com (Jonathan Goble)
Date: Thu, 28 Mar 2019 23:24:57 -0400
Subject: [Python-Dev] A request for PEP announcement format [was: PEP
 570]
In-Reply-To: <23709.35898.906806.70203@turnbull.sk.tsukuba.ac.jp>
References: <CAFjbc8FG2ihd96pQXF_i9HXb2VO=0cMFZjG4_5AJReoJS4uc+w@mail.gmail.com>
 <23709.35898.906806.70203@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CAK256p24z86FvHyHbywnT1sH9ZYCrPvgjuGh1xrJ=CVUsre6Qg@mail.gmail.com>

On Thu, Mar 28, 2019 at 11:11 PM Stephen J. Turnbull <
turnbull.stephen.fw at u.tsukuba.ac.jp> wrote:

>  >
> https://discuss.python.org/t/pep-570-Python-Positional-Only-Parameters/1078
>
> (sorry, case changed for emphasis so clicking won't work)
>

Clicking actually did work for me. Not only is that portion of the link not
case-sensitive, but it doesn't even matter what it is or if it's present at
all. https://discuss.python.org/t/pep-570-fgfdgfdgdfgd-parameters/1078 and
https://discuss.python.org/t/1078 both lead to the same thread, but if I
change the number at the end to 1077, I get a different thread.

So the system only uses the number at the end to identify the thread, and
the text is strictly for the benefit of human readability and not parsed by
the server. File that in the "probably useless information" category of
your brain. ;-)

*goes back to lurking*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190328/8d87ef0c/attachment.html>

From J.Demeyer at UGent.be  Fri Mar 29 02:25:26 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Fri, 29 Mar 2019 07:25:26 +0100
Subject: [Python-Dev] A request for PEP announcement format [was: PEP
 570]
In-Reply-To: <23709.35898.906806.70203@turnbull.sk.tsukuba.ac.jp>
References: <CAFjbc8FG2ihd96pQXF_i9HXb2VO=0cMFZjG4_5AJReoJS4uc+w@mail.gmail.com>
 <23709.35898.906806.70203@turnbull.sk.tsukuba.ac.jp>
Message-ID: <5C9DBA56.2020006@UGent.be>

On 2019-03-29 04:08, Stephen J. Turnbull wrote:
> In this case, it's here:
>
>   > https://discuss.python.org/t/pep-570-Python-Positional-Only-Parameters/1078

So, are we supposed to discuss PEPs on discuss.python.org now? That's 
fine for me, should I create a thread like that for PEP 580 too?

From christian at python.org  Fri Mar 29 06:24:41 2019
From: christian at python.org (Christian Heimes)
Date: Fri, 29 Mar 2019 11:24:41 +0100
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <CA+3bQGEodqbHk4A4fPAzRYeDmX0OG51DWN_Rq28aJfHt15UWdw@mail.gmail.com>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <CA+3bQGEodqbHk4A4fPAzRYeDmX0OG51DWN_Rq28aJfHt15UWdw@mail.gmail.com>
Message-ID: <4d1ef7fe-c235-f98e-4887-acc77606f485@python.org>

On 29/03/2019 01.02, Victor Stinner wrote:
> Hi,
> 
> I read quickly the PEP, I'm not sure that I understood it correctly,
> so here are some early questions more about the usage of the PEP, than
> its implementation.
> 
>> This is not sandboxing, as this proposal does not attempt to prevent
>> malicious behavior (though it enables some new options to do so).
>> See the `Why Not A Sandbox`_ section below for further discussion.
> 
> I don't understand well the overall security model. If malicious
> behaviors can still occur, what is the the purpose of auditing? For
> example, if an audit hook writes events into a local log file, the
> attacker can easily remove this log file, no?

An attacker may not have permission to mess with the auditing subsystem.
For example an attacker may be able to modify an application like a web
server or web application. Audit loggers typically run in a different,
more protected context. On modern, hardened operation systems root /
Adminstrator aren't all powerful, too. They are typically restricted by
additional policies like SELinux.

Further more, servers also send auditing data to remote nodes for
analysis. Keep in mind that auditing is not primarily about preventing
compromises. It's about detecting what, when, who, and how a system was
compromised.



>> Verified Open Hook
>> ------------------
>>
>> Most operating systems have a mechanism to distinguish between files
>> that can be executed and those that can not. For example, this may be an
>> execute bit in the permissions field, a verified hash of the file
>> contents to detect potential code tampering, or file system path
>> restrictions. These are an important security mechanism for preventing
>> execution of data or code that is not approved for a given environment.
>> Currently, Python has no way to integrate with these when launching
>> scripts or importing modules.
> 
> In my experience, it doesn't work just because Python has too many
> functions opening files indirectly or call external C libraries which
> open files.
> 
> I vaguely recall an exploit in my pysandbox project which uses the
> internal code of Python which displays a traceback... to read the
> content of an arbitrary file on the disk :-( Game over. I would never
> expect that there are so many ways to read a file in Python...

The verified open hook is not about sandboxing. It's a mechanism to
prevent a class of attacks like directory traversal attacks. On Linux,
the open-for-import hook could refuse access to .py and .pyc files that
do not have the user.python_code or root.python_code extended file
attribute. This verified open hook could have prevent the compromise of
wiki.python.org many years ago.


> Even when I restricted pysandbox to the bare minimum of the Python
> language (with no import), multiple exploits have been found.
> Moreover, at the end, Python just became useful.
> 
> More generally, there are a lot of codes in Python which allow
> arbitrary code injection :-( (Most of them are now fixed, hopefully!)
> 
> I did my best to modify as much functions as possible to implement the
> PEP 446 "Make newly created file descriptors non-inheritable", but I
> know that *many* functions call directly open() or fopen() and so
> create inheritable file descriptors. For example, the Python ssl
> module takes directly filenames and OpenSSL open directly files. It's
> just one example.
> 
> You will never be able to cover all cases.

I agree. Don't draw the wrong conclusion from your statement. PEP 578
adds hooks for auditing, which in return can be used to harden and log
an application. Unlike secure sandboxing, it doesn't have to be perfect.
Alex Gaynor summed this up in his blog post
https://alexgaynor.net/2018/jul/20/worst-truism-in-infosec/

> Having a single function which allows to open an arbitrary file
> without triggering an audit event would defeat the whole purpose of
> auditing, no? Again, maybe I didn't understand well the overall
> purpose of the PEP, sorry.

This case can be detected during development and QE phase. You simply
have to count the amount of open syscalls and compare it to the amount
of open auditing events.


>> The important performance impact is the case where events are being
>> raised but there are no hooks attached. This is the unavoidable case -
>> once a developer has added audit hooks they have explicitly chosen to
>> trade performance for functionality.
> 
> (The Linux kernel uses advance tooling to inject hooks: it has no
> impact on performances when no hook is used. Machine code of functions
> is patched to inject a hook. Impressive stuff :-))
> 
> Here I expect a small overhead. But the global overhead will be
> proportional to the number of hooks, no? Maybe it's not significant
> with the proposed list of events, but it will be more significant with
> 100 or 1000 events?
> 
> I'm not saying that it's a blocker issue, I'm just thinking aloud to
> make sure that I understood correctly :-)

The performance impact can be remedied and reduced with a simple check.
If there is no audit hook installed, it's just a matter of a pointer
deref + JNZ.

Christian

From christian at python.org  Fri Mar 29 06:24:41 2019
From: christian at python.org (Christian Heimes)
Date: Fri, 29 Mar 2019 11:24:41 +0100
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <CA+3bQGEodqbHk4A4fPAzRYeDmX0OG51DWN_Rq28aJfHt15UWdw@mail.gmail.com>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <CA+3bQGEodqbHk4A4fPAzRYeDmX0OG51DWN_Rq28aJfHt15UWdw@mail.gmail.com>
Message-ID: <4d1ef7fe-c235-f98e-4887-acc77606f485@python.org>

On 29/03/2019 01.02, Victor Stinner wrote:
> Hi,
> 
> I read quickly the PEP, I'm not sure that I understood it correctly,
> so here are some early questions more about the usage of the PEP, than
> its implementation.
> 
>> This is not sandboxing, as this proposal does not attempt to prevent
>> malicious behavior (though it enables some new options to do so).
>> See the `Why Not A Sandbox`_ section below for further discussion.
> 
> I don't understand well the overall security model. If malicious
> behaviors can still occur, what is the the purpose of auditing? For
> example, if an audit hook writes events into a local log file, the
> attacker can easily remove this log file, no?

An attacker may not have permission to mess with the auditing subsystem.
For example an attacker may be able to modify an application like a web
server or web application. Audit loggers typically run in a different,
more protected context. On modern, hardened operation systems root /
Adminstrator aren't all powerful, too. They are typically restricted by
additional policies like SELinux.

Further more, servers also send auditing data to remote nodes for
analysis. Keep in mind that auditing is not primarily about preventing
compromises. It's about detecting what, when, who, and how a system was
compromised.



>> Verified Open Hook
>> ------------------
>>
>> Most operating systems have a mechanism to distinguish between files
>> that can be executed and those that can not. For example, this may be an
>> execute bit in the permissions field, a verified hash of the file
>> contents to detect potential code tampering, or file system path
>> restrictions. These are an important security mechanism for preventing
>> execution of data or code that is not approved for a given environment.
>> Currently, Python has no way to integrate with these when launching
>> scripts or importing modules.
> 
> In my experience, it doesn't work just because Python has too many
> functions opening files indirectly or call external C libraries which
> open files.
> 
> I vaguely recall an exploit in my pysandbox project which uses the
> internal code of Python which displays a traceback... to read the
> content of an arbitrary file on the disk :-( Game over. I would never
> expect that there are so many ways to read a file in Python...

The verified open hook is not about sandboxing. It's a mechanism to
prevent a class of attacks like directory traversal attacks. On Linux,
the open-for-import hook could refuse access to .py and .pyc files that
do not have the user.python_code or root.python_code extended file
attribute. This verified open hook could have prevent the compromise of
wiki.python.org many years ago.


> Even when I restricted pysandbox to the bare minimum of the Python
> language (with no import), multiple exploits have been found.
> Moreover, at the end, Python just became useful.
> 
> More generally, there are a lot of codes in Python which allow
> arbitrary code injection :-( (Most of them are now fixed, hopefully!)
> 
> I did my best to modify as much functions as possible to implement the
> PEP 446 "Make newly created file descriptors non-inheritable", but I
> know that *many* functions call directly open() or fopen() and so
> create inheritable file descriptors. For example, the Python ssl
> module takes directly filenames and OpenSSL open directly files. It's
> just one example.
> 
> You will never be able to cover all cases.

I agree. Don't draw the wrong conclusion from your statement. PEP 578
adds hooks for auditing, which in return can be used to harden and log
an application. Unlike secure sandboxing, it doesn't have to be perfect.
Alex Gaynor summed this up in his blog post
https://alexgaynor.net/2018/jul/20/worst-truism-in-infosec/

> Having a single function which allows to open an arbitrary file
> without triggering an audit event would defeat the whole purpose of
> auditing, no? Again, maybe I didn't understand well the overall
> purpose of the PEP, sorry.

This case can be detected during development and QE phase. You simply
have to count the amount of open syscalls and compare it to the amount
of open auditing events.


>> The important performance impact is the case where events are being
>> raised but there are no hooks attached. This is the unavoidable case -
>> once a developer has added audit hooks they have explicitly chosen to
>> trade performance for functionality.
> 
> (The Linux kernel uses advance tooling to inject hooks: it has no
> impact on performances when no hook is used. Machine code of functions
> is patched to inject a hook. Impressive stuff :-))
> 
> Here I expect a small overhead. But the global overhead will be
> proportional to the number of hooks, no? Maybe it's not significant
> with the proposed list of events, but it will be more significant with
> 100 or 1000 events?
> 
> I'm not saying that it's a blocker issue, I'm just thinking aloud to
> make sure that I understood correctly :-)

The performance impact can be remedied and reduced with a simple check.
If there is no audit hook installed, it's just a matter of a pointer
deref + JNZ.

Christian


From christian at python.org  Fri Mar 29 06:34:20 2019
From: christian at python.org (Christian Heimes)
Date: Fri, 29 Mar 2019 11:34:20 +0100
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
Message-ID: <574f115f-2249-1920-d21c-caafed22125f@python.org>

On 28/03/2019 23.35, Steve Dower wrote:
> Audit Hook
> ----------
> 
> In order to observe actions taken by the runtime (on behalf of the
> caller), an API is required to raise messages from within certain
> operations. These operations are typically deep within the Python
> runtime or standard library, such as dynamic code compilation, module
> imports, DNS resolution, or use of certain modules such as ``ctypes``.
> 
> The following new C APIs allow embedders and CPython implementors to
> send and receive audit hook messages::
> 
> ?? # Add an auditing hook
> ?? typedef int (*hook_func)(const char *event, PyObject *args,
> ??????????????????????????? void *userData);
> ?? int PySys_AddAuditHook(hook_func hook, void *userData);
> 
> ?? # Raise an event with all auditing hooks
> ?? int PySys_Audit(const char *event, PyObject *args);
> 
> ?? # Internal API used during Py_Finalize() - not publicly accessible
> ?? void _Py_ClearAuditHooks(void);
> 
> The new Python APIs for receiving and raising audit hooks are::
> 
> ?? # Add an auditing hook
> ?? sys.addaudithook(hook: Callable[[str, tuple]])
> 
> ?? # Raise an event with all auditing hooks
> ?? sys.audit(str, *args)
> 
> 
> Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time,
> including before ``Py_Initialize()``, or by calling
> ``sys.addaudithook()`` from Python code. Hooks cannot be removed or
> replaced.

Hi Steve,

I wonder if the hooks could be replaced by a more efficient mechanism.
These days, Linux, macOS, and most recently Windows [1] support dtrace
probes. DTrace is a very powerful and efficient mechanism to trace
user-space processes from Kernel space. At least we should consider to
add DTrace probes to the auditing framework.

Regards,
Christian

[1]
https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902

From guido at python.org  Fri Mar 29 09:39:34 2019
From: guido at python.org (Guido van Rossum)
Date: Fri, 29 Mar 2019 06:39:34 -0700
Subject: [Python-Dev] A request for PEP announcement format [was: PEP
 570]
In-Reply-To: <5C9DBA56.2020006@UGent.be>
References: <CAFjbc8FG2ihd96pQXF_i9HXb2VO=0cMFZjG4_5AJReoJS4uc+w@mail.gmail.com>
 <23709.35898.906806.70203@turnbull.sk.tsukuba.ac.jp>
 <5C9DBA56.2020006@UGent.be>
Message-ID: <CAP7+vJ+-34UAS3WKUb6ONia_ghgiSztgEV1J65=vp71Qg7oQRg@mail.gmail.com>

On Thu, Mar 28, 2019 at 11:28 PM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:

> On 2019-03-29 04:08, Stephen J. Turnbull wrote:
> > In this case, it's here:
> >
> >   >
> https://discuss.python.org/t/pep-570-Python-Positional-Only-Parameters/1078
>
> So, are we supposed to discuss PEPs on discuss.python.org now? That's
> fine for me, should I create a thread like that for PEP 580 too?
>

It is up to the PEP author where a PEP will be discussed. You have the
option of using python-dev, Discourse or a dedicated mailing list.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190329/d01a2fc2/attachment.html>

From steve.dower at python.org  Fri Mar 29 11:29:31 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Mar 2019 08:29:31 -0700
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <574f115f-2249-1920-d21c-caafed22125f@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <574f115f-2249-1920-d21c-caafed22125f@python.org>
Message-ID: <75be29e3-a0ad-054a-8034-61bf8f03b177@python.org>

On 29Mar2019 0334, Christian Heimes wrote:
> On 28/03/2019 23.35, Steve Dower wrote:
>> Audit Hook
>> ----------
>>
>> In order to observe actions taken by the runtime (on behalf of the
>> caller), an API is required to raise messages from within certain
>> operations. These operations are typically deep within the Python
>> runtime or standard library, such as dynamic code compilation, module
>> imports, DNS resolution, or use of certain modules such as ``ctypes``.
>>
>> The following new C APIs allow embedders and CPython implementors to
>> send and receive audit hook messages::
>>
>>  ?? # Add an auditing hook
>>  ?? typedef int (*hook_func)(const char *event, PyObject *args,
>>  ??????????????????????????? void *userData);
>>  ?? int PySys_AddAuditHook(hook_func hook, void *userData);
>>
>>  ?? # Raise an event with all auditing hooks
>>  ?? int PySys_Audit(const char *event, PyObject *args);
>>
>>  ?? # Internal API used during Py_Finalize() - not publicly accessible
>>  ?? void _Py_ClearAuditHooks(void);
>>
>> The new Python APIs for receiving and raising audit hooks are::
>>
>>  ?? # Add an auditing hook
>>  ?? sys.addaudithook(hook: Callable[[str, tuple]])
>>
>>  ?? # Raise an event with all auditing hooks
>>  ?? sys.audit(str, *args)
>>
>>
>> Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time,
>> including before ``Py_Initialize()``, or by calling
>> ``sys.addaudithook()`` from Python code. Hooks cannot be removed or
>> replaced.
> 
> Hi Steve,
> 
> I wonder if the hooks could be replaced by a more efficient mechanism.
> These days, Linux, macOS, and most recently Windows [1] support dtrace
> probes. DTrace is a very powerful and efficient mechanism to trace
> user-space processes from Kernel space. At least we should consider to
> add DTrace probes to the auditing framework.
> 
> [1]
> https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902

Calling into those frameworks will still require as much work as these 
hooks do, and also make it very difficult to do things like auditing 
unit tests or using pure-Python code when you're not concerned about 
initialization (e.g. in a long-running web server).

So I'm inclined to say that if you want those probes, you can enable 
them by adding a hook that calls into them? A similar argument is made 
for using ETW on Windows (which will work on versions of Windows that 
have been released, unlike DTrace) (and yes, this is a real argument 
I've already had over this proposal ;) ), so I really think leaving it 
open-ended and Python-specific is the best approach.

(Reading further down the link you provided, it seems DTrace in Windows 
will only be enabled for essentially-administrators. So that rules it 
out as a substitute for this proposal in my opinion.)

Cheers,
Steve

From steve.dower at python.org  Fri Mar 29 11:43:42 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Mar 2019 08:43:42 -0700
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <4d1ef7fe-c235-f98e-4887-acc77606f485@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <CA+3bQGEodqbHk4A4fPAzRYeDmX0OG51DWN_Rq28aJfHt15UWdw@mail.gmail.com>
 <4d1ef7fe-c235-f98e-4887-acc77606f485@python.org>
Message-ID: <c9357baf-4ad4-8b6a-af63-32e953df58ea@python.org>

Thanks Christian for responding - I endorse and support all your 
comments. (I'd hoped that by explicitly saying "this is not a sandbox" 
it would avoid people thinking it was a sandbox, but apparently I would 
have been better just to avoid the keyword completely...)

On 29Mar2019 0324, Christian Heimes wrote:
> On 29/03/2019 01.02, Victor Stinner wrote:
>> Here I expect a small overhead. But the global overhead will be
>> proportional to the number of hooks, no? Maybe it's not significant
>> with the proposed list of events, but it will be more significant with
>> 100 or 1000 events?
>>
>> I'm not saying that it's a blocker issue, I'm just thinking aloud to
>> make sure that I understood correctly :-)
> 
> The performance impact can be remedied and reduced with a simple check.
> If there is no audit hook installed, it's just a matter of a pointer
> deref + JNZ.

Yep, the performance case we care about is when there are no hooks 
attached, since that's the only time a user cannot do anything to 
improve performance themselves. See the "Performance Impact" section in 
the PEP.

In my implementation the cost is about as low as I can make it - see 
https://github.com/python/cpython/pull/12613/files#diff-f38879f4833a6b6847e556b9a07bf4edR115 
(looking at it again I can probably avoid the exception preservation and 
a few conditionals at the end)

Basically, PySys_Audit takes a format string and arguments, rather than 
making callers eagerly construct the tuple that gets passed to the hook, 
and only actually allocates when there is a hook to call. There aren't 
even any Py_INCREF's if there are no hooks. And as Christian says, it's 
a deref+JNZ.

Now, if someone has implemented a hook and that hook has performance 
issues, yeah things will slow down. In general, the places where we are 
interested in hooks is where calls are being made into the operating 
system, so most of them will also involve a few syscalls and the cost of 
the hook should be minimal in comparison. But there isn't another way to 
provide the functionality - offloading it to the OS just means the OS is 
going to suffer the performance penalty, so it really is just moving the 
blame elsewhere. I dislike playing that game.

Cheers,
Steve

From vano at mail.mipt.ru  Fri Mar 29 13:44:02 2019
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Fri, 29 Mar 2019 20:44:02 +0300
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
Message-ID: <53df7704-bd34-34b8-3fd0-42896c78d238@mail.mipt.ru>

Like in the mktemp thread earlier, I would request a threat model (what use cases are supposed to be protected (in this case, by reporting 
rather than preventing) and from what threats) -- in the discussion, and eventually, in the PEP.
Without one, any claims and talks about whether something would be an effective security measure are pointless -- 'cuz you would never know 
if you accounted for everything and would not even have the definition of that "everything".

On 29.03.2019 1:35, Steve Dower wrote:
> Hi all
>
> Time is short, but I'm hoping to get PEP 578 (formerly PEP 551) into Python 3.8. Here's the current text for review and comment before I 
> submit to the Steering Council.
>
> The formatted text is at https://www.python.org/dev/peps/pep-0578/ (update just pushed, so give it an hour or so, but it's fundamentally 
> the same as what's there)
>
> No Discourse post, because we don't have a python-dev equivalent there yet, so please reply here for this one.
>
> Implementation is at https://github.com/zooba/cpython/tree/pep-578/ and my backport to 3.7 
> (https://github.com/zooba/cpython/tree/pep-578-3.7/) is already getting some real use (though this will not be added to 3.7, unless people 
> *really* want it, so the backport is just for reference).
>
> Cheers,
> Steve
>
> =====
>
> PEP: 578
> Title: Python Runtime Audit Hooks
> Version: $Revision$
> Last-Modified: $Date$
> Author: Steve Dower <steve.dower at python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 16-Jun-2018
> Python-Version: 3.8
> Post-History:
>
> Abstract
> ========
>
> This PEP describes additions to the Python API and specific behaviors
> for the CPython implementation that make actions taken by the Python
> runtime visible to auditing tools. Visibility into these actions
> provides opportunities for test frameworks, logging frameworks, and
> security tools to monitor and optionally limit actions taken by the
> runtime.
>
> This PEP proposes adding two APIs to provide insights into a running
> Python application: one for arbitrary events, and another specific to
> the module import system. The APIs are intended to be available in all
> Python implementations, though the specific messages and values used
> are unspecified here to allow implementations the freedom to determine
> how best to provide information to their users. Some examples likely
> to be used in CPython are provided for explanatory purposes.
>
> See PEP 551 for discussion and recommendations on enhancing the
> security of a Python runtime making use of these auditing APIs.
>
> Background
> ==========
>
> Python provides access to a wide range of low-level functionality on
> many common operating systems. While this is incredibly useful for
> "write-once, run-anywhere" scripting, it also makes monitoring of
> software written in Python difficult. Because Python uses native system
> APIs directly, existing monitoring tools either suffer from limited
> context or auditing bypass.
>
> Limited context occurs when system monitoring can report that an
> action occurred, but cannot explain the sequence of events leading to
> it. For example, network monitoring at the OS level may be able to
> report "listening started on port 5678", but may not be able to
> provide the process ID, command line, parent process, or the local
> state in the program at the point that triggered the action. Firewall
> controls to prevent such an action are similarly limited, typically
> to process names or some global state such as the current user, and
> in any case rarely provide a useful log file correlated with other
> application messages.
>
> Auditing bypass can occur when the typical system tool used for an
> action would ordinarily report its use, but accessing the APIs via
> Python do not trigger this. For example, invoking "curl" to make HTTP
> requests may be specifically monitored in an audited system, but
> Python's "urlretrieve" function is not.
>
> Within a long-running Python application, particularly one that
> processes user-provided information such as a web app, there is a risk
> of unexpected behavior. This may be due to bugs in the code, or
> deliberately induced by a malicious user. In both cases, normal
> application logging may be bypassed resulting in no indication that
> anything out of the ordinary has occurred.
>
> Additionally, and somewhat unique to Python, it is very easy to affect
> the code that is run in an application by manipulating either the
> import system's search path or placing files earlier on the path than
> intended. This is often seen when developers create a script with the
> same name as the module they intend to use - for example, a
> ``random.py`` file that attempts to import the standard library
> ``random`` module.
>
> This is not sandboxing, as this proposal does not attempt to prevent
> malicious behavior (though it enables some new options to do so).
> See the `Why Not A Sandbox`_ section below for further discussion.
>
> Overview of Changes
> ===================
>
> The aim of these changes is to enable both application developers and
> system administrators to integrate Python into their existing
> monitoring systems without dictating how those systems look or behave.
>
> We propose two API changes to enable this: an Audit Hook?and Verified
> Open Hook. Both are available from Python and native code, allowing
> applications and frameworks written in pure Python code to take
> advantage of the extra messages, while also allowing embedders or
> system administrators to deploy builds of Python where auditing is
> always enabled.
>
> Only CPython is bound to provide the native APIs as described here.
> Other implementations should provide the pure Python APIs, and
> may provide native versions as appropriate for their underlying
> runtimes. Auditing events are likewise considered implementation
> specific, but are bound by normal feature compatibility guarantees.
>
> Audit Hook
> ----------
>
> In order to observe actions taken by the runtime (on behalf of the
> caller), an API is required to raise messages from within certain
> operations. These operations are typically deep within the Python
> runtime or standard library, such as dynamic code compilation, module
> imports, DNS resolution, or use of certain modules such as ``ctypes``.
>
> The following new C APIs allow embedders and CPython implementors to
> send and receive audit hook messages::
>
> ?? # Add an auditing hook
> ?? typedef int (*hook_func)(const char *event, PyObject *args,
> ??????????????????????????? void *userData);
> ?? int PySys_AddAuditHook(hook_func hook, void *userData);
>
> ?? # Raise an event with all auditing hooks
> ?? int PySys_Audit(const char *event, PyObject *args);
>
> ?? # Internal API used during Py_Finalize() - not publicly accessible
> ?? void _Py_ClearAuditHooks(void);
>
> The new Python APIs for receiving and raising audit hooks are::
>
> ?? # Add an auditing hook
> ?? sys.addaudithook(hook: Callable[[str, tuple]])
>
> ?? # Raise an event with all auditing hooks
> ?? sys.audit(str, *args)
>
>
> Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time,
> including before ``Py_Initialize()``, or by calling
> ``sys.addaudithook()`` from Python code. Hooks cannot be removed or
> replaced.
>
> When events of interest are occurring, code can either call
> ``PySys_Audit()`` from C (while the GIL is held) or ``sys.audit()``. The
> string argument is the name of the event, and the tuple contains
> arguments. A given event name should have a fixed schema for arguments,
> which should be considered a public API (for each x.y version release),
> and thus should only change between feature releases with updated
> documentation.
>
> For maximum compatibility, events using the same name as an event in
> the reference interpreter CPython should make every attempt to use
> compatible arguments. Including the name or an abbreviation of the
> implementation in implementation-specific event names will also help
> prevent collisions. For example, a ``pypy.jit_invoked`` event is clearly
> distinguised from an ``ipy.jit_invoked`` event.
>
> When an event is audited, each hook is called in the order it was added
> with the event name and tuple. If any hook returns with an exception
> set, later hooks are ignored and *in general* the Python runtime should
> terminate. This is intentional to allow hook implementations to decide
> how to respond to any particular event. The typical responses will be to
> log the event, abort the operation with an exception, or to immediately
> terminate the process with an operating system exit call.
>
> When an event is audited but no hooks have been set, the ``audit()``
> function should impose minimal overhead. Ideally, each argument is a
> reference to existing data?rather than a value calculated just for the
> auditing call.
>
> As hooks may be Python objects, they need to be freed during
> ``Py_Finalize()``. To do this, we add an internal API
> ``_Py_ClearAuditHooks()`` that releases any Python hooks and any
> memory held. This is an internal function with no public export, and
> we recommend it raise its own audit event for all current hooks to
> ensure that unexpected calls are observed.
>
> Below in `Suggested Audit Hook Locations`_, we recommend some important
> operations that should raise audit events.
>
> Python implementations should document which operations will raise
> audit events, along with the event schema. It is intentional that
> ``sys.addaudithook(print)`` be a trivial way to display all messages.
>
> Verified Open Hook
> ------------------
>
> Most operating systems have a mechanism to distinguish between files
> that can be executed and those that can not. For example, this may be an
> execute bit in the permissions field, a verified hash of the file
> contents to detect potential code tampering, or file system path
> restrictions. These are an important security mechanism for preventing
> execution of data or code that is not approved for a given environment.
> Currently, Python has no way to integrate with these when launching
> scripts or importing modules.
>
> The new public C API for the verified open hook is::
>
> ?? # Set the handler
> ?? typedef PyObject *(*hook_func)(PyObject *path, void *userData)
> ?? int PyImport_SetOpenForImportHook(hook_func handler, void *userData)
>
> ?? # Open a file using the handler
> ?? PyObject *PyImport_OpenForImport(const char *path)
>
> The new public Python API for the verified open hook is::
>
> ?? # Open a file using the handler
> ?? importlib.util.open_for_import(path : str) -> io.IOBase
>
>
> The ``importlib.util.open_for_import()`` function is a drop-in
> replacement?for ``open(str(pathlike), 'rb')``. Its default behaviour is
> to open a file for raw, binary access. To change the behaviour a new
> handler should be set. Handler functions only accept ``str`` arguments.
> The C API ``PyImport_OpenForImport`` function assumes UTF-8 encoding.
>
> A custom handler may be set by calling ``PyImport_SetOpenForImportHook()``
> from C at any time, including before ``Py_Initialize()``. However, if a
> hook has already been set then the call will fail. When
> ``open_for_import()`` is called with a hook set, the hook will be passed
> the path and its return value will be returned directly. The returned
> object should be an open file-like object that supports reading raw
> bytes. This is explicitly intended to allow a ``BytesIO`` instance if
> the open handler has already read the entire file into memory.
>
> Note that these hooks can import and call the ``_io.open()`` function on
> CPython without triggering themselves. They can also use ``_io.BytesIO``
> to return a compatible result using an in-memory buffer.
>
> If the hook determines that the file should not be loaded, it should
> raise an exception of its choice, as well as performing any other
> logging.
>
> All import and execution functionality involving code from a file will
> be changed to use ``open_for_import()`` unconditionally. It is important
> to note that calls to ``compile()``, ``exec()`` and ``eval()`` do not go
> through this function - an audit hook that includes the code from these
> calls is the best opportunity to validate code that is read from the
> file. Given the current decoupling between import and execution in
> Python, most imported code will go through both ``open_for_import()``
> and the log hook for ``compile``, and so care should be taken to avoid
> repeating verification steps.
>
> There is no Python API provided for changing the open hook. To modify
> import behavior from Python code, use the existing functionality
> provided by ``importlib``.
>
> API Availability
> ----------------
>
> While all the functions added here are considered public and stable API,
> the behavior of the functions is implementation specific. Most
> descriptions here refer?to the CPython?implementation, and while other
> implementations should provide the functions, there is no requirement
> that they behave the same.
>
> For example, ``sys.addaudithook()`` and ``sys.audit()`` should exist but
> may do nothing. This allows code to make calls to ``sys.audit()``
> without having to test for existence, but it should not assume that its
> call will have any effect. (Including existence tests in
> security-critical code allows another vector to bypass auditing, so it
> is preferable that the function always exist.)
>
> ``importlib.util.open_for_import(path)`` should at a minimum always
> return ``_io.open(path, 'rb')``. Code using the function should make no
> further assumptions about what may occur, and implementations other than
> CPython are not required to let developers override the behavior of this
> function with a hook.
>
> Suggested Audit Hook Locations
> ==============================
>
> The locations and parameters in calls to ``sys.audit()`` or
> ``PySys_Audit()`` are to be determined by individual Python
> implementations. This is to allow maximum freedom for implementations
> to expose the operations that are most relevant to their platform,
> and to avoid or ignore potentially expensive or noisy events.
>
> Table 1 acts as both suggestions of operations that should trigger
> audit events on all implementations, and examples of event schemas.
>
> Table 2 provides further examples that are not required, but are
> likely to be available in CPython.
>
> Refer to the documentation associated with your version of Python to
> see which operations provide audit events.
>
> .. csv-table:: Table 1: Suggested Audit Hooks
> ?? :header: "API Function", "Event Name", "Arguments", "Rationale"
> ?? :widths: 2, 2, 3, 6
>
> ?? ``PySys_AddAuditHook``, ``sys.addaudithook``, "", "Detect when new
> ?? audit hooks are being added.
> ?? "
> ?? ``PyImport_SetOpenForImportHook``, ``setopenforimporthook``, "", "
> ?? Detects any attempt to set the ``open_for_import`` hook.
> ?? "
> ?? "``compile``, ``exec``, ``eval``, ``PyAst_CompileString``,
> ?? ``PyAST_obj2mod``", ``compile``, "``(code, filename_or_none)``", "
> ?? Detect dynamic code compilation, where ``code`` could be a string or
> ?? AST. Note that this will be called for regular imports of source
> ?? code, including those that were opened with ``open_for_import``.
> ?? "
> ?? "``exec``, ``eval``, ``run_mod``", ``exec``, "``(code_object,)``", "
> ?? Detect dynamic execution of code objects. This only occurs for
> ?? explicit calls, and is not raised for normal function invocation.
> ?? "
> ?? ``import``, ``import``, "``(module, filename, sys.path,
> ?? sys.meta_path, sys.path_hooks)``", "Detect when modules are
> ?? imported. This is raised before the module name is resolved to a
> ?? file. All arguments other than the module name may be ``None`` if
> ?? they are not used or available.
> ?? "
> ?? "``open``", ``open``, "``(path, mode, flags)``", "Detect when a file
> ?? is about to be opened. *path* and *mode* are the usual parameters to
> ?? ``open`` if available, while *flags* is provided instead of *mode*
> ?? in some cases.
> ?? "
> ?? ``PyEval_SetProfile``, ``sys.setprofile``, "", "Detect when code is
> ?? injecting trace functions. Because of the implementation, exceptions
> ?? raised from the hook will abort the operation, but will not be
> ?? raised in Python code. Note that ``threading.setprofile`` eventually
> ?? calls this function, so the event will be audited for each thread.
> ?? "
> ?? ``PyEval_SetTrace``, ``sys.settrace``, "", "Detect when code is
> ?? injecting trace functions. Because of the implementation, exceptions
> ?? raised from the hook will abort the operation, but will not be
> ?? raised in Python code. Note that ``threading.settrace`` eventually
> ?? calls this function, so the event will be audited for each thread.
> ?? "
> ?? "``_PyObject_GenericSetAttr``, ``check_set_special_type_attr``,
> ?? ``object_set_class``, ``func_set_code``, ``func_set_[kw]defaults``","
> ?? ``object.__setattr__``","``(object, attr, value)``","Detect monkey
> ?? patching of types and objects. This event
> ?? is raised for the ``__class__`` attribute and any attribute on
> ?? ``type`` objects.
> ?? "
> "``_PyObject_GenericSetAttr``",``object.__delattr__``,"``(object,
> ?? attr)``","Detect deletion of object attributes. This event is raised
> ?? for any attribute on ``type`` objects.
> ?? "
> "``Unpickler.find_class``",``pickle.find_class``,"``(module_name,
> ?? global_name)``","Detect imports and global name lookup when
> ?? unpickling.
> ?? "
>
>
> .. csv-table:: Table 2: Potential CPython Audit Hooks
> ?? :header: "API Function", "Event Name", "Arguments", "Rationale"
> ?? :widths: 2, 2, 3, 6
>
> ?? ``_PySys_ClearAuditHooks``, ``sys._clearaudithooks``, "", "Notifies
> ?? hooks they are being cleaned up, mainly in case the event is
> ?? triggered unexpectedly. This event cannot be aborted.
> ?? "
> ?? ``code_new``, ``code.__new__``, "``(bytecode, filename, name)``", "
> ?? Detect dynamic creation of code objects. This only occurs for
> ?? direct instantiation, and is not raised for normal compilation.
> ?? "
> ?? ``func_new_impl``, ``function.__new__``, "``(code,)``", "Detect
> ?? dynamic creation of function objects. This only occurs for direct
> ?? instantiation, and is not raised for normal compilation.
> ?? "
> ?? "``_ctypes.dlopen``, ``_ctypes.LoadLibrary``", ``ctypes.dlopen``, "
> ?? ``(module_or_path,)``", "Detect when native modules are used.
> ?? "
> ?? ``_ctypes._FuncPtr``, ``ctypes.dlsym``, "``(lib_object, name)``", "
> ?? Collect information about specific symbols retrieved from native
> ?? modules.
> ?? "
> ?? ``_ctypes._CData``, ``ctypes.cdata``, "``(ptr_as_int,)``", "Detect
> ?? when code is accessing arbitrary memory using ``ctypes``.
> ?? "
> ?? "``new_mmap_object``",``mmap.__new__``,"``(fileno, map_size, access,
> ?? offset)``", "Detects creation of mmap objects. On POSIX, access may
> ?? have been calculated from the ``prot`` and ``flags`` arguments.
> ?? "
> ?? ``sys._getframe``, ``sys._getframe``, "``(frame_object,)``", "Detect
> ?? when code is accessing frames directly.
> ?? "
> ?? ``sys._current_frames``, ``sys._current_frames``, "", "Detect when
> ?? code is accessing frames directly.
> ?? "
> ?? "``socket.bind``, ``socket.connect``, ``socket.connect_ex``,
> ?? ``socket.getaddrinfo``, ``socket.getnameinfo``, ``socket.sendmsg``,
> ?? ``socket.sendto``", ``socket.address``, "``(address,)``", "Detect
> ?? access to network resources. The address is unmodified from the
> ?? original call.
> ?? "
> ?? "``member_get``, ``func_get_code``, ``func_get_[kw]defaults``
> ?? ",``object.__getattr__``,"``(object, attr)``","Detect access to
> ?? restricted attributes. This event is raised for any built-in
> ?? members that are marked as restricted, and members that may allow
> ?? bypassing imports.
> ?? "
> ?? "``urllib.urlopen``",``urllib.Request``,"``(url, data, headers,
> ?? method)``", "Detects URL requests.
> ?? "
>
> Performance Impact
> ==================
>
> The important performance impact is the case where events are being
> raised but there are no hooks attached. This is the unavoidable case -
> once a developer has added audit hooks they have explicitly chosen to
> trade performance for functionality. Performance impact with hooks added
> are not of interest here, since this is opt-in functionality.
>
> Analysis using the Python Performance Benchmark Suite [1]_ shows no
> significant impact, with the vast majority of benchmarks showing
> between 1.05x faster to 1.05x slower.
>
> In our opinion, the performance impact of the set of auditing points
> described in this PEP is negligible.
>
> Rejected Ideas
> ==============
>
> Separate module for audit hooks
> -------------------------------
>
> The proposal is to add a new module for audit hooks, hypothetically
> ``audit``. This would separate the API and implementation from the
> ``sys`` module, and allow naming the C functions ``PyAudit_AddHook`` and
> ``PyAudit_Audit`` rather than the current variations.
>
> Any such module would need to be a built-in module that is guaranteed to
> always be present. The nature of these hooks is that they must be
> callable without condition, as any conditional imports or calls provide
> opportunities to intercept and suppress or modify events.
>
> Given it is one of the most core modules, the ``sys`` module is somewhat
> protected against module shadowing attacks. Replacing ``sys`` with a
> sufficiently functional module that the application can still run is a
> much more complicated task than replacing a module with only one
> function of interest. An attacker that has the ability to shadow the
> ``sys`` module is already capable of running arbitrary code from files,
> whereas an ``audit`` module could be replaced with a single line in a
> ``.pth`` file anywhere on the search path::
>
> ??? import sys; sys.modules['audit'] = type('audit', (object,),
> ??????? {'audit': lambda *a: None, 'addhook': lambda *a: None})
>
> Multiple layers of protection already exist for monkey patching attacks
> against either ``sys`` or ``audit``, but assignments or insertions to
> ``sys.modules`` are not audited.
>
> This idea is rejected because it makes it trivial to suppress all calls
> to ``audit``.
>
> Flag in sys.flags to indicate "audited" mode
> --------------------------------------------
>
> The proposal is to add a value in ``sys.flags`` to indicate when Python
> is running in a "secure" or "audited" mode. This would allow
> applications to detect when some features are enabled or when hooks
> have been added and modify their behaviour appropriately.
>
> Currently, we are not aware of any legitimate reasons for a program to
> behave differently in the presence of audit hooks.
>
> Both application-level APIs ``sys.audit`` and
> ``importlib.util.open_for_import`` are always present and functional,
> regardless of whether the regular ``python`` entry point or some
> alternative entry point is used. Callers cannot determine whether any
> hooks have been added (except by performing side-channel analysis), nor
> do they need to. The calls should be fast enough that callers do not
> need to avoid them, and the program is responsible for ensuring that
> any added hooks are fast enough to not affect application performance.
>
> The argument that this is "security by obscurity" is valid, but
> irrelevant. Security by obscurity is only an issue when there are no
> other protective mechanisms; obscurity as the first step in avoiding
> attack is strongly recommended (see `this article
> <https://danielmiessler.com/study/security-by-obscurity/>`_ for
> discussion).
>
> This idea is rejected because there are no appropriate reasons for an
> application to change its behaviour based on whether these APIs are in
> use.
>
> Why Not A Sandbox
> =================
>
> Sandboxing CPython has been attempted many times in the past, and each
> past attempt has failed. Fundamentally, the problem is that certain
> functionality has to be restricted when executing the sandboxed code,
> but otherwise needs to be available for normal operation of Python. For
> example, completely removing the ability to compile strings into
> bytecode also breaks the ability to import modules from source code, and
> if it is not completely removed then there are too many ways to get
> access to that functionality indirectly. There is not yet any feasible
> way to generically determine whether a given operation is "safe" or not.
> Further information and references available at [2]_.
>
> This proposal does not attempt to restrict functionality, but simply
> exposes the fact that the functionality is being used. Particularly for
> intrusion scenarios, detection is significantly more important than
> early prevention (as early prevention will generally drive attackers to
> use an alternate, less-detectable, approach). The availability of audit
> hooks alone does not change the attack surface of Python in any way, but
> they enable defenders to integrate Python into their environment in ways
> that are currently not possible.
>
> Since audit hooks have the ability to safely prevent an operation
> occuring, this feature does enable the ability to provide some level of
> sandboxing. In most cases, however, the intention is to enable logging
> rather than creating a sandbox.
>
> Relationship to PEP 551
> =======================
>
> This API was originally presented as part of
> `PEP 551 <https://www.python.org/dev/peps/pep-0551/>`_ Security
> Transparency in the Python Runtime.
>
> For simpler review purposes, and due to the broader applicability of
> these APIs beyond security, the API design is now presented separately.
>
> PEP 551 is an informational PEP discussing how to integrate Python into
> a secure or audited environment.
>
> References
> ==========
>
> .. [1] Python Performance Benchmark Suite `<https://github.com/python/performance>`_
>
> .. [2] Python Security model - Sandbox `<https://python-security.readthedocs.io/security.html#sandbox>`_
>
> Copyright
> =========
>
> Copyright (c) 2019 by Microsoft Corporation. This material may be
> distributed only subject to the terms and conditions set forth in the
> Open Publication License, v1.0 or later (the latest version is presently
> available at http://www.opencontent.org/openpub/).
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- 
Regards,
Ivan


From status at bugs.python.org  Fri Mar 29 14:08:08 2019
From: status at bugs.python.org (Python tracker)
Date: Fri, 29 Mar 2019 18:08:08 +0000 (UTC)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190329180808.804A552B01A@bugs.ams1.psf.io>


ACTIVITY SUMMARY (2019-03-22 - 2019-03-29)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    7048 ( -3)
  closed 41176 (+76)
  total  48224 (+73)

Open issues with patches: 2809 


Issues opened (50)
==================

#22189: collections.UserString missing some str methods
https://bugs.python.org/issue22189  reopened by rhettinger

#28190: Cross-build _curses failed if host ncurses headers and target 
https://bugs.python.org/issue28190  reopened by doko

#33356: Windows 10 buildbot: test__xxsubinterpreters.test_already_runn
https://bugs.python.org/issue33356  reopened by vstinner

#35155: Clarify Protocol Handlers in urllib.request Docs
https://bugs.python.org/issue35155  reopened by orsenthil

#36150: Possible assertion failures due to _ctypes.c's PyCData_reduce(
https://bugs.python.org/issue36150  reopened by serhiy.storchaka

#36344: install_certificates.command too complicated
https://bugs.python.org/issue36344  reopened by ned.deily

#36402: test_threading: test_threads_join_2() failed with "Fatal Pytho
https://bugs.python.org/issue36402  opened by vstinner

#36403: AsyncIterator on 3.7: __aiter__ no longer honors finally block
https://bugs.python.org/issue36403  opened by asksol

#36404: Document PendingDeprecationWarning as deprecated
https://bugs.python.org/issue36404  opened by inada.naoki

#36406: doctest.testmod(empty_package) raises TypeError in 3.7 (and no
https://bugs.python.org/issue36406  opened by Dutcho

#36408: Tkinter multi-processing performance, Linux 10-25 times faster
https://bugs.python.org/issue36408  opened by james.mccormack

#36409: plistlib old API should be removed
https://bugs.python.org/issue36409  opened by bigfootjon

#36411: Python 3 f.tell() gets out of sync with file pointer in binary
https://bugs.python.org/issue36411  opened by pewscorner

#36414: Multiple test failures in GCC and Clang optional builds on Tra
https://bugs.python.org/issue36414  opened by xtreak

#36416: bytes.rpartition bug in online documentation
https://bugs.python.org/issue36416  opened by pewscorner

#36417: unicode.isdecimal bug in online Python 2 documentation
https://bugs.python.org/issue36417  opened by pewscorner

#36418: urllib.parse.*Result: support _replace for additional computed
https://bugs.python.org/issue36418  opened by cheryl.sabella

#36419: IDLE autocomplete: refactor and polish code and tests
https://bugs.python.org/issue36419  opened by terry.reedy

#36422: tempfile.TemporaryDirectory() removes entire directory tree ev
https://bugs.python.org/issue36422  opened by riccardomurri

#36424: Pickle fails on frozen dataclass that has slots
https://bugs.python.org/issue36424  opened by drhagen

#36426: exec() issue when used inside function
https://bugs.python.org/issue36426  opened by schperplata

#36427: Document that PyEval_RestoreThread and PyGILState_Ensure can t
https://bugs.python.org/issue36427  opened by pablogsal

#36429: Fix starting IDLE with pyshell
https://bugs.python.org/issue36429  opened by terry.reedy

#36432: Running python test suite fails on macOS 10.14.4 with resource
https://bugs.python.org/issue36432  opened by ned.deily

#36434: Zipfile breaks if signalled during write()
https://bugs.python.org/issue36434  opened by and800

#36437: method_descriptor surprising error message when self is passed
https://bugs.python.org/issue36437  opened by vstinner

#36438: Python 3.5.7 import error on Cross compile
https://bugs.python.org/issue36438  opened by Brian Spratke

#36439: Inconsistencies with datetime.fromtimestamp(t) when t < 0
https://bugs.python.org/issue36439  opened by Tiger-222

#36440: more helpful diagnostics for parser module
https://bugs.python.org/issue36440  opened by A. Skrobov

#36442: Different ValueError for the same operation in List and Tuple
https://bugs.python.org/issue36442  opened by HardikPatel

#36445: bus error in test_gil test on armhf running with 64bit kernel
https://bugs.python.org/issue36445  opened by doko

#36448: Message "You will need to rebuild pythoncore to see the change
https://bugs.python.org/issue36448  opened by jdemeyer

#36449: __aexit__ is not called when a context manager is used in an a
https://bugs.python.org/issue36449  opened by Andrey Lemets

#36453: pkgutil.get_importer only return the first valid path_hook(imp
https://bugs.python.org/issue36453  opened by Windson Yang

#36454: test_time: test_monotonic() failed on AMD64 FreeBSD 10-STABLE 
https://bugs.python.org/issue36454  opened by vstinner

#36456: task.cancel unbound recursion
https://bugs.python.org/issue36456  opened by Dima.Tisnek

#36457: functools.singledispatchmethod interacts poorly with subclasse
https://bugs.python.org/issue36457  opened by Tim Mitchell2

#36460: Add AMP MIME type support
https://bugs.python.org/issue36460  opened by Daniel Black

#36461: timeit: Additional changes for autorange
https://bugs.python.org/issue36461  opened by cheryl.sabella

#36463: python37.dll crashing 0xc000041d
https://bugs.python.org/issue36463  opened by Savagery

#36464: Python 2.7 build install fails intermittently with -j on MacOS
https://bugs.python.org/issue36464  opened by madscientist

#36465: No longer enable Py_TRACE_REFS by default in debug build
https://bugs.python.org/issue36465  opened by vstinner

#36466: Adding a way to strip annotations from compiled bytecode
https://bugs.python.org/issue36466  opened by cary

#36467: IDLE to help with invisible characters
https://bugs.python.org/issue36467  opened by rhettinger

#36468: Treeview: wrong color change
https://bugs.python.org/issue36468  opened by guillaumeb

#36469: Stuck during interpreter exit, attempting to take the GIL
https://bugs.python.org/issue36469  opened by mocramis

#36470: dataclasses.replace raises an exception if InitVar with defaul
https://bugs.python.org/issue36470  opened by Greg Kuhn

#36471: PEP 432, PEP 587: Add _Py_RunMain()
https://bugs.python.org/issue36471  opened by vstinner

#36472: Some old PR with CLA not signed
https://bugs.python.org/issue36472  opened by eamanu

#36473: Detect all dictionary changes during iteration
https://bugs.python.org/issue36473  opened by thomas.perl



Most recent 15 issues with no replies (15)
==========================================

#36468: Treeview: wrong color change
https://bugs.python.org/issue36468

#36467: IDLE to help with invisible characters
https://bugs.python.org/issue36467

#36466: Adding a way to strip annotations from compiled bytecode
https://bugs.python.org/issue36466

#36457: functools.singledispatchmethod interacts poorly with subclasse
https://bugs.python.org/issue36457

#36456: task.cancel unbound recursion
https://bugs.python.org/issue36456

#36454: test_time: test_monotonic() failed on AMD64 FreeBSD 10-STABLE 
https://bugs.python.org/issue36454

#36445: bus error in test_gil test on armhf running with 64bit kernel
https://bugs.python.org/issue36445

#36438: Python 3.5.7 import error on Cross compile
https://bugs.python.org/issue36438

#36437: method_descriptor surprising error message when self is passed
https://bugs.python.org/issue36437

#36434: Zipfile breaks if signalled during write()
https://bugs.python.org/issue36434

#36424: Pickle fails on frozen dataclass that has slots
https://bugs.python.org/issue36424

#36419: IDLE autocomplete: refactor and polish code and tests
https://bugs.python.org/issue36419

#36418: urllib.parse.*Result: support _replace for additional computed
https://bugs.python.org/issue36418

#36417: unicode.isdecimal bug in online Python 2 documentation
https://bugs.python.org/issue36417

#36404: Document PendingDeprecationWarning as deprecated
https://bugs.python.org/issue36404



Most recent 15 issues waiting for review (15)
=============================================

#36473: Detect all dictionary changes during iteration
https://bugs.python.org/issue36473

#36471: PEP 432, PEP 587: Add _Py_RunMain()
https://bugs.python.org/issue36471

#36466: Adding a way to strip annotations from compiled bytecode
https://bugs.python.org/issue36466

#36465: No longer enable Py_TRACE_REFS by default in debug build
https://bugs.python.org/issue36465

#36448: Message "You will need to rebuild pythoncore to see the change
https://bugs.python.org/issue36448

#36440: more helpful diagnostics for parser module
https://bugs.python.org/issue36440

#36434: Zipfile breaks if signalled during write()
https://bugs.python.org/issue36434

#36429: Fix starting IDLE with pyshell
https://bugs.python.org/issue36429

#36427: Document that PyEval_RestoreThread and PyGILState_Ensure can t
https://bugs.python.org/issue36427

#36416: bytes.rpartition bug in online documentation
https://bugs.python.org/issue36416

#36409: plistlib old API should be removed
https://bugs.python.org/issue36409

#36406: doctest.testmod(empty_package) raises TypeError in 3.7 (and no
https://bugs.python.org/issue36406

#36404: Document PendingDeprecationWarning as deprecated
https://bugs.python.org/issue36404

#36398: A possible crash in structseq.c's structseq_repr()
https://bugs.python.org/issue36398

#36394: test_multiprocessing_spawn fails on Windows7 3.x buildbot
https://bugs.python.org/issue36394



Top 10 most discussed issues (10)
=================================

#36345: Deprecate Tools/scripts/serve.py in favour of python -m http.s
https://bugs.python.org/issue36345  12 msgs

#36305: Inconsistent behavior of pathlib.WindowsPath with drive paths
https://bugs.python.org/issue36305  10 msgs

#36347: Renaming the constants for the .flags of PyMemberDef
https://bugs.python.org/issue36347  10 msgs

#35983: tp_dealloc trashcan shouldn't be called for subclasses
https://bugs.python.org/issue35983   8 msgs

#36460: Add AMP MIME type support
https://bugs.python.org/issue36460   6 msgs

#36414: Multiple test failures in GCC and Clang optional builds on Tra
https://bugs.python.org/issue36414   5 msgs

#36427: Document that PyEval_RestoreThread and PyGILState_Ensure can t
https://bugs.python.org/issue36427   5 msgs

#36469: Stuck during interpreter exit, attempting to take the GIL
https://bugs.python.org/issue36469   5 msgs

#10716: Modernize pydoc to use better HTML and separate CSS
https://bugs.python.org/issue10716   4 msgs

#31327: bug in dateutil\tz\tz.py
https://bugs.python.org/issue31327   4 msgs



Issues closed (78)
==================

#5340: Change in cgi behavior breaks existing software
https://bugs.python.org/issue5340  closed by inada.naoki

#6422: timeit called from within Python should allow autoranging
https://bugs.python.org/issue6422  closed by cheryl.sabella

#12445: dict view values objects are missing tp_richcmp and tp_as_numb
https://bugs.python.org/issue12445  closed by inada.naoki

#19879: imageop: bug in error handler
https://bugs.python.org/issue19879  closed by inada.naoki

#21019: PyMethodDef ml_name is char* instead of const char*
https://bugs.python.org/issue21019  closed by inada.naoki

#21022: PyMemberDef doc is char* instead of const char*
https://bugs.python.org/issue21022  closed by inada.naoki

#22327: test_gdb failures on Ubuntu 14.10
https://bugs.python.org/issue22327  closed by cheryl.sabella

#22503: Signal stack overflow in faulthandler_user
https://bugs.python.org/issue22503  closed by vstinner

#23126: Add Python hook function to replace NameError
https://bugs.python.org/issue23126  closed by inada.naoki

#23205: Unit test needed for IDLE's GrepDialog.py's findfiles()
https://bugs.python.org/issue23205  closed by cheryl.sabella

#25614: Lib/code.py: InteractiveConsole.raw_input writes prompt to std
https://bugs.python.org/issue25614  closed by cheryl.sabella

#27464: Document that SplitResult & friends are namedtuples
https://bugs.python.org/issue27464  closed by martin.panter

#27797: ASCII file with UNIX line conventions and enough lines throws 
https://bugs.python.org/issue27797  closed by inada.naoki

#30348: IDLE: Add test_autocomplete unittests
https://bugs.python.org/issue30348  closed by terry.reedy

#31292: `python setup.py check --restructuredtext` fails when a includ
https://bugs.python.org/issue31292  closed by cheryl.sabella

#31822: Document that urllib.parse.{Defrag,Split,Parse}Result are name
https://bugs.python.org/issue31822  closed by cheryl.sabella

#32217: freeze.py fails to work.
https://bugs.python.org/issue32217  closed by cheryl.sabella

#32419: Add unittest support for pyc projects
https://bugs.python.org/issue32419  closed by inada.naoki

#32823: Regression in test -j behavior and time in 3.7.0b1
https://bugs.python.org/issue32823  closed by terry.reedy

#32950: profiling python gc
https://bugs.python.org/issue32950  closed by inada.naoki

#33043: Add a 'Contributing to Docs' link at the bottom of docs.python
https://bugs.python.org/issue33043  closed by willingc

#33319: `subprocess.run` documentation doesn't tell is using `stdout=P
https://bugs.python.org/issue33319  closed by gregory.p.smith

#33832: doc Add "magic method" entry to Glossary
https://bugs.python.org/issue33832  closed by eric.araujo

#34085: doc Improve wording on classmethod/staticmethod
https://bugs.python.org/issue34085  closed by cheryl.sabella

#34745: asyncio ssl memory leak
https://bugs.python.org/issue34745  closed by thehesiod

#34826: io.BufferedReader crashes in 2.7 on memoryview attribute acces
https://bugs.python.org/issue34826  closed by inada.naoki

#34995: functools.cached_property does not maintain the wrapped method
https://bugs.python.org/issue34995  closed by inada.naoki

#35165: Possible wrong method name in attribute references doc
https://bugs.python.org/issue35165  closed by cheryl.sabella

#35194: A typo in a constant in cp932 codec
https://bugs.python.org/issue35194  closed by inada.naoki

#35528: [DOC] [LaTeX] Sphinx 2.0 uses GNU FreeFont as default for xela
https://bugs.python.org/issue35528  closed by mdk

#36064: docs: urllib.request.Request not accepting iterables data type
https://bugs.python.org/issue36064  closed by mdk

#36143: Auto-generate Lib/keyword.py
https://bugs.python.org/issue36143  closed by pablogsal

#36202: Calling Py_DecodeLocale() before _PyPreConfig_Write() can prod
https://bugs.python.org/issue36202  closed by vstinner

#36218: .sort() segfaults consistently on crafted input
https://bugs.python.org/issue36218  closed by rhettinger

#36236: Python crash on macOS when CWD is invalid
https://bugs.python.org/issue36236  closed by ned.deily

#36245: PCBuild/build.bat errors, probably from space characters in pa
https://bugs.python.org/issue36245  closed by steve.dower

#36260: Cpython/Lib vulnerability found and request a patch submission
https://bugs.python.org/issue36260  closed by krnick

#36286: Random failure in test_idle
https://bugs.python.org/issue36286  closed by serhiy.storchaka

#36298: Lib/pyclbr.py crashes when the package spec cannot be determin
https://bugs.python.org/issue36298  closed by brett.cannon

#36301: Add _Py_PreInitialize() function
https://bugs.python.org/issue36301  closed by vstinner

#36318: Adding support for setting the "disabled" attribute of loggers
https://bugs.python.org/issue36318  closed by maggyero

#36326: Teach inpsect.getdoc() to read __slots__ with an optional data
https://bugs.python.org/issue36326  closed by rhettinger

#36340: 3.7.3rc1 Install Certificates fails on macOS
https://bugs.python.org/issue36340  closed by ned.deily

#36354: Use CreateProcessW for Python 2.7 on Windows.
https://bugs.python.org/issue36354  closed by gregory.p.smith

#36366: Patcher stop method should be idempotent
https://bugs.python.org/issue36366  closed by brett.cannon

#36370: Check for PyErr_Occurred() after PyImport_GetModule().
https://bugs.python.org/issue36370  closed by skrah

#36381: Deprecate "#" argument format without PY_SSIZE_T_CLEAN
https://bugs.python.org/issue36381  closed by inada.naoki

#36387: Refactor getenvironment() in _winapi.c
https://bugs.python.org/issue36387  closed by serhiy.storchaka

#36396: Remove fgBg param of idlelib.config.GetHighlight()
https://bugs.python.org/issue36396  closed by terry.reedy

#36401: Readonly properties should be marked as such in help()
https://bugs.python.org/issue36401  closed by rhettinger

#36405: Use dict unpacking in idlelib
https://bugs.python.org/issue36405  closed by terry.reedy

#36407: xml.dom.minidom wrong indentation writing for CDATA section
https://bugs.python.org/issue36407  closed by serhiy.storchaka

#36410: Proposal to make strip/lstrip/rstrip more explicit
https://bugs.python.org/issue36410  closed by rhettinger

#36412: A possible crash in dictobject.c's new_dict()
https://bugs.python.org/issue36412  closed by inada.naoki

#36413: Brief Tour of the Standard Library ??? Part II example problem
https://bugs.python.org/issue36413  closed by rhettinger

#36415: [math] Implement pow2 function
https://bugs.python.org/issue36415  closed by serhiy.storchaka

#36420: f_trace_opcodes setting and accessing opcodes
https://bugs.python.org/issue36420  closed by anthony shaw

#36421: A possible double decref in _ctypes.c's PyCArrayType_new()
https://bugs.python.org/issue36421  closed by serhiy.storchaka

#36423: www.pythontest.net is down
https://bugs.python.org/issue36423  closed by vstinner

#36425: Add Simplified Chinese to the language switcher
https://bugs.python.org/issue36425  closed by mdk

#36428: Support AutoAwait?
https://bugs.python.org/issue36428  closed by gvanrossum

#36430: A possible reference leak in itertools.count()
https://bugs.python.org/issue36430  closed by serhiy.storchaka

#36431: Use dict unpacking for merging two dicts
https://bugs.python.org/issue36431  closed by serhiy.storchaka

#36433: Unhelpful error message in classmethoddescr_call()
https://bugs.python.org/issue36433  closed by inada.naoki

#36435: multiprocessing crashes in Python 3.7.3 on Windows
https://bugs.python.org/issue36435  closed by paul.moore

#36436: _testcapi.pymem_buffer_overflow() doesn't handle memory alloca
https://bugs.python.org/issue36436  closed by vstinner

#36441: Cannot create venv with debug binaries installed
https://bugs.python.org/issue36441  closed by steve.dower

#36443: Disable coerce_c_locale and utf8_mode by default in _PyPreConf
https://bugs.python.org/issue36443  closed by vstinner

#36444: Python initialization: remove _PyMainInterpreterConfig
https://bugs.python.org/issue36444  closed by vstinner

#36446: Need Windows x86-64 MSI installer for Python >= 3.x.x
https://bugs.python.org/issue36446  closed by zach.ware

#36447: test__xxsubinterpreters leaked references and memory blocks
https://bugs.python.org/issue36447  closed by vstinner

#36450: 3D Array Assignment to all 2D sub-arrays
https://bugs.python.org/issue36450  closed by eric.snow

#36451: Docs zh-cn roots error in nav bar
https://bugs.python.org/issue36451  closed by mdk

#36452: Detect dict iteration "overflow" when changing keys
https://bugs.python.org/issue36452  closed by inada.naoki

#36455: collections.abc.Sequence doesn't support reflection/introspect
https://bugs.python.org/issue36455  closed by josh.r

#36458: Compile errors --without-threads
https://bugs.python.org/issue36458  closed by ned.deily

#36459: A possible double PyMem_FREE() due to tokenizer.c's tok_nextc(
https://bugs.python.org/issue36459  closed by serhiy.storchaka

#36462: CVE-2019-9674 : zip bomb vulnerability in Lib/zipfile.py
https://bugs.python.org/issue36462  closed by brett.cannon

From steve.dower at python.org  Fri Mar 29 14:12:23 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Mar 2019 11:12:23 -0700
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <53df7704-bd34-34b8-3fd0-42896c78d238@mail.mipt.ru>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <53df7704-bd34-34b8-3fd0-42896c78d238@mail.mipt.ru>
Message-ID: <6c198d4f-7efd-7b4a-5a42-44e39faf7807@python.org>

PEP 551 (referenced from this one) contains information about using 
these hooks for security purposes, along with other approaches to 
minimize the risk of having Python in your production environments. 
Threat models have to be designed by the user; we can't predict what it 
looks like for the incredibly diverse user base we have.

This PEP is explicitly only about the API changes.

Cheers,
Steve

On 29Mar2019 1044, Ivan Pozdeev via Python-Dev wrote:
> Like in the mktemp thread earlier, I would request a threat model (what 
> use cases are supposed to be protected (in this case, by reporting 
> rather than preventing) and from what threats) -- in the discussion, and 
> eventually, in the PEP.
> Without one, any claims and talks about whether something would be an 
> effective security measure are pointless -- 'cuz you would never know if 
> you accounted for everything and would not even have the definition of 
> that "everything".
> 
> On 29.03.2019 1:35, Steve Dower wrote:
>> Hi all
>>
>> Time is short, but I'm hoping to get PEP 578 (formerly PEP 551) into 
>> Python 3.8. Here's the current text for review and comment before I 
>> submit to the Steering Council.
>>
>> The formatted text is at https://www.python.org/dev/peps/pep-0578/ 
>> (update just pushed, so give it an hour or so, but it's fundamentally 
>> the same as what's there)
>>
>> No Discourse post, because we don't have a python-dev equivalent there 
>> yet, so please reply here for this one.
>>
>> Implementation is at https://github.com/zooba/cpython/tree/pep-578/ 
>> and my backport to 3.7 
>> (https://github.com/zooba/cpython/tree/pep-578-3.7/) is already 
>> getting some real use (though this will not be added to 3.7, unless 
>> people *really* want it, so the backport is just for reference).
>>
>> Cheers,
>> Steve
>>
>> =====
>>
>> PEP: 578
>> Title: Python Runtime Audit Hooks
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Steve Dower <steve.dower at python.org>
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 16-Jun-2018
>> Python-Version: 3.8
>> Post-History:
>>
>> Abstract
>> ========
>>
>> This PEP describes additions to the Python API and specific behaviors
>> for the CPython implementation that make actions taken by the Python
>> runtime visible to auditing tools. Visibility into these actions
>> provides opportunities for test frameworks, logging frameworks, and
>> security tools to monitor and optionally limit actions taken by the
>> runtime.
>>
>> This PEP proposes adding two APIs to provide insights into a running
>> Python application: one for arbitrary events, and another specific to
>> the module import system. The APIs are intended to be available in all
>> Python implementations, though the specific messages and values used
>> are unspecified here to allow implementations the freedom to determine
>> how best to provide information to their users. Some examples likely
>> to be used in CPython are provided for explanatory purposes.
>>
>> See PEP 551 for discussion and recommendations on enhancing the
>> security of a Python runtime making use of these auditing APIs.
>>
>> Background
>> ==========
>>
>> Python provides access to a wide range of low-level functionality on
>> many common operating systems. While this is incredibly useful for
>> "write-once, run-anywhere" scripting, it also makes monitoring of
>> software written in Python difficult. Because Python uses native system
>> APIs directly, existing monitoring tools either suffer from limited
>> context or auditing bypass.
>>
>> Limited context occurs when system monitoring can report that an
>> action occurred, but cannot explain the sequence of events leading to
>> it. For example, network monitoring at the OS level may be able to
>> report "listening started on port 5678", but may not be able to
>> provide the process ID, command line, parent process, or the local
>> state in the program at the point that triggered the action. Firewall
>> controls to prevent such an action are similarly limited, typically
>> to process names or some global state such as the current user, and
>> in any case rarely provide a useful log file correlated with other
>> application messages.
>>
>> Auditing bypass can occur when the typical system tool used for an
>> action would ordinarily report its use, but accessing the APIs via
>> Python do not trigger this. For example, invoking "curl" to make HTTP
>> requests may be specifically monitored in an audited system, but
>> Python's "urlretrieve" function is not.
>>
>> Within a long-running Python application, particularly one that
>> processes user-provided information such as a web app, there is a risk
>> of unexpected behavior. This may be due to bugs in the code, or
>> deliberately induced by a malicious user. In both cases, normal
>> application logging may be bypassed resulting in no indication that
>> anything out of the ordinary has occurred.
>>
>> Additionally, and somewhat unique to Python, it is very easy to affect
>> the code that is run in an application by manipulating either the
>> import system's search path or placing files earlier on the path than
>> intended. This is often seen when developers create a script with the
>> same name as the module they intend to use - for example, a
>> ``random.py`` file that attempts to import the standard library
>> ``random`` module.
>>
>> This is not sandboxing, as this proposal does not attempt to prevent
>> malicious behavior (though it enables some new options to do so).
>> See the `Why Not A Sandbox`_ section below for further discussion.
>>
>> Overview of Changes
>> ===================
>>
>> The aim of these changes is to enable both application developers and
>> system administrators to integrate Python into their existing
>> monitoring systems without dictating how those systems look or behave.
>>
>> We propose two API changes to enable this: an Audit Hook?and Verified
>> Open Hook. Both are available from Python and native code, allowing
>> applications and frameworks written in pure Python code to take
>> advantage of the extra messages, while also allowing embedders or
>> system administrators to deploy builds of Python where auditing is
>> always enabled.
>>
>> Only CPython is bound to provide the native APIs as described here.
>> Other implementations should provide the pure Python APIs, and
>> may provide native versions as appropriate for their underlying
>> runtimes. Auditing events are likewise considered implementation
>> specific, but are bound by normal feature compatibility guarantees.
>>
>> Audit Hook
>> ----------
>>
>> In order to observe actions taken by the runtime (on behalf of the
>> caller), an API is required to raise messages from within certain
>> operations. These operations are typically deep within the Python
>> runtime or standard library, such as dynamic code compilation, module
>> imports, DNS resolution, or use of certain modules such as ``ctypes``.
>>
>> The following new C APIs allow embedders and CPython implementors to
>> send and receive audit hook messages::
>>
>> ?? # Add an auditing hook
>> ?? typedef int (*hook_func)(const char *event, PyObject *args,
>> ??????????????????????????? void *userData);
>> ?? int PySys_AddAuditHook(hook_func hook, void *userData);
>>
>> ?? # Raise an event with all auditing hooks
>> ?? int PySys_Audit(const char *event, PyObject *args);
>>
>> ?? # Internal API used during Py_Finalize() - not publicly accessible
>> ?? void _Py_ClearAuditHooks(void);
>>
>> The new Python APIs for receiving and raising audit hooks are::
>>
>> ?? # Add an auditing hook
>> ?? sys.addaudithook(hook: Callable[[str, tuple]])
>>
>> ?? # Raise an event with all auditing hooks
>> ?? sys.audit(str, *args)
>>
>>
>> Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time,
>> including before ``Py_Initialize()``, or by calling
>> ``sys.addaudithook()`` from Python code. Hooks cannot be removed or
>> replaced.
>>
>> When events of interest are occurring, code can either call
>> ``PySys_Audit()`` from C (while the GIL is held) or ``sys.audit()``. The
>> string argument is the name of the event, and the tuple contains
>> arguments. A given event name should have a fixed schema for arguments,
>> which should be considered a public API (for each x.y version release),
>> and thus should only change between feature releases with updated
>> documentation.
>>
>> For maximum compatibility, events using the same name as an event in
>> the reference interpreter CPython should make every attempt to use
>> compatible arguments. Including the name or an abbreviation of the
>> implementation in implementation-specific event names will also help
>> prevent collisions. For example, a ``pypy.jit_invoked`` event is clearly
>> distinguised from an ``ipy.jit_invoked`` event.
>>
>> When an event is audited, each hook is called in the order it was added
>> with the event name and tuple. If any hook returns with an exception
>> set, later hooks are ignored and *in general* the Python runtime should
>> terminate. This is intentional to allow hook implementations to decide
>> how to respond to any particular event. The typical responses will be to
>> log the event, abort the operation with an exception, or to immediately
>> terminate the process with an operating system exit call.
>>
>> When an event is audited but no hooks have been set, the ``audit()``
>> function should impose minimal overhead. Ideally, each argument is a
>> reference to existing data?rather than a value calculated just for the
>> auditing call.
>>
>> As hooks may be Python objects, they need to be freed during
>> ``Py_Finalize()``. To do this, we add an internal API
>> ``_Py_ClearAuditHooks()`` that releases any Python hooks and any
>> memory held. This is an internal function with no public export, and
>> we recommend it raise its own audit event for all current hooks to
>> ensure that unexpected calls are observed.
>>
>> Below in `Suggested Audit Hook Locations`_, we recommend some important
>> operations that should raise audit events.
>>
>> Python implementations should document which operations will raise
>> audit events, along with the event schema. It is intentional that
>> ``sys.addaudithook(print)`` be a trivial way to display all messages.
>>
>> Verified Open Hook
>> ------------------
>>
>> Most operating systems have a mechanism to distinguish between files
>> that can be executed and those that can not. For example, this may be an
>> execute bit in the permissions field, a verified hash of the file
>> contents to detect potential code tampering, or file system path
>> restrictions. These are an important security mechanism for preventing
>> execution of data or code that is not approved for a given environment.
>> Currently, Python has no way to integrate with these when launching
>> scripts or importing modules.
>>
>> The new public C API for the verified open hook is::
>>
>> ?? # Set the handler
>> ?? typedef PyObject *(*hook_func)(PyObject *path, void *userData)
>> ?? int PyImport_SetOpenForImportHook(hook_func handler, void *userData)
>>
>> ?? # Open a file using the handler
>> ?? PyObject *PyImport_OpenForImport(const char *path)
>>
>> The new public Python API for the verified open hook is::
>>
>> ?? # Open a file using the handler
>> ?? importlib.util.open_for_import(path : str) -> io.IOBase
>>
>>
>> The ``importlib.util.open_for_import()`` function is a drop-in
>> replacement?for ``open(str(pathlike), 'rb')``. Its default behaviour is
>> to open a file for raw, binary access. To change the behaviour a new
>> handler should be set. Handler functions only accept ``str`` arguments.
>> The C API ``PyImport_OpenForImport`` function assumes UTF-8 encoding.
>>
>> A custom handler may be set by calling 
>> ``PyImport_SetOpenForImportHook()``
>> from C at any time, including before ``Py_Initialize()``. However, if a
>> hook has already been set then the call will fail. When
>> ``open_for_import()`` is called with a hook set, the hook will be passed
>> the path and its return value will be returned directly. The returned
>> object should be an open file-like object that supports reading raw
>> bytes. This is explicitly intended to allow a ``BytesIO`` instance if
>> the open handler has already read the entire file into memory.
>>
>> Note that these hooks can import and call the ``_io.open()`` function on
>> CPython without triggering themselves. They can also use ``_io.BytesIO``
>> to return a compatible result using an in-memory buffer.
>>
>> If the hook determines that the file should not be loaded, it should
>> raise an exception of its choice, as well as performing any other
>> logging.
>>
>> All import and execution functionality involving code from a file will
>> be changed to use ``open_for_import()`` unconditionally. It is important
>> to note that calls to ``compile()``, ``exec()`` and ``eval()`` do not go
>> through this function - an audit hook that includes the code from these
>> calls is the best opportunity to validate code that is read from the
>> file. Given the current decoupling between import and execution in
>> Python, most imported code will go through both ``open_for_import()``
>> and the log hook for ``compile``, and so care should be taken to avoid
>> repeating verification steps.
>>
>> There is no Python API provided for changing the open hook. To modify
>> import behavior from Python code, use the existing functionality
>> provided by ``importlib``.
>>
>> API Availability
>> ----------------
>>
>> While all the functions added here are considered public and stable API,
>> the behavior of the functions is implementation specific. Most
>> descriptions here refer?to the CPython?implementation, and while other
>> implementations should provide the functions, there is no requirement
>> that they behave the same.
>>
>> For example, ``sys.addaudithook()`` and ``sys.audit()`` should exist but
>> may do nothing. This allows code to make calls to ``sys.audit()``
>> without having to test for existence, but it should not assume that its
>> call will have any effect. (Including existence tests in
>> security-critical code allows another vector to bypass auditing, so it
>> is preferable that the function always exist.)
>>
>> ``importlib.util.open_for_import(path)`` should at a minimum always
>> return ``_io.open(path, 'rb')``. Code using the function should make no
>> further assumptions about what may occur, and implementations other than
>> CPython are not required to let developers override the behavior of this
>> function with a hook.
>>
>> Suggested Audit Hook Locations
>> ==============================
>>
>> The locations and parameters in calls to ``sys.audit()`` or
>> ``PySys_Audit()`` are to be determined by individual Python
>> implementations. This is to allow maximum freedom for implementations
>> to expose the operations that are most relevant to their platform,
>> and to avoid or ignore potentially expensive or noisy events.
>>
>> Table 1 acts as both suggestions of operations that should trigger
>> audit events on all implementations, and examples of event schemas.
>>
>> Table 2 provides further examples that are not required, but are
>> likely to be available in CPython.
>>
>> Refer to the documentation associated with your version of Python to
>> see which operations provide audit events.
>>
>> .. csv-table:: Table 1: Suggested Audit Hooks
>> ?? :header: "API Function", "Event Name", "Arguments", "Rationale"
>> ?? :widths: 2, 2, 3, 6
>>
>> ?? ``PySys_AddAuditHook``, ``sys.addaudithook``, "", "Detect when new
>> ?? audit hooks are being added.
>> ?? "
>> ?? ``PyImport_SetOpenForImportHook``, ``setopenforimporthook``, "", "
>> ?? Detects any attempt to set the ``open_for_import`` hook.
>> ?? "
>> ?? "``compile``, ``exec``, ``eval``, ``PyAst_CompileString``,
>> ?? ``PyAST_obj2mod``", ``compile``, "``(code, filename_or_none)``", "
>> ?? Detect dynamic code compilation, where ``code`` could be a string or
>> ?? AST. Note that this will be called for regular imports of source
>> ?? code, including those that were opened with ``open_for_import``.
>> ?? "
>> ?? "``exec``, ``eval``, ``run_mod``", ``exec``, "``(code_object,)``", "
>> ?? Detect dynamic execution of code objects. This only occurs for
>> ?? explicit calls, and is not raised for normal function invocation.
>> ?? "
>> ?? ``import``, ``import``, "``(module, filename, sys.path,
>> ?? sys.meta_path, sys.path_hooks)``", "Detect when modules are
>> ?? imported. This is raised before the module name is resolved to a
>> ?? file. All arguments other than the module name may be ``None`` if
>> ?? they are not used or available.
>> ?? "
>> ?? "``open``", ``open``, "``(path, mode, flags)``", "Detect when a file
>> ?? is about to be opened. *path* and *mode* are the usual parameters to
>> ?? ``open`` if available, while *flags* is provided instead of *mode*
>> ?? in some cases.
>> ?? "
>> ?? ``PyEval_SetProfile``, ``sys.setprofile``, "", "Detect when code is
>> ?? injecting trace functions. Because of the implementation, exceptions
>> ?? raised from the hook will abort the operation, but will not be
>> ?? raised in Python code. Note that ``threading.setprofile`` eventually
>> ?? calls this function, so the event will be audited for each thread.
>> ?? "
>> ?? ``PyEval_SetTrace``, ``sys.settrace``, "", "Detect when code is
>> ?? injecting trace functions. Because of the implementation, exceptions
>> ?? raised from the hook will abort the operation, but will not be
>> ?? raised in Python code. Note that ``threading.settrace`` eventually
>> ?? calls this function, so the event will be audited for each thread.
>> ?? "
>> ?? "``_PyObject_GenericSetAttr``, ``check_set_special_type_attr``,
>> ?? ``object_set_class``, ``func_set_code``, ``func_set_[kw]defaults``","
>> ?? ``object.__setattr__``","``(object, attr, value)``","Detect monkey
>> ?? patching of types and objects. This event
>> ?? is raised for the ``__class__`` attribute and any attribute on
>> ?? ``type`` objects.
>> ?? "
>> "``_PyObject_GenericSetAttr``",``object.__delattr__``,"``(object,
>> ?? attr)``","Detect deletion of object attributes. This event is raised
>> ?? for any attribute on ``type`` objects.
>> ?? "
>> "``Unpickler.find_class``",``pickle.find_class``,"``(module_name,
>> ?? global_name)``","Detect imports and global name lookup when
>> ?? unpickling.
>> ?? "
>>
>>
>> .. csv-table:: Table 2: Potential CPython Audit Hooks
>> ?? :header: "API Function", "Event Name", "Arguments", "Rationale"
>> ?? :widths: 2, 2, 3, 6
>>
>> ?? ``_PySys_ClearAuditHooks``, ``sys._clearaudithooks``, "", "Notifies
>> ?? hooks they are being cleaned up, mainly in case the event is
>> ?? triggered unexpectedly. This event cannot be aborted.
>> ?? "
>> ?? ``code_new``, ``code.__new__``, "``(bytecode, filename, name)``", "
>> ?? Detect dynamic creation of code objects. This only occurs for
>> ?? direct instantiation, and is not raised for normal compilation.
>> ?? "
>> ?? ``func_new_impl``, ``function.__new__``, "``(code,)``", "Detect
>> ?? dynamic creation of function objects. This only occurs for direct
>> ?? instantiation, and is not raised for normal compilation.
>> ?? "
>> ?? "``_ctypes.dlopen``, ``_ctypes.LoadLibrary``", ``ctypes.dlopen``, "
>> ?? ``(module_or_path,)``", "Detect when native modules are used.
>> ?? "
>> ?? ``_ctypes._FuncPtr``, ``ctypes.dlsym``, "``(lib_object, name)``", "
>> ?? Collect information about specific symbols retrieved from native
>> ?? modules.
>> ?? "
>> ?? ``_ctypes._CData``, ``ctypes.cdata``, "``(ptr_as_int,)``", "Detect
>> ?? when code is accessing arbitrary memory using ``ctypes``.
>> ?? "
>> ?? "``new_mmap_object``",``mmap.__new__``,"``(fileno, map_size, access,
>> ?? offset)``", "Detects creation of mmap objects. On POSIX, access may
>> ?? have been calculated from the ``prot`` and ``flags`` arguments.
>> ?? "
>> ?? ``sys._getframe``, ``sys._getframe``, "``(frame_object,)``", "Detect
>> ?? when code is accessing frames directly.
>> ?? "
>> ?? ``sys._current_frames``, ``sys._current_frames``, "", "Detect when
>> ?? code is accessing frames directly.
>> ?? "
>> ?? "``socket.bind``, ``socket.connect``, ``socket.connect_ex``,
>> ?? ``socket.getaddrinfo``, ``socket.getnameinfo``, ``socket.sendmsg``,
>> ?? ``socket.sendto``", ``socket.address``, "``(address,)``", "Detect
>> ?? access to network resources. The address is unmodified from the
>> ?? original call.
>> ?? "
>> ?? "``member_get``, ``func_get_code``, ``func_get_[kw]defaults``
>> ?? ",``object.__getattr__``,"``(object, attr)``","Detect access to
>> ?? restricted attributes. This event is raised for any built-in
>> ?? members that are marked as restricted, and members that may allow
>> ?? bypassing imports.
>> ?? "
>> ?? "``urllib.urlopen``",``urllib.Request``,"``(url, data, headers,
>> ?? method)``", "Detects URL requests.
>> ?? "
>>
>> Performance Impact
>> ==================
>>
>> The important performance impact is the case where events are being
>> raised but there are no hooks attached. This is the unavoidable case -
>> once a developer has added audit hooks they have explicitly chosen to
>> trade performance for functionality. Performance impact with hooks added
>> are not of interest here, since this is opt-in functionality.
>>
>> Analysis using the Python Performance Benchmark Suite [1]_ shows no
>> significant impact, with the vast majority of benchmarks showing
>> between 1.05x faster to 1.05x slower.
>>
>> In our opinion, the performance impact of the set of auditing points
>> described in this PEP is negligible.
>>
>> Rejected Ideas
>> ==============
>>
>> Separate module for audit hooks
>> -------------------------------
>>
>> The proposal is to add a new module for audit hooks, hypothetically
>> ``audit``. This would separate the API and implementation from the
>> ``sys`` module, and allow naming the C functions ``PyAudit_AddHook`` and
>> ``PyAudit_Audit`` rather than the current variations.
>>
>> Any such module would need to be a built-in module that is guaranteed to
>> always be present. The nature of these hooks is that they must be
>> callable without condition, as any conditional imports or calls provide
>> opportunities to intercept and suppress or modify events.
>>
>> Given it is one of the most core modules, the ``sys`` module is somewhat
>> protected against module shadowing attacks. Replacing ``sys`` with a
>> sufficiently functional module that the application can still run is a
>> much more complicated task than replacing a module with only one
>> function of interest. An attacker that has the ability to shadow the
>> ``sys`` module is already capable of running arbitrary code from files,
>> whereas an ``audit`` module could be replaced with a single line in a
>> ``.pth`` file anywhere on the search path::
>>
>> ??? import sys; sys.modules['audit'] = type('audit', (object,),
>> ??????? {'audit': lambda *a: None, 'addhook': lambda *a: None})
>>
>> Multiple layers of protection already exist for monkey patching attacks
>> against either ``sys`` or ``audit``, but assignments or insertions to
>> ``sys.modules`` are not audited.
>>
>> This idea is rejected because it makes it trivial to suppress all calls
>> to ``audit``.
>>
>> Flag in sys.flags to indicate "audited" mode
>> --------------------------------------------
>>
>> The proposal is to add a value in ``sys.flags`` to indicate when Python
>> is running in a "secure" or "audited" mode. This would allow
>> applications to detect when some features are enabled or when hooks
>> have been added and modify their behaviour appropriately.
>>
>> Currently, we are not aware of any legitimate reasons for a program to
>> behave differently in the presence of audit hooks.
>>
>> Both application-level APIs ``sys.audit`` and
>> ``importlib.util.open_for_import`` are always present and functional,
>> regardless of whether the regular ``python`` entry point or some
>> alternative entry point is used. Callers cannot determine whether any
>> hooks have been added (except by performing side-channel analysis), nor
>> do they need to. The calls should be fast enough that callers do not
>> need to avoid them, and the program is responsible for ensuring that
>> any added hooks are fast enough to not affect application performance.
>>
>> The argument that this is "security by obscurity" is valid, but
>> irrelevant. Security by obscurity is only an issue when there are no
>> other protective mechanisms; obscurity as the first step in avoiding
>> attack is strongly recommended (see `this article
>> <https://danielmiessler.com/study/security-by-obscurity/>`_ for
>> discussion).
>>
>> This idea is rejected because there are no appropriate reasons for an
>> application to change its behaviour based on whether these APIs are in
>> use.
>>
>> Why Not A Sandbox
>> =================
>>
>> Sandboxing CPython has been attempted many times in the past, and each
>> past attempt has failed. Fundamentally, the problem is that certain
>> functionality has to be restricted when executing the sandboxed code,
>> but otherwise needs to be available for normal operation of Python. For
>> example, completely removing the ability to compile strings into
>> bytecode also breaks the ability to import modules from source code, and
>> if it is not completely removed then there are too many ways to get
>> access to that functionality indirectly. There is not yet any feasible
>> way to generically determine whether a given operation is "safe" or not.
>> Further information and references available at [2]_.
>>
>> This proposal does not attempt to restrict functionality, but simply
>> exposes the fact that the functionality is being used. Particularly for
>> intrusion scenarios, detection is significantly more important than
>> early prevention (as early prevention will generally drive attackers to
>> use an alternate, less-detectable, approach). The availability of audit
>> hooks alone does not change the attack surface of Python in any way, but
>> they enable defenders to integrate Python into their environment in ways
>> that are currently not possible.
>>
>> Since audit hooks have the ability to safely prevent an operation
>> occuring, this feature does enable the ability to provide some level of
>> sandboxing. In most cases, however, the intention is to enable logging
>> rather than creating a sandbox.
>>
>> Relationship to PEP 551
>> =======================
>>
>> This API was originally presented as part of
>> `PEP 551 <https://www.python.org/dev/peps/pep-0551/>`_ Security
>> Transparency in the Python Runtime.
>>
>> For simpler review purposes, and due to the broader applicability of
>> these APIs beyond security, the API design is now presented separately.
>>
>> PEP 551 is an informational PEP discussing how to integrate Python into
>> a secure or audited environment.
>>
>> References
>> ==========
>>
>> .. [1] Python Performance Benchmark Suite 
>> `<https://github.com/python/performance>`_
>>
>> .. [2] Python Security model - Sandbox 
>> `<https://python-security.readthedocs.io/security.html#sandbox>`_
>>
>> Copyright
>> =========
>>
>> Copyright (c) 2019 by Microsoft Corporation. This material may be
>> distributed only subject to the terms and conditions set forth in the
>> Open Publication License, v1.0 or later (the latest version is presently
>> available at http://www.opencontent.org/openpub/).


From brett at python.org  Fri Mar 29 15:03:48 2019
From: brett at python.org (Brett Cannon)
Date: Fri, 29 Mar 2019 12:03:48 -0700
Subject: [Python-Dev] Please take your time reading PEPs (was: PEP 578:
 Python Runtime Audit Hooks)
In-Reply-To: <CA+3bQGEodqbHk4A4fPAzRYeDmX0OG51DWN_Rq28aJfHt15UWdw@mail.gmail.com>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <CA+3bQGEodqbHk4A4fPAzRYeDmX0OG51DWN_Rq28aJfHt15UWdw@mail.gmail.com>
Message-ID: <CAP1=2W5+L0P2cEz8M2A1ZDYM_1_=wL-_Qwn0kF41M=-x2XWA9A@mail.gmail.com>

On Thu, Mar 28, 2019 at 5:03 PM Victor Stinner <vstinner at redhat.com> wrote:

> Hi,
>
> I read quickly the PEP
>

I would like to encourage everyone to read PEPs so that they never feel the
need to write those words ever again. ;)

PEPs are never decided in less than 24 hours, so there is no rush to read a
PEP as quickly as possible in order to reply ASAP. We also have so much
volume as it is when discussing PEPs that I think we should be encouraging
people to take the time to be informed by reading thoroughly before
replying so the back-and-forth is minimized and optimized for impactful
discussions (personally, I would love it if we all aimed for one, thorough
response/day when discussing PEPs, but that's just me). Otherwise we end up
with way more time spent in replies to things that would not have been
necessary to ask if we took our time reading. Remember, every email you
send is read by tons of other people and so there's a real time commitment
you're asking of the world every time you hit that Reply button.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190329/6e262f8c/attachment.html>

From christian at python.org  Fri Mar 29 15:18:17 2019
From: christian at python.org (Christian Heimes)
Date: Fri, 29 Mar 2019 20:18:17 +0100
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
Message-ID: <2cb3740e-ebc2-1839-1d2e-73d1b9f0a445@python.org>

On 28/03/2019 23.35, Steve Dower wrote:
> Hi all
> 
> Time is short, but I'm hoping to get PEP 578 (formerly PEP 551) into
> Python 3.8. Here's the current text for review and comment before I
> submit to the Steering Council.
> 
> The formatted text is at https://www.python.org/dev/peps/pep-0578/
> (update just pushed, so give it an hour or so, but it's fundamentally
> the same as what's there)
> 
> No Discourse post, because we don't have a python-dev equivalent there
> yet, so please reply here for this one.
> 
> Implementation is at https://github.com/zooba/cpython/tree/pep-578/ and
> my backport to 3.7 (https://github.com/zooba/cpython/tree/pep-578-3.7/)
> is already getting some real use (though this will not be added to 3.7,
> unless people *really* want it, so the backport is just for reference).
> 
> Cheers,
> Steve
> 
> =====
> 
> PEP: 578
> Title: Python Runtime Audit Hooks
> Version: $Revision$
> Last-Modified: $Date$
> Author: Steve Dower <steve.dower at python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 16-Jun-2018
> Python-Version: 3.8
> Post-History:
> 
> Abstract
> ========
> 
> This PEP describes additions to the Python API and specific behaviors
> for the CPython implementation that make actions taken by the Python
> runtime visible to auditing tools. Visibility into these actions
> provides opportunities for test frameworks, logging frameworks, and
> security tools to monitor and optionally limit actions taken by the
> runtime.
> 
> This PEP proposes adding two APIs to provide insights into a running
> Python application: one for arbitrary events, and another specific to
> the module import system. The APIs are intended to be available in all
> Python implementations, though the specific messages and values used
> are unspecified here to allow implementations the freedom to determine
> how best to provide information to their users. Some examples likely
> to be used in CPython are provided for explanatory purposes.
> 
> See PEP 551 for discussion and recommendations on enhancing the
> security of a Python runtime making use of these auditing APIs.
> 
> Background
> ==========
> 
> Python provides access to a wide range of low-level functionality on
> many common operating systems. While this is incredibly useful for
> "write-once, run-anywhere" scripting, it also makes monitoring of
> software written in Python difficult. Because Python uses native system
> APIs directly, existing monitoring tools either suffer from limited
> context or auditing bypass.
> 
> Limited context occurs when system monitoring can report that an
> action occurred, but cannot explain the sequence of events leading to
> it. For example, network monitoring at the OS level may be able to
> report "listening started on port 5678", but may not be able to
> provide the process ID, command line, parent process, or the local
> state in the program at the point that triggered the action. Firewall
> controls to prevent such an action are similarly limited, typically
> to process names or some global state such as the current user, and
> in any case rarely provide a useful log file correlated with other
> application messages.
> 
> Auditing bypass can occur when the typical system tool used for an
> action would ordinarily report its use, but accessing the APIs via
> Python do not trigger this. For example, invoking "curl" to make HTTP
> requests may be specifically monitored in an audited system, but
> Python's "urlretrieve" function is not.
> 
> Within a long-running Python application, particularly one that
> processes user-provided information such as a web app, there is a risk
> of unexpected behavior. This may be due to bugs in the code, or
> deliberately induced by a malicious user. In both cases, normal
> application logging may be bypassed resulting in no indication that
> anything out of the ordinary has occurred.
> 
> Additionally, and somewhat unique to Python, it is very easy to affect
> the code that is run in an application by manipulating either the
> import system's search path or placing files earlier on the path than
> intended. This is often seen when developers create a script with the
> same name as the module they intend to use - for example, a
> ``random.py`` file that attempts to import the standard library
> ``random`` module.
> 
> This is not sandboxing, as this proposal does not attempt to prevent
> malicious behavior (though it enables some new options to do so).
> See the `Why Not A Sandbox`_ section below for further discussion.
> 
> Overview of Changes
> ===================
> 
> The aim of these changes is to enable both application developers and
> system administrators to integrate Python into their existing
> monitoring systems without dictating how those systems look or behave.
> 
> We propose two API changes to enable this: an Audit Hook?and Verified
> Open Hook. Both are available from Python and native code, allowing
> applications and frameworks written in pure Python code to take
> advantage of the extra messages, while also allowing embedders or
> system administrators to deploy builds of Python where auditing is
> always enabled.
> 
> Only CPython is bound to provide the native APIs as described here.
> Other implementations should provide the pure Python APIs, and
> may provide native versions as appropriate for their underlying
> runtimes. Auditing events are likewise considered implementation
> specific, but are bound by normal feature compatibility guarantees.
> 
> Audit Hook
> ----------
> 
> In order to observe actions taken by the runtime (on behalf of the
> caller), an API is required to raise messages from within certain
> operations. These operations are typically deep within the Python
> runtime or standard library, such as dynamic code compilation, module
> imports, DNS resolution, or use of certain modules such as ``ctypes``.
> 
> The following new C APIs allow embedders and CPython implementors to
> send and receive audit hook messages::
> 
> ?? # Add an auditing hook
> ?? typedef int (*hook_func)(const char *event, PyObject *args,
> ??????????????????????????? void *userData);
> ?? int PySys_AddAuditHook(hook_func hook, void *userData);
> 
> ?? # Raise an event with all auditing hooks
> ?? int PySys_Audit(const char *event, PyObject *args);
> 
> ?? # Internal API used during Py_Finalize() - not publicly accessible
> ?? void _Py_ClearAuditHooks(void);
> 
> The new Python APIs for receiving and raising audit hooks are::
> 
> ?? # Add an auditing hook
> ?? sys.addaudithook(hook: Callable[[str, tuple]])
> 
> ?? # Raise an event with all auditing hooks
> ?? sys.audit(str, *args)
> 
> 
> Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time,
> including before ``Py_Initialize()``, or by calling
> ``sys.addaudithook()`` from Python code. Hooks cannot be removed or
> replaced.
> 
> When events of interest are occurring, code can either call
> ``PySys_Audit()`` from C (while the GIL is held) or ``sys.audit()``. The
> string argument is the name of the event, and the tuple contains
> arguments. A given event name should have a fixed schema for arguments,
> which should be considered a public API (for each x.y version release),
> and thus should only change between feature releases with updated
> documentation.
> 
> For maximum compatibility, events using the same name as an event in
> the reference interpreter CPython should make every attempt to use
> compatible arguments. Including the name or an abbreviation of the
> implementation in implementation-specific event names will also help
> prevent collisions. For example, a ``pypy.jit_invoked`` event is clearly
> distinguised from an ``ipy.jit_invoked`` event.
> 
> When an event is audited, each hook is called in the order it was added
> with the event name and tuple. If any hook returns with an exception
> set, later hooks are ignored and *in general* the Python runtime should
> terminate. This is intentional to allow hook implementations to decide
> how to respond to any particular event. The typical responses will be to
> log the event, abort the operation with an exception, or to immediately
> terminate the process with an operating system exit call.
> 
> When an event is audited but no hooks have been set, the ``audit()``
> function should impose minimal overhead. Ideally, each argument is a
> reference to existing data?rather than a value calculated just for the
> auditing call.
> 
> As hooks may be Python objects, they need to be freed during
> ``Py_Finalize()``. To do this, we add an internal API
> ``_Py_ClearAuditHooks()`` that releases any Python hooks and any
> memory held. This is an internal function with no public export, and
> we recommend it raise its own audit event for all current hooks to
> ensure that unexpected calls are observed.
> 
> Below in `Suggested Audit Hook Locations`_, we recommend some important
> operations that should raise audit events.
> 
> Python implementations should document which operations will raise
> audit events, along with the event schema. It is intentional that
> ``sys.addaudithook(print)`` be a trivial way to display all messages.
> 
> Verified Open Hook
> ------------------
> 
> Most operating systems have a mechanism to distinguish between files
> that can be executed and those that can not. For example, this may be an
> execute bit in the permissions field, a verified hash of the file
> contents to detect potential code tampering, or file system path
> restrictions. These are an important security mechanism for preventing
> execution of data or code that is not approved for a given environment.
> Currently, Python has no way to integrate with these when launching
> scripts or importing modules.
> 
> The new public C API for the verified open hook is::
> 
> ?? # Set the handler
> ?? typedef PyObject *(*hook_func)(PyObject *path, void *userData)
> ?? int PyImport_SetOpenForImportHook(hook_func handler, void *userData)
> 
> ?? # Open a file using the handler
> ?? PyObject *PyImport_OpenForImport(const char *path)
> 
> The new public Python API for the verified open hook is::
> 
> ?? # Open a file using the handler
> ?? importlib.util.open_for_import(path : str) -> io.IOBase
> 
> 
> The ``importlib.util.open_for_import()`` function is a drop-in
> replacement?for ``open(str(pathlike), 'rb')``. Its default behaviour is
> to open a file for raw, binary access. To change the behaviour a new
> handler should be set. Handler functions only accept ``str`` arguments.
> The C API ``PyImport_OpenForImport`` function assumes UTF-8 encoding.

[...]

> All import and execution functionality involving code from a file will
> be changed to use ``open_for_import()`` unconditionally. It is important
> to note that calls to ``compile()``, ``exec()`` and ``eval()`` do not go
> through this function - an audit hook that includes the code from these
> calls is the best opportunity to validate code that is read from the
> file. Given the current decoupling between import and execution in
> Python, most imported code will go through both ``open_for_import()``
> and the log hook for ``compile``, and so care should be taken to avoid
> repeating verification steps.
> 
> There is no Python API provided for changing the open hook. To modify
> import behavior from Python code, use the existing functionality
> provided by ``importlib``.

I think that the import hook needs to be extended. It only works for
simple Python files or pyc files. There are at least two other important
scenarios: zipimport and shared libraries.

For example how does the importhook work in regarding of alternative
importers like zipimport? What does the import hook 'see' for an import
from a zipfile?

Shared libraries are trickier. libc doesn't define a way to dlopen()
from a file descriptor. dlopen() takes a file name, but a file name
leaves the audit hook open to a TOCTOU attack.

Christian

From steve.dower at python.org  Fri Mar 29 16:10:38 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Mar 2019 13:10:38 -0700
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <2cb3740e-ebc2-1839-1d2e-73d1b9f0a445@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <2cb3740e-ebc2-1839-1d2e-73d1b9f0a445@python.org>
Message-ID: <ad05e257-d046-bba8-b070-c24c80a1521e@python.org>

On 29Mar2019 1218, Christian Heimes wrote:
> On 28/03/2019 23.35, Steve Dower wrote:
>> The ``importlib.util.open_for_import()`` function is a drop-in
>> replacement?for ``open(str(pathlike), 'rb')``. Its default behaviour is
>> to open a file for raw, binary access. To change the behaviour a new
>> handler should be set. Handler functions only accept ``str`` arguments.
>> The C API ``PyImport_OpenForImport`` function assumes UTF-8 encoding.
> 
> [...]
> 
>> All import and execution functionality involving code from a file will
>> be changed to use ``open_for_import()`` unconditionally. It is important
>> to note that calls to ``compile()``, ``exec()`` and ``eval()`` do not go
>> through this function - an audit hook that includes the code from these
>> calls is the best opportunity to validate code that is read from the
>> file. Given the current decoupling between import and execution in
>> Python, most imported code will go through both ``open_for_import()``
>> and the log hook for ``compile``, and so care should be taken to avoid
>> repeating verification steps.
>>
>> There is no Python API provided for changing the open hook. To modify
>> import behavior from Python code, use the existing functionality
>> provided by ``importlib``.
> 
> I think that the import hook needs to be extended. It only works for
> simple Python files or pyc files. There are at least two other important
> scenarios: zipimport and shared libraries.
> 
> For example how does the importhook work in regarding of alternative
> importers like zipimport? What does the import hook 'see' for an import
> from a zipfile?

Yes, good point. I think opening the zip file with open_for_import() is 
the right place to do it, as this operation relates to opening the file 
on disk rather than files within it.

> Shared libraries are trickier. libc doesn't define a way to dlopen()
> from a file descriptor. dlopen() takes a file name, but a file name
> leaves the audit hook open to a TOCTOU attack.

For Windows, at least, the operating system can run its own validation 
on native modules (if you're using a feature like DeviceGuard, for 
example), so the hook likely isn't necessary for those purposes. I 
believe some configurations of Linux allow this as well?

But there's likely no better option here than a combination of good ACLs 
and checking by filename, which at least lets you whitelist the files 
you know you want to allow. Similarly for the zip file - if you trust a 
particular file and trust your ACLs, checking by filename is fine. That 
said, specific audit events for "I'm about to open this zip/dlopen this 
file for import" are very easy to add. (The PEP proposes many examples, 
but is not trying to be exhaustive. If accepted, we should feel free to 
add new events as we identify places where they matter.)

Aside: an important aspect of this per-file approach to execution is 
that the idea is generally to *enable* the files you trust, rather than 
disable the files that are bad. So the detection routines are typically 
"does this match a known hash" or "is this in a secure location", which 
for a carefully deployed system are already known values, rather than 
trying to figure out whether a file might do a bad thing. If you can't 
validate the files in your deployment match the ones you thought you 
were deploying, you are so far from needing this that it doesn't even 
matter, but most of the deployments I work with are *at least* this well 
controlled.

Cheers,
Steve

From steve.dower at python.org  Fri Mar 29 19:26:24 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Mar 2019 16:26:24 -0700
Subject: [Python-Dev] Strange umask(?)/st_mode issue
Message-ID: <d46f7ad5-6a52-e43b-884f-59eddbea4bcb@python.org>

Hi

I'm trying to track down the cause of the failed Linux tests on this 
build (it's also been happening on random PR builds for the last day or 
two, but this is the first I've seen it happen on already merged code):

https://dev.azure.com/Python/cpython/_build/results?buildId=40189

One of the failed tests is this one from test_tarfile.py (and I think 
this is representative of the others):

def test_file_mode(self):
     # Test for issue #8464: Create files with correct
     # permissions.
     if os.path.exists(tmpname):
         support.unlink(tmpname)

     original_umask = os.umask(0o022)
     try:
         tar = tarfile.open(tmpname, self.mode)
         tar.close()
         mode = os.stat(tmpname).st_mode & 0o777
         self.assertEqual(mode, 0o644, "wrong file permissions")
     finally:
         os.umask(original_umask)

It's failing because the mode of the new TAR file is 0o666 rather than 
0o644. I'd guess that this means the umask() call is not having effect? 
But honestly this is beyond me - I don't have any idea what would cause 
Python to ignore that, and I don't see anything in the commit history 
that could be at fault.

I'm hoping someone can tell me that it's definitely a platform 
configuration issue or it's definitely interference between two of our 
tests (which run in random order, which would explain the intermittent 
nature of this, but I haven't spotted any tests that seem likely either).

To see the other failed PR builds, the full list is at 
https://dev.azure.com/Python/cpython/_build?definitionId=9 and most of 
the ones from today have failed because of whatever is causing it.

Any help?

Thanks,
Steve

From njs at vorpus.org  Fri Mar 29 20:31:18 2019
From: njs at vorpus.org (Nathaniel Smith)
Date: Fri, 29 Mar 2019 17:31:18 -0700
Subject: [Python-Dev] Strange umask(?)/st_mode issue
In-Reply-To: <d46f7ad5-6a52-e43b-884f-59eddbea4bcb@python.org>
References: <d46f7ad5-6a52-e43b-884f-59eddbea4bcb@python.org>
Message-ID: <CAPJVwBkgQds9zyOQvV6h=kor+73763Ae_F1LOcbC=_eskCLrkA@mail.gmail.com>

That does sound strange. How easily can you reproduce it? That majorly
effects how I would try to debug something like this...

If you're able to get an strace of a failed run then that would probably
tell us a *lot*.

On Fri, Mar 29, 2019, 16:28 Steve Dower <steve.dower at python.org> wrote:

> Hi
>
> I'm trying to track down the cause of the failed Linux tests on this
> build (it's also been happening on random PR builds for the last day or
> two, but this is the first I've seen it happen on already merged code):
>
> https://dev.azure.com/Python/cpython/_build/results?buildId=40189
>
> One of the failed tests is this one from test_tarfile.py (and I think
> this is representative of the others):
>
> def test_file_mode(self):
>      # Test for issue #8464: Create files with correct
>      # permissions.
>      if os.path.exists(tmpname):
>          support.unlink(tmpname)
>
>      original_umask = os.umask(0o022)
>      try:
>          tar = tarfile.open(tmpname, self.mode)
>          tar.close()
>          mode = os.stat(tmpname).st_mode & 0o777
>          self.assertEqual(mode, 0o644, "wrong file permissions")
>      finally:
>          os.umask(original_umask)
>
> It's failing because the mode of the new TAR file is 0o666 rather than
> 0o644. I'd guess that this means the umask() call is not having effect?
> But honestly this is beyond me - I don't have any idea what would cause
> Python to ignore that, and I don't see anything in the commit history
> that could be at fault.
>
> I'm hoping someone can tell me that it's definitely a platform
> configuration issue or it's definitely interference between two of our
> tests (which run in random order, which would explain the intermittent
> nature of this, but I haven't spotted any tests that seem likely either).
>
> To see the other failed PR builds, the full list is at
> https://dev.azure.com/Python/cpython/_build?definitionId=9 and most of
> the ones from today have failed because of whatever is causing it.
>
> Any help?
>
> Thanks,
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190329/3ad6d3a5/attachment.html>

From vstinner at redhat.com  Fri Mar 29 21:30:21 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Sat, 30 Mar 2019 02:30:21 +0100
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <119e26f0-d826-7e13-aa4a-e0a67bea3265@python.org>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <70c245c7-8f40-a51b-934b-af958f7cd849@python.org>
 <CAP1=2W6+udqgYbXCvGQAg4tbuKdU8LfrDeCrrvwMoxtZZ7UN1g@mail.gmail.com>
 <CA+3bQGG3BAp_4pEqkY9JiJOEStkX4KA=cdaYbTv30FospKC-7w@mail.gmail.com>
 <119e26f0-d826-7e13-aa4a-e0a67bea3265@python.org>
Message-ID: <CA+3bQGE+vLcsWpkQvSu_ThNvq3DYAR0pL=Z92FFLAUb+fVm2JQ@mail.gmail.com>

The purpose of the PEP 587 is to have a working document so everyone
can look at the proposed API (stay focused to the API rather than
bothering with the implementation). IMHO it's now time to get more
people looking at the Python Initialization.

> But there are enough of us
> with fuzzy but valid ideas in our heads that we really need that
> brainstorming session to mix them together and find something feasible.
> Maybe we're best to put it off until PyCon at this point.

Python 3.8 feature freeze is scheduled at the end of May, less than
one month after the PyCon. It might be a little bit too late, no?

Would you mind to elaborate these ideas?

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.

From steve.dower at python.org  Fri Mar 29 22:30:37 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Mar 2019 19:30:37 -0700
Subject: [Python-Dev] Strange umask(?)/st_mode issue
In-Reply-To: <CAPJVwBkgQds9zyOQvV6h=kor+73763Ae_F1LOcbC=_eskCLrkA@mail.gmail.com>
References: <d46f7ad5-6a52-e43b-884f-59eddbea4bcb@python.org>
 <CAPJVwBkgQds9zyOQvV6h=kor+73763Ae_F1LOcbC=_eskCLrkA@mail.gmail.com>
Message-ID: <13f98061-6f64-2e8b-de66-d84a7be00a17@python.org>

On 29Mar.2019 1731, Nathaniel Smith wrote:
> That does sound strange. 

Oh good, it's not just me :)

> How easily can you reproduce it? That majorly
> effects how I would try to debug something like this...

No idea. Looking at the builds on Pipelines again, it seems that all the
builds since about 10am PDT have been failing, so perhaps they rolled
out something that changed how umask works? I don't have any idea
whether that's possible though.

> If you're able to get an strace of a failed run then that would probably
> tell us a *lot*.

I'll need help with that :) But if it's just commands in the
.azure-pipelines/posix-steps.yml file then sending a PR with them added
should do it.

Though given there seems to be a clear start time and no relevant
change, I'm inclined to report it to the Pipelines team (which I already
have) and see what they say. Normally my internal CPython builds catch
issues in the hosted VMs before they reach public accounts, but this may
have been a hotfix for some other issue.

Cheers,
Steve

From cs at cskk.id.au  Fri Mar 29 22:39:47 2019
From: cs at cskk.id.au (Cameron Simpson)
Date: Sat, 30 Mar 2019 13:39:47 +1100
Subject: [Python-Dev] Strange umask(?)/st_mode issue
In-Reply-To: <13f98061-6f64-2e8b-de66-d84a7be00a17@python.org>
References: <13f98061-6f64-2e8b-de66-d84a7be00a17@python.org>
Message-ID: <20190330023947.GA62291@cskk.homeip.net>

On 29Mar2019 19:30, Steve Dower <steve.dower at python.org> wrote:
>On 29Mar.2019 1731, Nathaniel Smith wrote:
>> That does sound strange.
>
>Oh good, it's not just me :)
>
>> How easily can you reproduce it? That majorly
>> effects how I would try to debug something like this...
>
>No idea. Looking at the builds on Pipelines again, it seems that all the
>builds since about 10am PDT have been failing, so perhaps they rolled
>out something that changed how umask works? I don't have any idea
>whether that's possible though.
>
>> If you're able to get an strace of a failed run then that would probably
>> tell us a *lot*.
>
>I'll need help with that :) But if it's just commands in the
>.azure-pipelines/posix-steps.yml file then sending a PR with them added
>should do it.

Can you get a branch into your pipeline? Then you could just hack the 
tarfile test with something quick and dirty like:

    pid = os.getpid()
    system("strace -p %d 2>/path/to/strace.out &" % pid)
    time.sleep(2)   # get strace heaps of time to start

just before the tarfile open. A ghastly hack but it would get you 
debugging info. You could even decide to remove the strace.out file if 
the umask issue doesn't show, if it is erratic (can't see why it would 
be though).

Cheers,
Cameron Simpson <cs at cskk.id.au>

From steve.dower at python.org  Fri Mar 29 22:42:10 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Mar 2019 19:42:10 -0700
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <CA+3bQGE+vLcsWpkQvSu_ThNvq3DYAR0pL=Z92FFLAUb+fVm2JQ@mail.gmail.com>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <70c245c7-8f40-a51b-934b-af958f7cd849@python.org>
 <CAP1=2W6+udqgYbXCvGQAg4tbuKdU8LfrDeCrrvwMoxtZZ7UN1g@mail.gmail.com>
 <CA+3bQGG3BAp_4pEqkY9JiJOEStkX4KA=cdaYbTv30FospKC-7w@mail.gmail.com>
 <119e26f0-d826-7e13-aa4a-e0a67bea3265@python.org>
 <CA+3bQGE+vLcsWpkQvSu_ThNvq3DYAR0pL=Z92FFLAUb+fVm2JQ@mail.gmail.com>
Message-ID: <91eec784-28ab-024f-42a6-8a1e5d37d9bf@python.org>

On 29Mar.2019 1830, Victor Stinner wrote:
> The purpose of the PEP 587 is to have a working document so everyone
> can look at the proposed API (stay focused to the API rather than
> bothering with the implementation). IMHO it's now time to get more
> people looking at the Python Initialization.
> 
>> But there are enough of us
>> with fuzzy but valid ideas in our heads that we really need that
>> brainstorming session to mix them together and find something feasible.
>> Maybe we're best to put it off until PyCon at this point.
> 
> Python 3.8 feature freeze is scheduled at the end of May, less than
> one month after the PyCon. It might be a little bit too late, no?

I don't think we want to rush this in for 3.8 at this point anyway. The
design of how Python is embedded is one of those things that could
drastically affect the scenarios it gets used for in the future
(probably half of my tasks at work right now involve embedding CPython),
so I'd like to get it right.

> Would you mind to elaborate these ideas?

I'd love to, but I don't have them all straight right now, and one of
the problems with putting them in writing is I don't get immediate
feedback when I'm not being clear enough or if there is missing context.
I know you personally have seen most of my ideas, because I keep pinging
you on them ;)

My big one is what I posted on capi-sig about being able to classify our
APIs better and define scenarios where they are ready for use, as well
as breaking up unnecessary dependencies so that embedders have more
flexibility (the rings and layers post). I posted a few examples of how
initialization "could" be on various bugs I've had to deal with relating
to it, and obviously I've been pushing the embeddable distro for Windows
for a while (which is surprisingly popular with a very specific subset
of users), as well as using it myself, so there are things that just
annoy me enough about what we currently have.

But I really do think this should start as a high bandwidth, in-person
brainstorm session to get through the first few big scenarios. Then
it'll be easy to open those up to review and let anyone submit their
needs for hosting Python. And once we've collated a good set of "needs"
we'll have a chance of designing the configuration and initialization
APIs that will satisfy most/all of them. Maybe in time for 3.9 (or 3.10,
if our RM gets the accelerated release cycle he wants ;) ).

I personally think being able to embed Python easily and safely in other
applications will be a powerful feature that will allow many
non-developers to write code to get their work done, as we already see
with Jupyter (and family). More are coming, but the responsibility is on
us to make it successful. I want to get it right.

Cheers,
Steve

From steve.dower at python.org  Fri Mar 29 22:44:00 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Mar 2019 19:44:00 -0700
Subject: [Python-Dev] Strange umask(?)/st_mode issue
In-Reply-To: <20190330023947.GA62291@cskk.homeip.net>
References: <13f98061-6f64-2e8b-de66-d84a7be00a17@python.org>
 <20190330023947.GA62291@cskk.homeip.net>
Message-ID: <55e6f051-179d-73b4-4cca-b91c5c81b498@python.org>

On 29Mar.2019 1939, Cameron Simpson wrote:
> Can you get a branch into your pipeline? Then you could just hack the
> tarfile test with something quick and dirty like:
> 
> ?? pid = os.getpid()
> ?? system("strace -p %d 2>/path/to/strace.out &" % pid)
> ?? time.sleep(2)?? # get strace heaps of time to start
> 
> just before the tarfile open. A ghastly hack but it would get you
> debugging info. You could even decide to remove the strace.out file if
> the umask issue doesn't show, if it is erratic (can't see why it would
> be though).

Perfect, I'll give this a go. Thanks!

From songofacandy at gmail.com  Fri Mar 29 23:20:07 2019
From: songofacandy at gmail.com (Inada Naoki)
Date: Sat, 30 Mar 2019 12:20:07 +0900
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
Message-ID: <CAEfz+Tyuwzc6PfTDkCFUcrU6GkZH4uuFf6hPN1EqFoxiv1JA5A@mail.gmail.com>

I don't like adding more Python callback from low level.

Python runtime is very complicated already, especially __del__,
shutdown process, and multi threading.  Python callback from low level
is source of very difficult bugs always.

Additionally, if we used the PEP for logging complex application,
the log will be unreliable.  For example:

1. Want to open file A in C code, call callback.
2. In the callback, "A is opened" is logged.
3. In the same callback, import may be happened and logged.
4. In the same callback, other thread may be run and any thing can be logged.
5. Many many other things happens and callback is called.
6. Then, open the file A.

In this example, logged event ordering and timing is very different from
real event ordering and timing.

I prefer low level tool to trace low level thing, although it lacks some
application context.  Maybe, DTrace will become more important tool.

https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902

Regards,
-- 
Inada Naoki  <songofacandy at gmail.com>

From ncoghlan at gmail.com  Sat Mar 30 06:28:44 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 30 Mar 2019 20:28:44 +1000
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
 <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
 <CAEfz+Tzj5uh3K9yOF-k+1S1_s_32CSz2_Ad4kB6WH+Y7m_c1BA@mail.gmail.com>
 <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>
Message-ID: <CADiSq7e64Ju1HA-nr8g_0vG7ChGHdhvUVsNpnyO0iYE1ECtS3w@mail.gmail.com>

On Wed, 27 Mar 2019 at 20:29, Inada Naoki <songofacandy at gmail.com> wrote:
>
> On Mon, Mar 25, 2019 at 10:11 PM Inada Naoki <songofacandy at gmail.com> wrote:
> >
> > C, Rust, Java, Ruby, PHP, don't have PendingDeprecation.
> > Programmers only need Deprecation.  Why programmers need PendingDeprecation
> > only in Python?
> >
>
> Any comments about this?
>
> I want to document PendingDeprecationWarning is deprecated [1].
> May I merge the PR?  Should I start poll on discuss.python.org?

No, just leave it alone, as removing it is pointless churn for no good
reason. If folks don't want to use it, they don't have to use it.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 30 06:31:25 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 30 Mar 2019 20:31:25 +1000
Subject: [Python-Dev] Removing PendingDeprecationWarning
In-Reply-To: <CAEfz+TytJj=oMXKUE5T8dJj0Jq6ZRp6Uex-JSwmpSaOt39Mx2Q@mail.gmail.com>
References: <CAEfz+Tzht97YUsYXFjErc4uKP3J89Yx2ycqhbfk=OkY0w37Yog@mail.gmail.com>
 <CADiSq7d-01UmD9tDg-VEotMGpUSpXgbH8TuuDQFCKWi7jC-39Q@mail.gmail.com>
 <CAEfz+Tzak3XDzSqV=JwWOZN4MkEK8RCTh7rQuNZijVbZ3pE_Aw@mail.gmail.com>
 <CADiSq7eagL+3ZSPvveUocF9-3OB5B_ryyH0DNMTZD72sOJoG7w@mail.gmail.com>
 <CAEfz+Tzj5uh3K9yOF-k+1S1_s_32CSz2_Ad4kB6WH+Y7m_c1BA@mail.gmail.com>
 <CAEfz+TxwTgN=cu0t1OoKqNdJTSrNpvYkVvqUoTHOsE4RgQA9OA@mail.gmail.com>
 <18ccdacf-8fc8-5130-b4ba-89df84e02987@python.org>
 <CAEfz+TytJj=oMXKUE5T8dJj0Jq6ZRp6Uex-JSwmpSaOt39Mx2Q@mail.gmail.com>
Message-ID: <CADiSq7etd9fs2OmmoWMYzuzLxe793PpmCd=8Mrv+tdmFjwdZ=Q@mail.gmail.com>

On Thu, 28 Mar 2019 at 12:31, Inada Naoki <songofacandy at gmail.com> wrote:
> I didn't know "right way" to chose DeprecationWarning or
> PendingDeprecationWarning.

That's just a documentation fix: "If you're not sure whether to use
DeprecationWarning or PendingDeprecationWarning, use
DeprecationWarning".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 30 07:07:34 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 30 Mar 2019 21:07:34 +1000
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <91eec784-28ab-024f-42a6-8a1e5d37d9bf@python.org>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <70c245c7-8f40-a51b-934b-af958f7cd849@python.org>
 <CAP1=2W6+udqgYbXCvGQAg4tbuKdU8LfrDeCrrvwMoxtZZ7UN1g@mail.gmail.com>
 <CA+3bQGG3BAp_4pEqkY9JiJOEStkX4KA=cdaYbTv30FospKC-7w@mail.gmail.com>
 <119e26f0-d826-7e13-aa4a-e0a67bea3265@python.org>
 <CA+3bQGE+vLcsWpkQvSu_ThNvq3DYAR0pL=Z92FFLAUb+fVm2JQ@mail.gmail.com>
 <91eec784-28ab-024f-42a6-8a1e5d37d9bf@python.org>
Message-ID: <CADiSq7eF7hPV3SMjYSmKT3o5WgNb=MX-qDFw9KOzx12XXt70rg@mail.gmail.com>

On Sat, 30 Mar 2019 at 12:45, Steve Dower <steve.dower at python.org> wrote:
>
> On 29Mar.2019 1830, Victor Stinner wrote:
> > The purpose of the PEP 587 is to have a working document so everyone
> > can look at the proposed API (stay focused to the API rather than
> > bothering with the implementation). IMHO it's now time to get more
> > people looking at the Python Initialization.
> >
> >> But there are enough of us
> >> with fuzzy but valid ideas in our heads that we really need that
> >> brainstorming session to mix them together and find something feasible.
> >> Maybe we're best to put it off until PyCon at this point.
> >
> > Python 3.8 feature freeze is scheduled at the end of May, less than
> > one month after the PyCon. It might be a little bit too late, no?
>
> I don't think we want to rush this in for 3.8 at this point anyway. The
> design of how Python is embedded is one of those things that could
> drastically affect the scenarios it gets used for in the future
> (probably half of my tasks at work right now involve embedding CPython),
> so I'd like to get it right.

Victor and I chatted about this, and I think it would be good to get
something in to Python 3.8 that gives applications embedding CPython
access to the same level of control that we have from the native
CPython CLI application - the long and the short of the *nix embedding
bug reports that have come in since Python 3.7 was released is that
locale coercion and UTF-8 mode don't quite work properly when an
application is relying solely on the Py_Initialize() and Py_Main()
APIs and doesn't have access to the extra preconfiguration steps that
have been added to get everything to work nicely together and avoid
mojibake in the native CPython CLI app. Victor's gone to great lengths
to try to make them work, but the unfortunate fact is that by the time
they're called, too many other things have often happened in the
embedding application for CPython to be able to get all the data
completely self-consistent.

Thus the two changes go hand in hand: reverting the old initialization
APIs back to their Python 3.6 behaviour to fix the embedding
regressions our two PEPs inadvertently introduced for some
applications when running in the POSIX locale, while also exposing new
initialization APIs so embedding apps can explicitly opt in to
behaving the same way as the CPython CLI does. Affected apps would
then switch to Python 3.8 at the earliest opportunity, and stop
supporting Python 3.7 as the embedded Python version.

The absolute bare minimum version of PEP 587 that we need for that
purpose is to expose the PreInitialize API, as that's the one that
allows the active text encoding to be set early enough to avoid
mojibake: https://www.python.org/dev/peps/pep-0587/#pre-initialization-with-pypreconfig

The rest of the proposal in PEP 587 then comes from wanting to publish
an API that matches the one we're now using ourselves, rather than
coming up with something more speculative.

However, I was also going to suggest that you (Steve) might make a
good BDFL-Delegate for these PEPs - there aren't that many of us
familiar with this part of the code base, and Victor, Eric, and I are
all way too close to the concrete API design to judge it objectively,
while you not only maintain the embeddable CPython bundle for Windows,
you also have access to users of that bundle that might be able to
provide you with additional feedback :)

Cheers,
Nick.

P.S. I've also posted a draft update to PEP 432 that modifies it to
reflect Victor's extraction of the part we already have as a private
API to PEP 587:https://github.com/python/peps/pull/965

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 30 10:47:32 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 31 Mar 2019 00:47:32 +1000
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <ad05e257-d046-bba8-b070-c24c80a1521e@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <2cb3740e-ebc2-1839-1d2e-73d1b9f0a445@python.org>
 <ad05e257-d046-bba8-b070-c24c80a1521e@python.org>
Message-ID: <CADiSq7ftogxBEZMu1suwMhpyBPehPteqU4EJr4YqxprpFYP-Nw@mail.gmail.com>

On Sat, 30 Mar 2019 at 06:14, Steve Dower <steve.dower at python.org> wrote:
> On 29Mar2019 1218, Christian Heimes wrote:
> > On 28/03/2019 23.35, Steve Dower wrote:
> >> There is no Python API provided for changing the open hook. To modify
> >> import behavior from Python code, use the existing functionality
> >> provided by ``importlib``.
> >
> > I think that the import hook needs to be extended. It only works for
> > simple Python files or pyc files. There are at least two other important
> > scenarios: zipimport and shared libraries.
> >
> > For example how does the importhook work in regarding of alternative
> > importers like zipimport? What does the import hook 'see' for an import
> > from a zipfile?
>
> Yes, good point. I think opening the zip file with open_for_import() is
> the right place to do it, as this operation relates to opening the file
> on disk rather than files within it.

I like this PEP in principle, but the specific "open_for_import" name
bothers me a lot, as it implies that "importing" is the only situation
where a file will be opened for code execution. Accordingly, I think
proceeding with that name will increase the risk of unintentional
audit bypasses, as folks (both within the core development team and
outside it) use the regular open function for data that they then pass
to exec(), or for some other purpose that nevertheless allows
arbitrary code execution (e.g. the shelve module loading pickle files
from disk).

If this part of the API were lower down the stack (e.g.
"_io.open_for_code_execution") then I think it would make more sense -
APIs like tokenize.open(), runpy.run_path(), PyRun_SimpleFile(),
shelve, etc, could use that, without having to introduce a dependency
on importlib to get access to the functionality.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 30 11:08:11 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 31 Mar 2019 01:08:11 +1000
Subject: [Python-Dev] Tests for internal functionality
In-Reply-To: <q6jpgk$2uhn$1@blaine.gmane.org>
References: <87f2d4c9-b9e8-6a63-e4a8-e464a0d63be6@mail.mipt.ru>
 <q6jpgk$2uhn$1@blaine.gmane.org>
Message-ID: <CADiSq7cJxEA52yp_HE4baGTUCL5JLUyiLCpLGb-A6A=mxwiDFw@mail.gmail.com>

On Sun, 17 Mar 2019 at 07:27, Terry Reedy <tjreedy at udel.edu> wrote:
>
> On 3/16/2019 3:10 AM, Ivan Pozdeev via Python-Dev wrote:
> > In https://github.com/python/cpython/pull/6541 , I was requested to add
> > tests for an internal C function.
> >
> > As I wrote in
> > https://github.com/python/cpython/pull/6541#issuecomment-445514807 ,
> > it's not clear from the codebase
> >
> > 1) where tests for internal (as opposed to public) functionality should
> > be located
>
> In the same file where they would be if public, which is what you seemed
> to find.  Label somehow, such as 'class InternalsTest(TestCase):' or
> 'test__private' (with double __ indicating test of _private).

A belated follow-up here: in most cases, such tests should be marked
with https://docs.python.org/3/library/test.html#test.support.cpython_only,
so they get skipped automatically when the test suite is run against
another implementation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Mar 30 11:26:49 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 31 Mar 2019 01:26:49 +1000
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <20190325103022.GA39362@cskk.homeip.net>
References: <09eceb89-a207-b65b-2990-18a1ace92a79@udel.edu>
 <20190325103022.GA39362@cskk.homeip.net>
Message-ID: <CADiSq7cCD_xhB58jqd3QL8+RuJ4bL8OjdUKQSBV+cJ_ZGPqp=g@mail.gmail.com>

On Mon, 25 Mar 2019 at 20:34, Cameron Simpson <cs at cskk.id.au> wrote:
> Clearly the above needs to accomodate this, possibly with a fallback
> guess. Is sniffing the end components of __file__ at all sane? ending in
> idlelib/pyshell.py or pyshell.py? Or is that just getting baroque?
>
> I don't think these are strictly the same from some kind of purist
> viewpoint: the path might be anything - _is_ it reasonable to suppose
> that it has a module name (== importable/finding through the import
> path)?

Directly executing files from inside Python packages is explicitly
unsupported, and nigh guaranteed to result in a broken import setup,
as relative imports won't work, and absolute imports will most likely
result in a second copy of the script module getting loaded.

The problem is that __main__ always thinks it is a top-level module
for directly executed scripts - it needs the package structure
information from the "-m" switch to learn otherwise.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve.dower at python.org  Sat Mar 30 12:00:13 2019
From: steve.dower at python.org (Steve Dower)
Date: Sat, 30 Mar 2019 09:00:13 -0700
Subject: [Python-Dev] Strange umask(?)/st_mode issue
In-Reply-To: <55e6f051-179d-73b4-4cca-b91c5c81b498@python.org>
References: <13f98061-6f64-2e8b-de66-d84a7be00a17@python.org>
 <20190330023947.GA62291@cskk.homeip.net>
 <55e6f051-179d-73b4-4cca-b91c5c81b498@python.org>
Message-ID: <3210441d-6094-b53e-6bf7-4b7c4cfb16ea@python.org>

On 29Mar.2019 1944, Steve Dower wrote:
> On 29Mar.2019 1939, Cameron Simpson wrote:
>> Can you get a branch into your pipeline? Then you could just hack the
>> tarfile test with something quick and dirty like:
>>
>> ?? pid = os.getpid()
>> ?? system("strace -p %d 2>/path/to/strace.out &" % pid)
>> ?? time.sleep(2)?? # get strace heaps of time to start
>>
>> just before the tarfile open. A ghastly hack but it would get you
>> debugging info. You could even decide to remove the strace.out file if
>> the umask issue doesn't show, if it is erratic (can't see why it would
>> be though).
> 
> Perfect, I'll give this a go. Thanks!

I set up a PR to collect this trace and the results are at:
https://github.com/python/cpython/pull/12625

However, I suspect it's a non-result:

umask(022)                              = 022
open("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
write(3, "BZh91AY&SYY{\270\344\0\0\24P\0\300\0\4\0\0\10
\0000\314\5)\246"..., 46) = 46
close(3)                                = 0
stat("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
{st_mode=S_IFREG|0666, st_size=46, ...}) = 0

Happy to take more suggestions if anyone has them.

Thanks,
Steve

From steve.dower at python.org  Sat Mar 30 12:05:10 2019
From: steve.dower at python.org (Steve Dower)
Date: Sat, 30 Mar 2019 09:05:10 -0700
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <CAEfz+Tyuwzc6PfTDkCFUcrU6GkZH4uuFf6hPN1EqFoxiv1JA5A@mail.gmail.com>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <CAEfz+Tyuwzc6PfTDkCFUcrU6GkZH4uuFf6hPN1EqFoxiv1JA5A@mail.gmail.com>
Message-ID: <f8db2c1b-08ed-512d-2e84-7564da5436bd@python.org>

On 29Mar.2019 2020, Inada Naoki wrote:
> I don't like adding more Python callback from low level.
> 
> Python runtime is very complicated already, especially __del__,
> shutdown process, and multi threading.  Python callback from low level
> is source of very difficult bugs always.

Asynchronous callbacks, sure. These are all synchronous and always as
part of a Python operation while the GIL is held (I think putting them
immediately after parameter validation is the best place, but am
hesitant to make that a hard rule).

> Additionally, if we used the PEP for logging complex application,
> the log will be unreliable.  

I think "then don't do that" is a valid response here. There are plenty
of ways to use Python features incorrectly and cause problems (I've
diagnosed many of them :) ). This doesn't make that any more or less true.

> 1. Want to open file A in C code, call callback.
> 2. In the callback, "A is opened" is logged.
> 3. In the same callback, import may be happened and logged.
> 4. In the same callback, other thread may be run and any thing can be logged.
> 5. Many many other things happens and callback is called.
> 6. Then, open the file A.
> 
> In this example, logged event ordering and timing is very different from
> real event ordering and timing.

Yep. If you implement a bad hook then you get bad results. But at least
the user can fix that themselves (since they caused it themselves).

I've already used these hooks for a number of useful purposes, and it
isn't difficult to write one that works just fine.

> I prefer low level tool to trace low level thing, although it lacks some
> application context.  Maybe, DTrace will become more important tool.
> 
> https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902

See my other reply about this. DTrace on Windows is a very long way from
being suitable for production environments.

Cheers,
Steve

From steve.dower at python.org  Sat Mar 30 12:13:30 2019
From: steve.dower at python.org (Steve Dower)
Date: Sat, 30 Mar 2019 09:13:30 -0700
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <CADiSq7ftogxBEZMu1suwMhpyBPehPteqU4EJr4YqxprpFYP-Nw@mail.gmail.com>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <2cb3740e-ebc2-1839-1d2e-73d1b9f0a445@python.org>
 <ad05e257-d046-bba8-b070-c24c80a1521e@python.org>
 <CADiSq7ftogxBEZMu1suwMhpyBPehPteqU4EJr4YqxprpFYP-Nw@mail.gmail.com>
Message-ID: <6ded2c50-bf28-1376-7b0c-9cc6839be56b@python.org>

On 30Mar.2019 0747, Nick Coghlan wrote:
> I like this PEP in principle, but the specific "open_for_import" name
> bothers me a lot, as it implies that "importing" is the only situation
> where a file will be opened for code execution. 
> 
> If this part of the API were lower down the stack (e.g.
> "_io.open_for_code_execution") then I think it would make more sense -
> APIs like tokenize.open(), runpy.run_path(), PyRun_SimpleFile(),
> shelve, etc, could use that, without having to introduce a dependency
> on importlib to get access to the functionality.

It was called "open_for_exec" at one point, though I forget exactly why
we changed it. But I have no problem with moving it. Something like this?

PyImport_OpenForImport -> PyIO_OpenForExec
PyImport_SetOpenForImportHook -> PyIO_SetOpenForExecHook
importlib.util.open_for_import -> _io.open_for_exec

Or more in line with Nick's suggestion:

PyImport_OpenForImport -> PyIO_OpenExecutableCode
PyImport_SetOpenForImportHook -> PyIO_SetOpenExecutableCodeHook
importlib.util.open_for_import -> _io.open_executable_code

I dropped "For", but I don't really care that much about the name. I'd
be okay dropping either "executable" or "code" as well - I don't really
have a good sense of which will make people more likely to use this
correctly.

Cheers,
Steve

From mark at hotpy.org  Sat Mar 30 12:30:58 2019
From: mark at hotpy.org (Mark Shannon)
Date: Sat, 30 Mar 2019 16:30:58 +0000
Subject: [Python-Dev] BDFL-Delegate appointments for several PEPs
In-Reply-To: <CA+=+wqBsUKWHRqjhRFShmY77vaxDPTFaafWtOB8dnyXQSthiZw@mail.gmail.com>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
 <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
 <CA+=+wqBsUKWHRqjhRFShmY77vaxDPTFaafWtOB8dnyXQSthiZw@mail.gmail.com>
Message-ID: <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org>

Hi Petr,

On 27/03/2019 1:50 pm, Petr Viktorin wrote:
> On Sun, Mar 24, 2019 at 4:22 PM Mark Shannon <mark at hotpy.org> wrote:
>>
>> Hi Petr,
>>
>> Regarding PEPs 576 and 580.
>> Over the new year, I did a thorough analysis of possible approaches to
>> possible calling conventions for use in the CPython ecosystems and came
>> up with a new PEP.
>> The draft can be found here:
>> https://github.com/markshannon/peps/blob/new-calling-convention/pep-9999.rst
>>
>> I was hoping to profile a branch with the various experimental changes
>> cherry-picked together, but don't seemed to have found the time :(
>>
>> I'd like to have a testable branch, before formally submitting the PEP,
>> but I'd thought you should be aware of the PEP.
>>
>> Cheers,
>> Mark.
> 
> Hello Mark,
> Thank you for letting me know! I wish I knew of this back in January,
> when you committed the first draft. This is unfair to the competing
> PEP, which is ready and was waiting for the new govenance. We have
> lost three months that could be spent pondering the ideas in the
> pre-PEP.

I realize this is less than ideal. I had planned to publish this in 
December, but life intervened. Nothing bad, just too busy.

> Do you think you will find the time to piece things together? Is there
> anything that you already know should be changed?

I've submitted the final PEP and minimal implementation
https://github.com/python/peps/pull/960
https://github.com/python/cpython/compare/master...markshannon:vectorcall-minimal

> 
> Do you have any comments on [Jeroen's comparison]?

It is rather out of date, but two comments.
1. `_PyObject_FastCallKeywords()` is used as an example of a call in 
CPython. It is an internal implementation detail and not a common path.
2. The claim that PEP 580 allows "certain optimizations because other 
code can make assumptions" is flawed. In general, the caller cannot make 
assumptions about the callee or vice-versa. Python is a dynamic language.

> 
> The pre-PEP is simpler then PEP 580, because it solves simpler issues.

The fundamental issue being addressed is the same, and it is this:
Currently third-party C code can either be called quickly or have access 
to the callable object, not both. Both PEPs address this.

> I'll need to confirm that it won't paint us into a corner -- that
> there's a way to address all the issues in PEP 579 in the future.

PEP 579 is mainly a list of supposed flaws with the 
'builtin_function_or_method' class.
The general thrust of PEP 579 seems to be that builtin-functions and 
builtin-methods should be more flexible and extensible than they are. I 
don't agree. If you want different behaviour, then use a different 
object. Don't try an cram all this extra behaviour into a pre-existing 
object.

However, if we assume that we are talking about callables implemented in 
C, in general, then there are 3 key issues covered by PEP 579.

1. Inspection and documentation; it is hard for extensions to have 
docstrings and signatures. Worth addressing, but completely orthogonal 
to PEP 590.
2. Extensibility and performance; extensions should have the power of 
Python functions without suffering slow calls. Allowing the C code 
access to the callable object is a general solution to this problem. 
Both PEP 580 and PEP 590 do this.
3. Exposing the underlying implementation and signature of the C code, 
so that optimisers can avoid unnecessary boxing. This may be worth 
doing, but until we have an adaptive optimiser capable of exploiting 
this information, this is premature. Neither PEP 580 nor PEP 590 
explicit allow or prevent this.

> 
> The pre-PEP claims speedups of 2% in initial experiments, with
> expected overall performance gain of 4% for the standard benchmark
> suite. That's pretty big.

That's because there is a lot of code around calls in CPython, and it 
has grown in a rather haphazard fashion. Victor's work to add the 
"FASTCALL" protocol has helped. PEP 590 seeks to formalise and extend 
that, so that it can be used more consistently and efficiently.

> As far as I can see, PEP 580 claims not much improvement in CPython,
> but rather large improvements for extensions (Mistune with Cython).

Calls to and from extension code are slow because they have to use the 
`tp_call` calling convention (or lose access to the callable object).
With a calling convention that does not have any special cases,
extensions can be as fast as builtin functions. Both PEP 580 and PEP 590 
attempt to do this, but PEP 590 is more efficient.

> 
> The pre-PEP has a complication around offsetting arguments by 1 to
> allow bound methods forward calls cheaply. I fear that this optimizes
> for current usage with its limitations.

It's optimising for the common case, while allowing the less common.
Bound methods and classes need to add one additional argument. Other 
rarer cases, like `partial` may need to allocate memory, but can still 
add or remove any number of arguments.

> PEP 580's cc_parent allows bound methods to have access to the class,
> and through that, the module object where they are defined and the
> corresponding module state. To support this, vector calls would need a
> two-argument offset.

Not true. The first argument in the vector call is the callable itself. 
Through that it, any callable can access its class, its module or any 
other object it wants.

> (That seems to illustrate the main difference between the motivations
> of the two PEPs: one focuses on extensibility; the other on optimizing
> existing use cases.)

I'll reiterate that PEP 590 is more general than PEP 580 and that once 
the callable's code has access to the callable object (as both PEPs 
allow) then anything is possible. You can't can get more extensible than 
that.

> 
> The pre-PEP's "any third-party class implementing the new call
> interface will not be usable as a base class" looks quite limiting.

PEP 580 has the same limitation for the same reasons. The limitation is 
necessary for correctness if an object supports calls via `__call__` and 
through another calling convention.

> 
> 
> 
> [Jeroen's comparison]:
> https://mail.python.org/pipermail/python-dev/2018-July/154238.html
> 


Cheers,
Mark.

From vano at mail.mipt.ru  Sat Mar 30 14:26:30 2019
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Sat, 30 Mar 2019 21:26:30 +0300
Subject: [Python-Dev] Strange umask(?)/st_mode issue
In-Reply-To: <3210441d-6094-b53e-6bf7-4b7c4cfb16ea@python.org>
References: <13f98061-6f64-2e8b-de66-d84a7be00a17@python.org>
 <20190330023947.GA62291@cskk.homeip.net>
 <55e6f051-179d-73b4-4cca-b91c5c81b498@python.org>
 <3210441d-6094-b53e-6bf7-4b7c4cfb16ea@python.org>
Message-ID: <f4a5d491-b8f0-fc25-0beb-fc01dad589ca@mail.mipt.ru>

On 30.03.2019 19:00, Steve Dower wrote:
> On 29Mar.2019 1944, Steve Dower wrote:
>> On 29Mar.2019 1939, Cameron Simpson wrote:
>>> Can you get a branch into your pipeline? Then you could just hack the
>>> tarfile test with something quick and dirty like:
>>>
>>>  ?? pid = os.getpid()
>>>  ?? system("strace -p %d 2>/path/to/strace.out &" % pid)
>>>  ?? time.sleep(2)?? # get strace heaps of time to start
>>>
>>> just before the tarfile open. A ghastly hack but it would get you
>>> debugging info. You could even decide to remove the strace.out file if
>>> the umask issue doesn't show, if it is erratic (can't see why it would
>>> be though).
>> Perfect, I'll give this a go. Thanks!
> I set up a PR to collect this trace and the results are at:
> https://github.com/python/cpython/pull/12625
>
> However, I suspect it's a non-result:
>
> umask(022)                              = 022
> open("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
> O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
> write(3, "BZh91AY&SYY{\270\344\0\0\24P\0\300\0\4\0\0\10
> \0000\314\5)\246"..., 46) = 46
> close(3)                                = 0
> stat("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
> {st_mode=S_IFREG|0666, st_size=46, ...}) = 0
>
> Happy to take more suggestions if anyone has them.

According to https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#use-a-microsoft-hosted-agent 
, MS uses Ubuntu 16.04

http://manpages.ubuntu.com/manpages/xenial/man2/umask.2.html suggests that umask is ignored if the parent directory has a default ACL (a 
newly-created dir inherits those from its parent).

As per https://linuxconfig.org/how-to-manage-acls-on-linux , the following commands should show if acls are enabled on the current FS and if 
any are active on the dir:

DEVICE=$(df <location> | tail -n +2 | awk '{print $1}')
sudo tune2fs -l $DEVICE | grep -w "Default mount options"
mount | grep -w $DEVICE
getfacl <location>

In `getfacl' output for a directory, entries that start with "default:" list the default ACL .

`setfacl -b <location>' removes all ACLs from location.

> Thanks,
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- 
Regards,
Ivan


From greg at krypto.org  Sat Mar 30 14:30:14 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 30 Mar 2019 11:30:14 -0700
Subject: [Python-Dev] Strange umask(?)/st_mode issue
In-Reply-To: <3210441d-6094-b53e-6bf7-4b7c4cfb16ea@python.org>
References: <13f98061-6f64-2e8b-de66-d84a7be00a17@python.org>
 <20190330023947.GA62291@cskk.homeip.net>
 <55e6f051-179d-73b4-4cca-b91c5c81b498@python.org>
 <3210441d-6094-b53e-6bf7-4b7c4cfb16ea@python.org>
Message-ID: <CAGE7PNLeN1bRFMQMuQ6AFrYYwEzcPYVDhy5hXG-FbRju5FDpCA@mail.gmail.com>

I wouldn't expect it to be the case in a CI environment but I believe a
umask can be overridden if the filesystem is mounted and configured with
acls set?  (oh, hah, Ivan just said the same thing)

-gps

On Sat, Mar 30, 2019 at 9:05 AM Steve Dower <steve.dower at python.org> wrote:

> On 29Mar.2019 1944, Steve Dower wrote:
> > On 29Mar.2019 1939, Cameron Simpson wrote:
> >> Can you get a branch into your pipeline? Then you could just hack the
> >> tarfile test with something quick and dirty like:
> >>
> >>    pid = os.getpid()
> >>    system("strace -p %d 2>/path/to/strace.out &" % pid)
> >>    time.sleep(2)   # get strace heaps of time to start
> >>
> >> just before the tarfile open. A ghastly hack but it would get you
> >> debugging info. You could even decide to remove the strace.out file if
> >> the umask issue doesn't show, if it is erratic (can't see why it would
> >> be though).
> >
> > Perfect, I'll give this a go. Thanks!
>
> I set up a PR to collect this trace and the results are at:
> https://github.com/python/cpython/pull/12625
>
> However, I suspect it's a non-result:
>
> umask(022)                              = 022
>
> open("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
> O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
> write(3, "BZh91AY&SYY{\270\344\0\0\24P\0\300\0\4\0\0\10
> \0000\314\5)\246"..., 46) = 46
> close(3)                                = 0
>
> stat("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
> {st_mode=S_IFREG|0666, st_size=46, ...}) = 0
>
> Happy to take more suggestions if anyone has them.
>
> Thanks,
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190330/9c169221/attachment.html>

From kushal at locationd.net  Sat Mar 30 14:00:25 2019
From: kushal at locationd.net (Kushal Kumaran)
Date: Sat, 30 Mar 2019 11:00:25 -0700
Subject: [Python-Dev] Strange umask(?)/st_mode issue
In-Reply-To: <3210441d-6094-b53e-6bf7-4b7c4cfb16ea@python.org> (Steve Dower's
 message of "Sat, 30 Mar 2019 09:00:13 -0700")
References: <13f98061-6f64-2e8b-de66-d84a7be00a17@python.org>
 <20190330023947.GA62291@cskk.homeip.net>
 <55e6f051-179d-73b4-4cca-b91c5c81b498@python.org>
 <3210441d-6094-b53e-6bf7-4b7c4cfb16ea@python.org>
Message-ID: <87va0044zq.fsf@copper.locationd.net>

Steve Dower <steve.dower at python.org> writes:

> On 29Mar.2019 1944, Steve Dower wrote:
>> On 29Mar.2019 1939, Cameron Simpson wrote:
>>> Can you get a branch into your pipeline? Then you could just hack the
>>> tarfile test with something quick and dirty like:
>>>
>>> ?? pid = os.getpid()
>>> ?? system("strace -p %d 2>/path/to/strace.out &" % pid)
>>> ?? time.sleep(2)?? # get strace heaps of time to start
>>>
>>> just before the tarfile open. A ghastly hack but it would get you
>>> debugging info. You could even decide to remove the strace.out file if
>>> the umask issue doesn't show, if it is erratic (can't see why it would
>>> be though).
>> 
>> Perfect, I'll give this a go. Thanks!
>
> I set up a PR to collect this trace and the results are at:
> https://github.com/python/cpython/pull/12625
>
> However, I suspect it's a non-result:
>
> umask(022)                              = 022
> open("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
> O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
> write(3, "BZh91AY&SYY{\270\344\0\0\24P\0\300\0\4\0\0\10
> \0000\314\5)\246"..., 46) = 46
> close(3)                                = 0
> stat("/home/vsts/work/1/s/build/test_python_5154/@test_5154_tmp-tardir/tmp.tar",
> {st_mode=S_IFREG|0666, st_size=46, ...}) = 0
>
> Happy to take more suggestions if anyone has them.
>

The file must exist already.  Try moving the strace capturing to before
the unlink.

-- 
regards,
kushal

From J.Demeyer at UGent.be  Sat Mar 30 18:36:50 2019
From: J.Demeyer at UGent.be (Jeroen Demeyer)
Date: Sat, 30 Mar 2019 23:36:50 +0100
Subject: [Python-Dev] PEP 580/590 discussion
In-Reply-To: <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org>
References: <CADiSq7f7GRJjZmoLAxpN5hovMT5=M73aJUi14HHFDNG=hssV7w@mail.gmail.com>
 <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org>
 <CA+=+wqBsUKWHRqjhRFShmY77vaxDPTFaafWtOB8dnyXQSthiZw@mail.gmail.com>
 <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org>
Message-ID: <5C9FEF82.50207@UGent.be>

On 2019-03-30 17:30, Mark Shannon wrote:
> 2. The claim that PEP 580 allows "certain optimizations because other
> code can make assumptions" is flawed. In general, the caller cannot make
> assumptions about the callee or vice-versa. Python is a dynamic language.

PEP 580 is meant for extension classes, not Python classes. Extension 
classes are not dynamic. When you implement tp_call in a given way, the 
user cannot change it. So if a class implements the C call protocol or 
the vectorcall protocol, callers can make assumptions about what that means.

> PEP 579 is mainly a list of supposed flaws with the
> 'builtin_function_or_method' class.
> The general thrust of PEP 579 seems to be that builtin-functions and
> builtin-methods should be more flexible and extensible than they are. I
> don't agree. If you want different behaviour, then use a different
> object. Don't try an cram all this extra behaviour into a pre-existing
> object.

I think that there is a misunderstanding here. I fully agree with the 
"use a different object" solution. This isn't a new solution: it's 
already possible to implement those different objects (Cython does it). 
It's just that this solution comes at a performance cost and that's what 
we want to avoid.

> I'll reiterate that PEP 590 is more general than PEP 580 and that once
> the callable's code has access to the callable object (as both PEPs
> allow) then anything is possible. You can't can get more extensible than
> that.

I would argue the opposite: PEP 590 defines a fixed protocol that is not 
easy to extend. PEP 580 on the other hand uses a new data structure 
PyCCallDef which could easily be extended in the future (this will 
intentionally never be part of the stable ABI, so we can do that).

I have also argued before that the generality of PEP 590 is a bad thing 
rather than a good thing: by defining a more rigid protocol as in PEP 
580, more optimizations are possible.

> PEP 580 has the same limitation for the same reasons. The limitation is
> necessary for correctness if an object supports calls via `__call__` and
> through another calling convention.

I don't think that this limitation is needed in either PEP. As I 
explained at the top of this email, it can easily be solved by not using 
the protocol for Python classes. What is wrong with my proposal in PEP 
580: https://www.python.org/dev/peps/pep-0580/#inheritance


Jeroen.

From steve.dower at python.org  Sat Mar 30 20:48:56 2019
From: steve.dower at python.org (Steve Dower)
Date: Sat, 30 Mar 2019 17:48:56 -0700
Subject: [Python-Dev] New Python Initialization API
In-Reply-To: <91eec784-28ab-024f-42a6-8a1e5d37d9bf@python.org>
References: <CA+3bQGHabA_eeDbTQ_KGSTgq9UKEdMuTDTbDXSMgs61SPGPmkQ@mail.gmail.com>
 <70c245c7-8f40-a51b-934b-af958f7cd849@python.org>
 <CAP1=2W6+udqgYbXCvGQAg4tbuKdU8LfrDeCrrvwMoxtZZ7UN1g@mail.gmail.com>
 <CA+3bQGG3BAp_4pEqkY9JiJOEStkX4KA=cdaYbTv30FospKC-7w@mail.gmail.com>
 <119e26f0-d826-7e13-aa4a-e0a67bea3265@python.org>
 <CA+3bQGE+vLcsWpkQvSu_ThNvq3DYAR0pL=Z92FFLAUb+fVm2JQ@mail.gmail.com>
 <91eec784-28ab-024f-42a6-8a1e5d37d9bf@python.org>
Message-ID: <6a17e990-7e2a-0544-1e8e-9b16d05e4df6@python.org>

Here is my first review of https://www.python.org/dev/peps/pep-0587/ and
in general I think it's very good. There are some things I'd like to
consider changing before we embed them permanently in a public API, and
as I'm still keen to have the ability to write Python code for
configuration (to replace getpath.c, for example) I have a bias towards
making that work more smoothly with these changes when we get to it.

I think my biggest point (about halfway down) is that I'd rather use
argv/environ/etc. to *initialize* PyConfig and then only use the config
for initializing the runtime. That way it's more transparent for users
and more difficult for us to add options that embedders can't access.

The appendix is excellent, by the way. Very useful detail to have
written down.

Cheers,
Steve


> ``PyWideCharList`` is a list of ``wchar_t*`` strings.

I always forget whether "const" is valid in C99, but if it is, can we
make this a list of const strings?

I also prefer a name like ``PyWideStringList``, since that's what it is
(the other places we use WideChar in the C API refer only to a single
string, as far as I'm aware).

> ``PyInitError`` is a structure to store an error message or an exit code
> for the Python Initialization.

I love this struct! Currently it's private, but I wonder whether it's
worth making it public as PyError (or PyErrorInfo)? We obviously can't
replace all uses of int as a return value throughout the API, but I
think it has uses elsewhere and we may as well protect against having to
rename in the future.

> * ``exitcode`` (``int``): if greater or equal to zero, argument passed to
>  ``exit()``

Windows is likely to need/want negative exit codes, as many system
errors are represented as 0x80000000|(source of error)|(specific code).

> * ``user_err`` (int): if non-zero, the error is caused by the user
>   configuration, otherwise it's an internal Python error.

Maybe we could just encode this as "positive exitcode is user error,
negative is internal error"? I'm pretty sure struct return values are
passed by reference in most C calling conventions, so the size of the
struct isn't a big deal, but without a proper bool type it may look like
this is a second error code (like errno/winerror in a few places).

> ``PyPreConfig`` structure is used to pre-initialize Python:
>
> * Set the memory allocator
> * Configure the LC_CTYPE locale
> * Set the UTF-8 mode

I think we should have the isolated flag in here - oh wait, we do - I
think we should have the isolated/use_environment options listed in this
list :)

> Functions to pre-initialize Python:
>
> * ``PyInitError Py_PreInitialize(const PyPreConfig *config)``
> * ``PyInitError Py_PreInitializeFromArgs( const PyPreConfig *config,
int argc, char **argv)``
> * ``PyInitError Py_PreInitializeFromWideArgs( const PyPreConfig
*config, int argc, wchar_t **argv)``

I hope to one day be able to support multiple runtimes per process - can
we have an opaque PyRuntime object exposed publicly now and passed into
these functions?

(FWIW, I think we're a long way from being able to support multiple
runtimes *simultaneously*, so the initial implementation here would be
to have a PyRuntime_Create() that returns our global one once and then
errors until it's finalised. The migration path is probably to enable
switching of the current runtime via a dedicated function (i.e. one
active at a time, probably with thread local storage), since we have no
"context" parameter for C API functions, and obviously there are still
complexities such as poorly written extension modules that nonetheless
can be dealt with in embedding scenarios by simply not using them. This
doesn't seem like an unrealistic future, *unless* we add a whole lot of
new APIs now that can't allow it :) )

> ``PyPreConfig`` fields:
>
> * ``coerce_c_locale_warn``: if non-zero, emit a warning if the C locale
>   is coerced.
> * ``coerce_c_locale``: if equals to 2, coerce the C locale; if equals to
>   1, read the LC_CTYPE to decide if it should be coerced.

Can we use another value for coerce_c_locale to determine whether to
warn or not? Save a field.

> * ``legacy_windows_fs_encoding`` (Windows only): if non-zero, set the
>   Python filesystem encoding to ``"mbcs"``.
> * ``utf8_mode``: if non-zero, enable the UTF-8 mode

Why not just set the encodings here? The "PreInitializeFromArgs"
functions can override it based on the other variables we have, and
embedders have a more obvious question to answer than "do I want legacy
behaviour in my app".

Obviously we are not ready to import most encodings after pre
initialization, but I think that's okay. Embedders who set something
outside the range of what can be used without importing encodings will
get an error to that effect if we try.

In fact, I'd be totally okay with letting embedders specify their own
function pointer here to do encoding/decoding between Unicode and the OS
preferred encoding. That would let them use any approach they like.
Similarly for otherwise-unprintable messages (On an earlier project when
I was embedding Python into Windows Store apps - back when their API was
more heavily restricted - this would have been very helpful.)

> Example of Python initialization enabling the isolated mode::
>
>     PyConfig config = PyConfig_INIT;
>     config.isolated = 1;

Haven't we already used extenal values by this point that should have
been isolated? I'd rather have the isolation up front. Or better yet,
make isolation the default unless you call one of the "FromArgs"
functions, and then we don't actually need the config setting at all.

> PyConfig fields:

Before I start discussing individual groups of fields, I would really
like to see the following significant change here (because it'll help
keep us honest and prevent us breaking embedders in the future).

Currently you have three functions, that take a PyConfig and optionally
also use the environment/argv to figure out the settings:

> * ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``
> * ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int
argc, char **argv)``
> * ``PyInitError Py_InitializeFromWideArgs(const PyConfig *config, int
argc, wchar_t **argv)``

I would much prefer to see this flipped around, so that there is one
initialize function taking PyConfig, and two functions that will fill
out the PyConfig based on the environment:

(note two of the "const"s are gone)

* ``PyInitError Py_SetConfigFromArgs(PyConfig *config, int argc, char
**argv)``
* ``PyInitError Py_SetConfigFromWideArgs(PyConfig *config, int argc,
wchar_t **argv)``
* ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``

This means that callers who want to behave like Python will request an
equivalent config and then use it. Those callers who want to be *nearly*
like Python can change things in between. And the precedence rules get
simpler because Py_SetConfig* just overwrites anything:

PyConfig config = PyConfig_INIT;
Py_SetConfigFromWideArgs(&config, argc, argv);
/* optionally change any settings */
Py_InitializeFromConfig(&config);

We could even split out PyMainConfig here and have another function to
collect the settings to pass to Py_RunMain() (such as the script or
module name, things to print on exit, etc.).

Our python.c then uses this configuration, so it gets a few lines longer
than at present. But it becomes a more useful example for people who
want a nearly-like-Python version, and also ensures that any new
configuration we support will be available to embedders.

So with that said, here's what I think about the fields:

> * ``argv``: ``sys.argv``
> * ``base_exec_prefix``: ``sys.base_exec_prefix``
> * ``base_prefix``: ``sys.base_prefix``
> * ``exec_prefix``: ``sys.exec_prefix``
> * ``executable``: ``sys.executable``
> * ``prefix``: ``sys.prefix``
> * ``xoptions``: ``sys._xoptions``

I like all of these, as they nicely map to their sys members.

> * ``module_search_path_env``: ``PYTHONPATH`` environment variale value
> * ``module_search_paths``, ``use_module_search_paths``: ``sys.path``

Why not just have "path" to mirror sys.path? If we go to a Py_SetConfig
approach then all the logic for inferring the path is in there, and if
we don't then the FromArgs() function will either totally override or
ignore it. Either way, no embedder needs to set these individually.

> * ``home``: Python home
> * ``program_name``: Program name
> * ``program``: ``argv[0]`` or an empty string
> * ``user_site_directory``: if non-zero, add user site directory to
>   ``sys.path``

Similarly, these play a role when inferring the regular Python sys.path,
but are not something you should need to use when embedding.

> * ``dll_path`` (Windows only): Windows DLL path

I'd have to look up exactly how this is used, but I don't think it's
configurable (certainly not by this point where it's already been loaded).

> * ``filesystem_encoding``: Filesystem encoding,
>   ``sys.getfilesystemencoding()``
> * ``filesystem_errors``: Filesystem encoding errors,
>   ``sys.getfilesystemencodeerrors()``

See above. If we need these earlier in initialization (and I agree we
probably do), then let's just put them in the pre-initialize config.

> * ``dump_refs``: if non-zero, display all objects still alive at exit
> * ``inspect``: enter interactive mode after executing a script or a
>   command
> * ``interactive``: interactive mode
> * ``malloc_stats``: if non-zero, dump memory allocation statistics
>   at exit
> * ``quiet``: quiet mode (ex: don't display the copyright and version
>   messages even in interactive mode)
> * ``show_alloc_count``: show allocation counts at exit
> * ``show_ref_count``: show total reference count at exit
> * ``site_import``: import the ``site`` module at startup?
> * ``skip_source_first_line``: skip the first line of the source

These all seem to be flags specific to regular Python and not embedders
(providing we have ways for embedders to achieve the same thing through
their own API calls, but even if we don't, I'd still rather not codify
them as public runtime API). Having them be an optional flag for
Py_RunMain() or a PyMainConfig struct would keep them closer to where
they belong.

> * ``faulthandler``: if non-zero, call ``faulthandler.enable()``
> * ``install_signal_handlers``: install signal handlers?
> * ``tracemalloc``: if non-zero, call ``tracemalloc.start(value)``

Probably it's not a good idea to install signal handlers too early, but
I think the other two should be able to start any time between pre-init
and actual init, no? So make PyFaultHandler_Init() and
PyTraceMalloc_Init() public and say "if you want them, call them".

> * ``run_command``: ``-c COMMAND`` argument
> * ``run_filename``: ``python3 SCRIPT`` argument
> * ``run_module``: ``python3 -m MODULE`` argument

I think these should be in a separate configuration struct for
Py_RunMain(), probably along with the section above too.

(The other fields all seem fine to me.)

> This PEP adds a new ``Py_UnixMain()`` function which takes command line
> arguments as bytes::
>
>     int Py_UnixMain(int argc, char **argv)

I was part of the discussion about this, so I have some more context
than what's in the PEP.

Given your next example shows this function would be about six lines
long, why do we want to add it? Better to deprecate Py_Main() completely
and just copy/paste those six lines from elsewhere (which also helps in
the case where you need to do one little thing more and end up having to
find those six lines anyway, as in the same example).

> Open Questions
> ==============
>
> * Do we need to add an API for import ``inittab``?

I don't think so. Once embedders have a three-step initialization
(pre-init, init, runmain) there are two valid places they can call their
own init functions.

> * What about the stable ABI? Should we add a version into
>   ``PyPreConfig`` and ``PyConfig`` structures somehow? The Windows API
>   is known for its ABI stability and it stores the structure size into
>   the structure directly. Do the same?

Yeah, I think so. We already have the Py*_INIT macros, so we could just
use a version number that we increment manually, and that will let us
optionally ignore added members (or deprecated members that have not
been physically removed).

(Though I'd rather make it easier for applications to include a local
copy of Python rather than have to safely deal with whatever is
installed on the system. But that's a Windows-ism, so making both
approaches work is important :) )

> * The PEP 432 stores ``PYTHONCASEOK`` into the config. Do we need
>   to add something for that into ``PyConfig``? How would it be exposed
>   at the Python level for ``importlib``? Passed as an argument to
>   ``importlib._bootstrap._setup()`` maybe? It can be added later if
>   needed.

Could we convert it into an xoption? It's very rarely used, to my knowledge.

> * ``python._pth`` (Windows only)

I'd like to make this for all platforms, though my understanding was
that since Python is not really relocatable on other OS's it isn't so
important. And if the rest of this change happens it'll be easier to
implement anyway :)


From christian at python.org  Sun Mar 31 08:38:38 2019
From: christian at python.org (Christian Heimes)
Date: Sun, 31 Mar 2019 14:38:38 +0200
Subject: [Python-Dev] PEP 578: Python Runtime Audit Hooks
In-Reply-To: <ad05e257-d046-bba8-b070-c24c80a1521e@python.org>
References: <daa13e96-fa07-8703-4d73-234a862baf05@python.org>
 <2cb3740e-ebc2-1839-1d2e-73d1b9f0a445@python.org>
 <ad05e257-d046-bba8-b070-c24c80a1521e@python.org>
Message-ID: <8466c9c4-b5dc-c6c5-6fe4-a49dc2f4f968@python.org>

On 29/03/2019 21.10, Steve Dower wrote:
>> For example how does the importhook work in regarding of alternative
>> importers like zipimport? What does the import hook 'see' for an import
>> from a zipfile?
> 
> Yes, good point. I think opening the zip file with open_for_import() is
> the right place to do it, as this operation relates to opening the file
> on disk rather than files within it.

+1

>> Shared libraries are trickier. libc doesn't define a way to dlopen()
>> from a file descriptor. dlopen() takes a file name, but a file name
>> leaves the audit hook open to a TOCTOU attack.
> 
> For Windows, at least, the operating system can run its own validation
> on native modules (if you're using a feature like DeviceGuard, for
> example), so the hook likely isn't necessary for those purposes. I
> believe some configurations of Linux allow this as well?
> 
> But there's likely no better option here than a combination of good ACLs
> and checking by filename, which at least lets you whitelist the files
> you know you want to allow. Similarly for the zip file - if you trust a
> particular file and trust your ACLs, checking by filename is fine. That
> said, specific audit events for "I'm about to open this zip/dlopen this
> file for import" are very easy to add. (The PEP proposes many examples,
> but is not trying to be exhaustive. If accepted, we should feel free to
> add new events as we identify places where they matter.)

The Linux effort is called Integrity Measurement Architecture (IMA) and
Linux Extended Verification Module (EVM). I have no practical experience
with IMA yet.

I don't like the fact that the PEP requires users to learn and use an
additional layer to handle native code. Although we cannot provide a
fully secure hook for native code, we could at least try to provide a
best effort hook and document the limitations. A bit more information
would make the verified open function more useful, too.

PyObject *PyImport_OpenForExecution(
    const char *path,
    const char *intent,
    int flags,
    PyObject *context
)

- Path is an absolute (!) file path. The PEP doesn't specify if the file
name is relative or absolute. IMO it should be always absolute.

- The new intent argument lets the caller pass information how it
intents to use the file, e.g. pythoncode, zipimport, nativecode (for
loading a shared library/DLL), ctypes, ... This allows the verify hook
to react on the intent and provide different verifications for e.g.
Python code and native modules.

- The flags argument is for additional flags, e.g. return an opened file
or None, open the file in text or binary mode, ...

- Context is an optional Python object from the caller's context. For
the import system, it could be the loader instance.

Examples:

PyImport_OpenForImport(
    '/usr/lib64/python3.7/__pycache__/os.cpython-37.pyc',
    'bytecode',
    PY_IMPORT_OPENFILE,
    SourceFileLoader_instance
) -> fileobject

PyImport_OpenForImport(
    '/lib64/libc.so.6',
    'ctypes',
    PY_IMPORT_NONE,
    NULL
) -> None

> Aside: an important aspect of this per-file approach to execution is
> that the idea is generally to *enable* the files you trust, rather than
> disable the files that are bad. So the detection routines are typically
> "does this match a known hash" or "is this in a secure location", which
> for a carefully deployed system are already known values, rather than
> trying to figure out whether a file might do a bad thing. If you can't
> validate the files in your deployment match the ones you thought you
> were deploying, you are so far from needing this that it doesn't even
> matter, but most of the deployments I work with are *at least* this well
> controlled.

Absolutely!

On Linux, trust settings could be stored in extended file attributes.
Linux has multiple namespaces for extended attributes. User attributes
can be modified by every process that has write permission to an inode.
The security namespace is only writable for processes with
CAP_SYS_ADMIN. A secure loader could check for presence of
'user.org.python.bytecode' attribute or compare the content of the file
to hashsum in 'security.org.python.bytecode.sha256'.

Christian


From aratzml at opendeusto.es  Sun Mar 31 06:30:41 2019
From: aratzml at opendeusto.es (Aratz Manterola Lasa)
Date: Sun, 31 Mar 2019 12:30:41 +0200
Subject: [Python-Dev] Actor Model in Python
Message-ID: <CAHGMwNOyA_W4ebhMn8gg8DTfjO-5MFdfZAOthQjTVcuWEAkRqQ@mail.gmail.com>

Hello,
I was wondering if there was any Python project aiming to implement the
actor model for python concurrency. ?Does anyone know it?
Aratz.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190331/8a61ceab/attachment.html>