From mertz at gnosis.cx  Wed Jan  2 12:34:55 2019
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 2 Jan 2019 12:34:55 -0500
Subject: [Python-Dev] Interested in serving on Steering Council
Message-ID: <CAEbHw4bhjt3k-Q57BnSWw150B2Pint1465ick+775i-eaYeVBA@mail.gmail.com>

Hi Core Developers,

First thing: I am *not* a CPython committer.  I think most of you who are
will be somewhat familiar with me though.

Second: I was a Director of the PSF for a long while, and continue to chair
some Working Groups.  I've been mentioned in some PEPs.  I have written a
lot of articles and have given a lot of talks about Python, including about
recent or pending PEPs and similar matters.  I continue to work and train
around Python and open source (now with a focus on "data science", whatever
that is).

Third: I follow python-ideas and python-dev rather closely, and fairly
often contribute ideas to those lists.

Fourth: As I read PEP 8016, I cannot nominate myself to the Steering
Committee.  That seems good and proper to me.  But I believe I would be a
relevant and helpful member of the future Steering Committee if someone
wishes to nominate me and if the voters wish to elect me.

Thanks, David Mertz...

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190102/bd4b3ea4/attachment.html>

From solipsis at pitrou.net  Wed Jan  2 13:15:17 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 2 Jan 2019 19:15:17 +0100
Subject: [Python-Dev] Interested in serving on Steering Council
References: <CAEbHw4bhjt3k-Q57BnSWw150B2Pint1465ick+775i-eaYeVBA@mail.gmail.com>
Message-ID: <20190102191517.7fc2094c@fsol>

On Wed, 2 Jan 2019 12:34:55 -0500
David Mertz <mertz at gnosis.cx> wrote:
> Hi Core Developers,
> 
> First thing: I am *not* a CPython committer.  I think most of you who are
> will be somewhat familiar with me though.
> 
> Second: I was a Director of the PSF for a long while, and continue to chair
> some Working Groups.  I've been mentioned in some PEPs.  I have written a
> lot of articles and have given a lot of talks about Python, including about
> recent or pending PEPs and similar matters.  I continue to work and train
> around Python and open source (now with a focus on "data science", whatever
> that is).
> 
> Third: I follow python-ideas and python-dev rather closely, and fairly
> often contribute ideas to those lists.
> 
> Fourth: As I read PEP 8016, I cannot nominate myself to the Steering
> Committee.  That seems good and proper to me.  But I believe I would be a
> relevant and helpful member of the future Steering Committee if someone
> wishes to nominate me and if the voters wish to elect me.

The primary question I would ask an external candidate is: how is it
that you never became a core developer (which implies some amount of
effort and dedication) but nevertheless would be willing to spend the
effort and dedication needed for serving on a Steering Council (*)?

(*) (or Committee, I don't remember :-))

Regards

Antoine.



From paul at ganssle.io  Wed Jan  2 14:15:43 2019
From: paul at ganssle.io (Paul Ganssle)
Date: Wed, 2 Jan 2019 14:15:43 -0500
Subject: [Python-Dev] Return type of datetime subclasses added to timedelta
Message-ID: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>

Happy New Year everyone!

I would like to start a thread here for wider feedback on my proposal to
change the return type of the addition operation between a datetime
subclass and a timedelta. Currently, adding a timedelta to a subclass of
datetime /always/ returns a datetime rather than an instance of the
datetime subclass.

I have an open PR implementing this, PR #10902
<https://github.com/python/cpython/pull/10902>, but I know it's a major
change so I did not want to move forward without more discussion. I
first brought this up on datetime-SIG
<https://mail.python.org/archives/list/datetime-sig at python.org/thread/TGB3VZS5EKM4R2VFUA44323FZFRN2DSJ/>
[1], and we decided to move the discussion over here because the people
most likely to object to the change would be on this list and not on
datetime-SIG.

In addition to the datetime-SIG thread, you may find a detailed
rationale for the change in bpo-35364
<https://bugs.python.org/issue35364#msg331065> [2],? and a rationale for
why we would want to (and arguably already /do/) support subclassing
datetime in bpo-32417 <https://bugs.python.org/issue32417#msg331353> [3].

A short version of the strongest rationale for changing how this works
is that it is causing inconsistencies in how subclassing is handled in
alternate constructors of datetime. For a given subclass of datetime
(which I will call DateTimeSub), nearly all alternate constructors
already support subclasses correctly - DateTimeSub.fromtimestamp(x) will
return a DateTimeSub, for example. However, because DateTimeSub +
timedelta returns datetime, any alternate constructor implemented in
terms of timedelta additions will leak that implementation detail by
returning a datetime object instead of the subclass. The biggest problem
is that datetime.fromutc is defined in terms of timedelta addition, so
DateTimeSub.now() returns a DateTimeSub object, but
DateTimeSub.now(timezone.utc) returns a datetime object! This is one of
the most annoying things to work around when building a datetime
subclass, and I don't know of any situation where someone /wants/ their
subclass to be lost on addition with a timedelta.

From my understanding, this has been discussed before and the original
objection was that this implementation assumes that the datetime
subclass has a constructor with the same (or a sufficiently similar)
signature as datetime. This may be a legitimate gripe, but unfortunately
that ship has sailed long ago. All of datetime's alternate constructors
make this assumption. Any subclass that does not meet this requirement
must have worked around it long ago (or they don't care about alternate
constructors).

Thanks for your attention, I look forward to your replies.

Best,

Paul

[1]
https://mail.python.org/archives/list/datetime-sig at python.org/thread/TGB3VZS5EKM4R2VFUA44323FZFRN2DSJ/

[2] https://bugs.python.org/issue35364#msg331065

[3] https://bugs.python.org/issue32417#msg331353


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190102/87450a51/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190102/87450a51/attachment.sig>

From steve.dower at python.org  Thu Jan  3 22:27:10 2019
From: steve.dower at python.org (Steve Dower)
Date: Fri, 4 Jan 2019 14:27:10 +1100
Subject: [Python-Dev] Interested in serving on Steering Council
In-Reply-To: <20190102191517.7fc2094c@fsol>
References: <CAEbHw4bhjt3k-Q57BnSWw150B2Pint1465ick+775i-eaYeVBA@mail.gmail.com>
 <20190102191517.7fc2094c@fsol>
Message-ID: <cb08756a-cf4b-f6ad-5147-7ef3e41f4c95@python.org>

On 03Jan.2019 0515, Antoine Pitrou wrote:
> The primary question I would ask an external candidate is: how is it
> that you never became a core developer (which implies some amount of
> effort and dedication) but nevertheless would be willing to spend the
> effort and dedication needed for serving on a Steering Council (*)?
> 
> (*) (or Committee, I don't remember :-)
David may of course provide an answer for himself, but allow me to
provide my answer (and this is why I pushed for allowing external
nominations).

Historically, the only reason to become a core committer was to commit
code. Some of us no doubt desired or demonstrated greater influence, but
all of us have committed code or reviewed and merged PRs, either
directly to CPython or one of the related projects.

This is not a job for everyone, but it's been the only job we had on offer.

The closest alternative job was to be elected to the board of the Python
Software Foundation. But this is still not a job for everyone. They also
are not considered core committers, despite making significant
contributions.

We now have a new job on offer. Exactly what that job involves isn't
quite defined yet, but it will certainly include some amount of
project/program/process management, likely some customer/user engagement
(or relationship management, if you prefer), and potentially some
independent decision making.

Guido is the only core developer who has previously contributed to
Python in this way (whatever "this way" turns out to mean). The rest of
us happily worked under "someone else" doing it.

Meanwhile, many non-core committers in the Python community have spent
their time building companies, consulting businesses or educational
courses. Spending time writing code and reviewing PRs is not how they
want to contribute, and so they have contributed in other ways -
including writing and often reviewing PEPs. There was no need for them
to be a core committer, since they weren't doing any of the
committer-specific tasks.

In the PEP 8016 discussions (pre vote), we agreed that if we chose to
elect someone who is not currently a core developer, we would also
probably vote to make them a core developer, so there is no harm in
allowing externals to be nominated. Also since the core committers are
voluntarily submitting to their guidance, it makes sense that voting to
elect/dissolve is restricted to us.

In summary, members of the Steering Council are filling a new role with
only one explicit precedent within the core committers. The
qualifications are different, and so the pool of candidates is
different. The existing core committers will submit to the Steering
Council, and so are the ones who elect them.

(Note that I've carefully used "core committer" and "core developer"
above. I believe it's very important to distinguish between "write
access on GitHub" and "project decision maker", and no reason to force
an arbitrary overlap.)

Cheers,
Steve

From solipsis at pitrou.net  Fri Jan  4 05:04:54 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 4 Jan 2019 11:04:54 +0100
Subject: [Python-Dev] Interested in serving on Steering Council
In-Reply-To: <cb08756a-cf4b-f6ad-5147-7ef3e41f4c95@python.org>
References: <CAEbHw4bhjt3k-Q57BnSWw150B2Pint1465ick+775i-eaYeVBA@mail.gmail.com>
 <20190102191517.7fc2094c@fsol>
 <cb08756a-cf4b-f6ad-5147-7ef3e41f4c95@python.org>
Message-ID: <20190104110454.045cb0de@fsol>

On Fri, 4 Jan 2019 14:27:10 +1100
Steve Dower <steve.dower at python.org> wrote:
> 
> We now have a new job on offer. Exactly what that job involves isn't
> quite defined yet, but it will certainly include some amount of
> project/program/process management, likely some customer/user engagement
> (or relationship management, if you prefer), and potentially some
> independent decision making.
> 
> Guido is the only core developer who has previously contributed to
> Python in this way (whatever "this way" turns out to mean).

Not exactly.  Nick's role on packaging comes to mind.  More modestly,
several of us have served as BDFL delegates, have steered various
processes (such as VCS migration), and/or have been responsible
(officially or not) for subparts of the project (such as documentation,
buildbots, version control...).

> In the PEP 8016 discussions (pre vote), we agreed that if we chose to
> elect someone who is not currently a core developer, we would also
> probably vote to make them a core developer, so there is no harm in
> allowing externals to be nominated.

The Council is going to be a 5-person body, some some amount of
involvement and dedication is expected from each of the Council's
members if we want it to function correctly (it's probably not just a
supervision body where you can participate in a meeting every 3 months,
answer a couple e-mails and call it done).

I already have a hard time imagining my level of involvement being
enough for candidating on the Council.  So I would be skeptical of
voting for someone who hasn't submitted a single patch to the codebase
in 10+ years, for example.

Moreover, someone who has never contributed to the codebase hasn't
really experienced how contributing works, which doesn't make them a
very good candidate for managing contributors, IMHO.

Regards

Antoine.

From status at bugs.python.org  Fri Jan  4 13:07:38 2019
From: status at bugs.python.org (Python tracker)
Date: Fri, 04 Jan 2019 18:07:38 +0000
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190104180738.1.A332AF26462CD784@roundup.psfhosted.org>


ACTIVITY SUMMARY (2018-12-28 - 2019-01-04)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6922 (+19)
  closed 40487 (+34)
  total  47409 (+53)

Open issues with patches: 2748 


Issues opened (34)
==================

#35606: Add prod() function to the math module
https://bugs.python.org/issue35606  opened by rhettinger

#35608: python3 multiprocessing queue deadlock when use thread and pro
https://bugs.python.org/issue35608  opened by beruhan

#35610: IDLE: replace use of EditorWindow.context_use_ps1
https://bugs.python.org/issue35610  opened by terry.reedy

#35611: open doesn't call IncrementalEncoder with final=True
https://bugs.python.org/issue35611  opened by haney

#35615: "RuntimeError: Dictionary changed size during iteration" when 
https://bugs.python.org/issue35615  opened by ltfish

#35616: Change references to '4.0'.
https://bugs.python.org/issue35616  opened by terry.reedy

#35617: unittest discover does not work with implicit namespaces
https://bugs.python.org/issue35617  opened by Simon Fagerholm

#35618: Allow users to set suffix list in cookiejar policy
https://bugs.python.org/issue35618  opened by xtreak

#35619: Support custom data descriptors in pydoc
https://bugs.python.org/issue35619  opened by serhiy.storchaka

#35620: asyncio test failure on appveyor
https://bugs.python.org/issue35620  opened by terry.reedy

#35621: asyncio.create_subprocess_exec() only works with main event lo
https://bugs.python.org/issue35621  opened by sth

#35622: Add support for Linux SCHED_DEADLINE
https://bugs.python.org/issue35622  opened by mb_

#35624: Shelve sync issues while using Gevent
https://bugs.python.org/issue35624  opened by Oded Engel

#35625: documentation of list, set & dict comprehension make no mentio
https://bugs.python.org/issue35625  opened by bzip2

#35627: multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7
https://bugs.python.org/issue35627  opened by June Kim

#35628: Allow lazy loading of translations in gettext.
https://bugs.python.org/issue35628  opened by s-ball

#35629: hang and/or leaked processes with multiprocessing.Pool(...).im
https://bugs.python.org/issue35629  opened by Anthony Sottile

#35632: support unparse for Suite ast
https://bugs.python.org/issue35632  opened by thautwarm

#35633: test_eintr fails on AIX since fcntl functions were modified
https://bugs.python.org/issue35633  opened by Michael.Felt

#35634: kwargs regression when there are multiple entries with the sam
https://bugs.python.org/issue35634  opened by iceboy

#35635: asyncio.create_subprocess_exec() only works in main thread
https://bugs.python.org/issue35635  opened by stefan

#35636: remove redundant check in unicode_hash(PyObject *self)
https://bugs.python.org/issue35636  opened by Ma Lin

#35638: Introduce fixed point locale aware format type for floating po
https://bugs.python.org/issue35638  opened by steelman

#35639: Lowecasing Unicode Characters
https://bugs.python.org/issue35639  opened by kingofsevens

#35640: Allow passing PathLike arguments to SimpleHTTPRequestHandler
https://bugs.python.org/issue35640  opened by eamanu

#35642: _asynciomodule.c compiled in both pythoncore.vcxproj and _asyn
https://bugs.python.org/issue35642  opened by Gregory.Szorc

#35644: venv doesn't work on Windows when no venvlauncher executable p
https://bugs.python.org/issue35644  opened by Ray Donnelly

#35647: Cookie path check returns incorrect results
https://bugs.python.org/issue35647  opened by xtreak

#35649: http.client doesn't close. Infinite loop
https://bugs.python.org/issue35649  opened by skorpeo

#35651: PEP 257 (active) references PEP 258 (rejected) as if it were a
https://bugs.python.org/issue35651  opened by ExplodingCabbage

#35652: Add use_srcentry parameter to shutil.copytree() II
https://bugs.python.org/issue35652  opened by flokX

#35654: Remove 'guarantee' that sorting only relies on __lt__ from sor
https://bugs.python.org/issue35654  opened by mjpieters

#35656: More matchers in unittest.mock
https://bugs.python.org/issue35656  opened by Petter S

#35657: multiprocessing.Process.join() ignores timeout if child proces
https://bugs.python.org/issue35657  opened by Huazuo Gao



Most recent 15 issues with no replies (15)
==========================================

#35656: More matchers in unittest.mock
https://bugs.python.org/issue35656

#35652: Add use_srcentry parameter to shutil.copytree() II
https://bugs.python.org/issue35652

#35651: PEP 257 (active) references PEP 258 (rejected) as if it were a
https://bugs.python.org/issue35651

#35647: Cookie path check returns incorrect results
https://bugs.python.org/issue35647

#35642: _asynciomodule.c compiled in both pythoncore.vcxproj and _asyn
https://bugs.python.org/issue35642

#35640: Allow passing PathLike arguments to SimpleHTTPRequestHandler
https://bugs.python.org/issue35640

#35635: asyncio.create_subprocess_exec() only works in main thread
https://bugs.python.org/issue35635

#35632: support unparse for Suite ast
https://bugs.python.org/issue35632

#35628: Allow lazy loading of translations in gettext.
https://bugs.python.org/issue35628

#35625: documentation of list, set & dict comprehension make no mentio
https://bugs.python.org/issue35625

#35622: Add support for Linux SCHED_DEADLINE
https://bugs.python.org/issue35622

#35621: asyncio.create_subprocess_exec() only works with main event lo
https://bugs.python.org/issue35621

#35619: Support custom data descriptors in pydoc
https://bugs.python.org/issue35619

#35618: Allow users to set suffix list in cookiejar policy
https://bugs.python.org/issue35618

#35610: IDLE: replace use of EditorWindow.context_use_ps1
https://bugs.python.org/issue35610



Most recent 15 issues waiting for review (15)
=============================================

#35652: Add use_srcentry parameter to shutil.copytree() II
https://bugs.python.org/issue35652

#35642: _asynciomodule.c compiled in both pythoncore.vcxproj and _asyn
https://bugs.python.org/issue35642

#35638: Introduce fixed point locale aware format type for floating po
https://bugs.python.org/issue35638

#35636: remove redundant check in unicode_hash(PyObject *self)
https://bugs.python.org/issue35636

#35633: test_eintr fails on AIX since fcntl functions were modified
https://bugs.python.org/issue35633

#35619: Support custom data descriptors in pydoc
https://bugs.python.org/issue35619

#35617: unittest discover does not work with implicit namespaces
https://bugs.python.org/issue35617

#35616: Change references to '4.0'.
https://bugs.python.org/issue35616

#35615: "RuntimeError: Dictionary changed size during iteration" when 
https://bugs.python.org/issue35615

#35610: IDLE: replace use of EditorWindow.context_use_ps1
https://bugs.python.org/issue35610

#35606: Add prod() function to the math module
https://bugs.python.org/issue35606

#35605: backported patch requires new sphinx, minimum sphinx version w
https://bugs.python.org/issue35605

#35603: table header in output of difflib.HtmlDiff.make_table is not e
https://bugs.python.org/issue35603

#35598: IDLE: Modernize config_key module
https://bugs.python.org/issue35598

#35596: Fatal Python error: initfsencoding: unable to load the file sy
https://bugs.python.org/issue35596



Top 10 most discussed issues (10)
=================================

#35598: IDLE: Modernize config_key module
https://bugs.python.org/issue35598  11 msgs

#35638: Introduce fixed point locale aware format type for floating po
https://bugs.python.org/issue35638  10 msgs

#35431: Add a function for computing binomial coefficients to the math
https://bugs.python.org/issue35431   9 msgs

#35636: remove redundant check in unicode_hash(PyObject *self)
https://bugs.python.org/issue35636   9 msgs

#35605: backported patch requires new sphinx, minimum sphinx version w
https://bugs.python.org/issue35605   8 msgs

#35214: Get the test suite passing with clang Memory Sanitizer enabled
https://bugs.python.org/issue35214   7 msgs

#35644: venv doesn't work on Windows when no venvlauncher executable p
https://bugs.python.org/issue35644   7 msgs

#35603: table header in output of difflib.HtmlDiff.make_table is not e
https://bugs.python.org/issue35603   6 msgs

#35608: python3 multiprocessing queue deadlock when use thread and pro
https://bugs.python.org/issue35608   6 msgs

#35654: Remove 'guarantee' that sorting only relies on __lt__ from sor
https://bugs.python.org/issue35654   6 msgs



Issues closed (34)
==================

#19120: shlex.shlex.lineno reports a different number depending on the
https://bugs.python.org/issue19120  closed by cheryl.sabella

#20182: Derby #13: Convert 50 sites to Argument Clinic across 5 files
https://bugs.python.org/issue20182  closed by taleinat

#20849: add exist_ok to shutil.copytree
https://bugs.python.org/issue20849  closed by giampaolo.rodola

#28503: [Patch] '_crypt' module: fix implicit declaration of crypt(), 
https://bugs.python.org/issue28503  closed by gregory.p.smith

#32492: C Fast path for namedtuple's property/itemgetter pair
https://bugs.python.org/issue32492  closed by rhettinger

#33234: Improve list() pre-sizing for inputs with known lengths
https://bugs.python.org/issue33234  closed by pablogsal

#33987: IDLE: use ttk.Frame for ttk widgets
https://bugs.python.org/issue33987  closed by terry.reedy

#34055: IDLE: erroneous 'smart' indents in shell
https://bugs.python.org/issue34055  closed by terry.reedy

#35225: test_faulthandler fails under ubsan
https://bugs.python.org/issue35225  closed by gregory.p.smith

#35570: 2to3 creates code using deprecated imp module
https://bugs.python.org/issue35570  closed by terry.reedy

#35583: (glibc2.28/MIPS32EL) python 3.7.x interpreter segmentation fau
https://bugs.python.org/issue35583  closed by broly

#35588: Speed up mod/divmod/floordiv for Fraction type
https://bugs.python.org/issue35588  closed by serhiy.storchaka

#35594: Python script generating Segmentation Fault
https://bugs.python.org/issue35594  closed by terry.reedy

#35601: Race condition in test_signal_handling_args x86-64 High Sierra
https://bugs.python.org/issue35601  closed by pablogsal

#35602: cleanup code may fail in test_asyncio.test_unix_events.Selecto
https://bugs.python.org/issue35602  closed by pablogsal

#35607: python3 multiprocessing queue deadlock when use thread and pro
https://bugs.python.org/issue35607  closed by beruhan

#35609: Improve of abc.py docstring
https://bugs.python.org/issue35609  closed by serhiy.storchaka

#35612: Text wrap over text containing tab character
https://bugs.python.org/issue35612  closed by Devika Sondhi

#35613: Escaping string containing invalid characters as per XML
https://bugs.python.org/issue35613  closed by ned.deily

#35614: Broken help() on metaclasses
https://bugs.python.org/issue35614  closed by ncoghlan

#35623: Segfault in test_bigmem.ListTest.test_sort
https://bugs.python.org/issue35623  closed by benjamin.peterson

#35626: Python dictreader KeyError issue
https://bugs.python.org/issue35626  closed by eric.smith

#35630: Missing code tag for "python3" in README.rst
https://bugs.python.org/issue35630  closed by benjamin.peterson

#35631: Improve typing docs wrt abstract/concrete collection types
https://bugs.python.org/issue35631  closed by levkivskyi

#35637: Factorial should be able to evaluate float arguments
https://bugs.python.org/issue35637  closed by serhiy.storchaka

#35641: IDLE: calltips not properly formatted for functions without do
https://bugs.python.org/issue35641  closed by taleinat

#35643: SyntaxWarning: invalid escape sequence in Modules/_sha3/cleanu
https://bugs.python.org/issue35643  closed by benjamin.peterson

#35645: Alarm usage
https://bugs.python.org/issue35645  closed by steven.daprano

#35646: python -v writes to stderr
https://bugs.python.org/issue35646  closed by eric.smith

#35648: Add use_srcentry parameter to shutil.copytree()
https://bugs.python.org/issue35648  closed by flokX

#35650: cygwin treats X and X.exe as the same file
https://bugs.python.org/issue35650  closed by zach.ware

#35653: All regular expression match groups are the empty string
https://bugs.python.org/issue35653  closed by mrabarnett

#35655: documentation have wrong info about fibo module examples
https://bugs.python.org/issue35655  closed by christian.heimes

#35658: Reggie_Linear_Regression_Solution.ipynb devide by 10 diff with
https://bugs.python.org/issue35658  closed by steven.daprano

From mertz at gnosis.cx  Fri Jan  4 15:24:20 2019
From: mertz at gnosis.cx (David Mertz)
Date: Fri, 4 Jan 2019 15:24:20 -0500
Subject: [Python-Dev] Interested in serving on Steering Council
In-Reply-To: <cb08756a-cf4b-f6ad-5147-7ef3e41f4c95@python.org>
References: <CAEbHw4bhjt3k-Q57BnSWw150B2Pint1465ick+775i-eaYeVBA@mail.gmail.com>
 <20190102191517.7fc2094c@fsol>
 <cb08756a-cf4b-f6ad-5147-7ef3e41f4c95@python.org>
Message-ID: <CAEbHw4b-dWKeYWaYoGBS1HMxJjuN0gquM99mKfuNvvU3wmwzGQ@mail.gmail.com>

I do not wish to presume too much on the judgement of the core developers.
But I thank Steve Dower for his characterizations which pretty much exactly
explain why I've had those involvements with the Python community and
language I have had, and haven't done other things.

I've been part of the Python community since 1998, but really active in it
since about 2001.  During the early 2000s, I wrote a large number of widely
read articles promoting Python, often delving into explaining semi-obscure
features and language design issues.  Most of these were with in my column
_Charming Python_.  I believe that several changes in Python itself?such as
making coroutines easier to use and the design of metaclasses and class
decorators?were significantly influenced by things I wrote on the topics.

Mostly in the period after writing that column, i.e. during the 2010s, I
served as a Director of the PSF; both before and since my time as a
Director, I've chaired several PSF committees.  That likewise felt like a
way I could advance Python best, but from more of an organizational or
social perspective than a technical one.  It is interesting to me that
whereas when I started volunteering for the PSF, there was significant
overlap between the PSF board and the core-committers, I think there is
little or no overlap today.  For better or worse, PSF is much more
community than technical today.  I feel like my own skills and interest
remain somewhat at the intersection of those aspects of Python.

I did not choose during that time, nor since, to become a CPython core
developer.  I've certainly contributed to other projects in the Python
ecosystem (I'm not sure if those are "related projects" in the sense Steve
mentions).  Part of that is time commitment needed, but more of it is my
personal strategic choices about what I could best do to advance Python in
the world.  I've felt I can do more by writing, speaking, and participating
in the PSF, than I would have by working on the CPython code base itself.

In particular, I always felt that I am not nearly as strong of a *C*
developer as are most core developers.  In Python itself, yes, but not in
C.  I am certain that I could have found some small bug to fix or small
feature to add, and gotten it accepted.  But doing that would have taken me
comparatively more effort than it would many others; I felt that effort was
better targeted towards educating Python users and teaching the user-level
language design choices Python has made.

If the core developers feel that the overwhelming qualification for the
Steering Committee is familiarity with the C code base of CPython, then
indeed I am not the best candidate for that.  If language design issues are
more important?and especially if thinking about Python's place among users
and industry are important, then I think I'm a very good candidate for the
role.  In particular, I believe my judgement about "Is this feature good
for Python users?" would be as good as that of most anyone (maybe other
than Guido); but I recognize that my judgement about "Is this feature
straightforward to implement in CPython?" or "What are the performance
implications of this features?" are weaker than those of most core
developers.  Not to say I have *no* instinct about those other questions,
but I know to defer.

Best, David...


> (*) (or Committee, I don't remember :-)
> David may of course provide an answer for himself, but allow me to
> provide my answer (and this is why I pushed for allowing external
> nominations).
>
> Historically, the only reason to become a core committer was to commit
> code. Some of us no doubt desired or demonstrated greater influence, but
> all of us have committed code or reviewed and merged PRs, either
> directly to CPython or one of the related projects.
>
> This is not a job for everyone, but it's been the only job we had on offer.
>
> The closest alternative job was to be elected to the board of the Python
> Software Foundation. But this is still not a job for everyone. They also
> are not considered core committers, despite making significant
> contributions.
>
> We now have a new job on offer. Exactly what that job involves isn't
> quite defined yet, but it will certainly include some amount of
> project/program/process management, likely some customer/user engagement
> (or relationship management, if you prefer), and potentially some
> independent decision making.
>
> Guido is the only core developer who has previously contributed to
> Python in this way (whatever "this way" turns out to mean). The rest of
> us happily worked under "someone else" doing it.
>
> Meanwhile, many non-core committers in the Python community have spent
> their time building companies, consulting businesses or educational
> courses. Spending time writing code and reviewing PRs is not how they
> want to contribute, and so they have contributed in other ways -
> including writing and often reviewing PEPs. There was no need for them
> to be a core committer, since they weren't doing any of the
> committer-specific tasks.
>
> In the PEP 8016 discussions (pre vote), we agreed that if we chose to
> elect someone who is not currently a core developer, we would also
> probably vote to make them a core developer, so there is no harm in
> allowing externals to be nominated. Also since the core committers are
> voluntarily submitting to their guidance, it makes sense that voting to
> elect/dissolve is restricted to us.
>
> In summary, members of the Steering Council are filling a new role with
> only one explicit precedent within the core committers. The
> qualifications are different, and so the pool of candidates is
> different. The existing core committers will submit to the Steering
> Council, and so are the ones who elect them.
>
> (Note that I've carefully used "core committer" and "core developer"
> above. I believe it's very important to distinguish between "write
> access on GitHub" and "project decision maker", and no reason to force
> an arbitrary overlap.)
>
> Cheers,
> Steve
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190104/30ae92e5/attachment.html>

From solipsis at pitrou.net  Fri Jan  4 15:36:05 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 4 Jan 2019 21:36:05 +0100
Subject: [Python-Dev] Interested in serving on Steering Council
In-Reply-To: <CAEbHw4b-dWKeYWaYoGBS1HMxJjuN0gquM99mKfuNvvU3wmwzGQ@mail.gmail.com>
References: <CAEbHw4bhjt3k-Q57BnSWw150B2Pint1465ick+775i-eaYeVBA@mail.gmail.com>
 <20190102191517.7fc2094c@fsol>
 <cb08756a-cf4b-f6ad-5147-7ef3e41f4c95@python.org>
 <CAEbHw4b-dWKeYWaYoGBS1HMxJjuN0gquM99mKfuNvvU3wmwzGQ@mail.gmail.com>
Message-ID: <20190104213605.57fba25b@fsol>


Hi David,

On Fri, 4 Jan 2019 15:24:20 -0500
David Mertz <mertz at gnosis.cx> wrote:
> 
> I've been part of the Python community since 1998, but really active in it
> since about 2001.  During the early 2000s, I wrote a large number of widely
> read articles promoting Python, often delving into explaining semi-obscure
> features and language design issues.  Most of these were with in my column
> _Charming Python_.  I believe that several changes in Python itself?such as
> making coroutines easier to use and the design of metaclasses and class
> decorators?were significantly influenced by things I wrote on the topics.
> [snip]

Those are useful things to know, thank you.

> If the core developers feel that the overwhelming qualification for the
> Steering Committee is familiarity with the C code base of CPython, then
> indeed I am not the best candidate for that.

Obviously not the overwhelming qualification (though at least _some_ of
the committee members would have to be familiar with the C code base, I
think).

> If language design issues are
> more important?and especially if thinking about Python's place among users
> and industry are important, then I think I'm a very good candidate for the
> role.

That, but I think also familiarity with the development and
contribution process, will definitely play a role.  In other words, if
some external candidate gets elected I would hope they take the time to
become familiar with how things work in that regard, and try to
contribute themselves (not necessarily to make important contributions
to the codebase but to understand the daily routine).

Regards

Antoine.

From alexander.belopolsky at gmail.com  Sat Jan  5 03:55:44 2019
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Sat, 5 Jan 2019 11:55:44 +0300
Subject: [Python-Dev] Return type of datetime subclasses added to
 timedelta
In-Reply-To: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
References: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
Message-ID: <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>

On Wed, Jan 2, 2019 at 10:18 PM Paul Ganssle <paul at ganssle.io> wrote:

> .. the original objection was that this implementation assumes that the
> datetime subclass has a constructor with the same (or a sufficiently
> similar) signature as datetime.
>
While this was used as a possible rationale for the way standard types
behave, the main objection to changing datetime classes is that it will
make them behave differently from builtins.  For example:

>>> class F(float):
...     pass
...
>>> type(F.fromhex('AA'))
<class '__main__.F'>
>>> type(F(1) + F(2))
<class 'float'>

This may be a legitimate gripe, but unfortunately that ship has sailed long
> ago. All of datetime's alternate constructors make this assumption. Any
> subclass that does not meet this requirement must have worked around it
> long ago (or they don't care about alternate constructors).
>

This is right, but the same argument is equally applicable to int, float,
etc. subclasses.  If you want to limit your change to datetime types you
should explain what makes these types special.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190105/2b3720ca/attachment.html>

From pmiscml at gmail.com  Sun Jan  6 06:10:48 2019
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Sun, 6 Jan 2019 13:10:48 +0200
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
Message-ID: <20190106131048.6e0a3dd8@x230>

Hello,

As explained in
https://docs.python.org/3/reference/compound_stmts.html#the-try-statement ,

except E as N:
    foo

is actually compiled as:

except E as N:
    try:
        foo
    finally:
        del N

But looking at the generated bytecode, it's worse than that:

             16 STORE_NAME               1 (N)
             18 POP_TOP
             20 SETUP_FINALLY            8 (to 30)

  4          22 LOAD_NAME                2 (foo)
             24 POP_TOP
             26 POP_BLOCK
             28 LOAD_CONST               0 (None)
        >>   30 LOAD_CONST               0 (None)
             32 STORE_NAME               1 (N)
             34 DELETE_NAME              1 (N)
             36 END_FINALLY


It's clear that what happens there is that first None is stored to N,
just to del it as the next step. Indeed, that's what done in the
compile.c:

https://github.com/python/cpython/blob/master/Python/compile.c#L2905

Storing None looks superfluous. For example, DELETE_FAST explicitly
stores NULL on delete.
https://github.com/python/cpython/blob/master/Python/ceval.c#L2249 

Likewise, for DELETE_NAME/DELETE_GLOBAL, PyObject_DelItem() is
called which translates to:

m->mp_ass_subscript(o, key, (PyObject*)NULL);

So hopefully any compliant mapping implementation actually clears
stored value, not leaving it dangling.

The "store None" behavior can be traced down to introduction of the
"del except target var" behavior back in 2007:
https://github.com/python/cpython/commit/b940e113bf90ff71b0ef57414ea2beea9d2a4bc0#diff-cb296cc5109f5640ff3f6d7198a6abeeR1999

There's no clear explanation why it's done like that, so probably an
artifact of the initial implementation. Note that even
https://github.com/python/cpython/commit/520b7ae27e39d1c77ea74ccd1b184d7cb43f9dcb
which did quite a bunch of refactoring to "except" implementation, and
reformatted this code, otherwise left it in place.

P.S.
I actually wanted to argue that such an implementation is hardly ideal
at all. Consider:

----
e = "my precious data"

try:
    1/0
except Exception as e:
    pass

# Where's my *global* variable?
# Worse, my variable can be gone or not, depending on whether exception
# triggered or not.
print(e)
----

So, perhaps the change should be not removing "e = None" part, but
conversely, removing the "del e" part.


-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From rosuav at gmail.com  Sun Jan  6 06:19:39 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 6 Jan 2019 22:19:39 +1100
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <20190106131048.6e0a3dd8@x230>
References: <20190106131048.6e0a3dd8@x230>
Message-ID: <CAPTjJmqhK24uFSpbQTe3qUhDU_tO+ENhw2psXj0g5XDg+2wgOg@mail.gmail.com>

On Sun, Jan 6, 2019 at 10:12 PM Paul Sokolovsky <pmiscml at gmail.com> wrote:
> It's clear that what happens there is that first None is stored to N,
> just to del it as the next step. Indeed, that's what done in the
> compile.c:
>
> https://github.com/python/cpython/blob/master/Python/compile.c#L2905
>
> Storing None looks superfluous.
>
> There's no clear explanation why it's done like that, so probably an
> artifact of the initial implementation.

With something as clearly deliberate as this, it's generally safe to
assume there's a good reason for it. But I would very much like to see
that reason documented, preferably with an example of something that
could otherwise fail.

> P.S.
> I actually wanted to argue that such an implementation is hardly ideal
> at all. Consider:
>
> ----
> e = "my precious data"
>
> try:
>     1/0
> except Exception as e:
>     pass
>
> # Where's my *global* variable?
> # Worse, my variable can be gone or not, depending on whether exception
> # triggered or not.
> print(e)
> ----
>
> So, perhaps the change should be not removing "e = None" part, but
> conversely, removing the "del e" part.

No, there's a good reason for having the "del e", and that's to
prevent reference loops (since the exception includes the full
traceback, including all local variables). However, I would argue -
and HAVE argued - that this would be a perfect situation for an inner
scope, where the inner variable "e" is actually shadowing, rather than
overwriting, your outer "e". But the push-back against *that* proposal
was insane.

ChrisA

From pmiscml at gmail.com  Sun Jan  6 06:40:43 2019
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Sun, 6 Jan 2019 13:40:43 +0200
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <CAPTjJmqhK24uFSpbQTe3qUhDU_tO+ENhw2psXj0g5XDg+2wgOg@mail.gmail.com>
References: <20190106131048.6e0a3dd8@x230>
 <CAPTjJmqhK24uFSpbQTe3qUhDU_tO+ENhw2psXj0g5XDg+2wgOg@mail.gmail.com>
Message-ID: <20190106134043.54b6d12c@x230>

Hello,

On Sun, 6 Jan 2019 22:19:39 +1100
Chris Angelico <rosuav at gmail.com> wrote:

> > It's clear that what happens there is that first None is stored to
> > N, just to del it as the next step. Indeed, that's what done in the
> > compile.c:
> >
> > https://github.com/python/cpython/blob/master/Python/compile.c#L2905
> >
> > Storing None looks superfluous.
> >
> > There's no clear explanation why it's done like that, so probably an
> > artifact of the initial implementation.  
> 
> With something as clearly deliberate as this, it's generally safe to
> assume there's a good reason for it.

Absolutely, and one doesn't need to look far for an example. For 20
years Python was devoid of ":=", then, suddenly... ;-).

> But I would very much like to see
> that reason documented, preferably with an example of something that
> could otherwise fail.

Yeah, I'm keen to know too!

[]

> > So, perhaps the change should be not removing "e = None" part, but
> > conversely, removing the "del e" part.  
> 
> No, there's a good reason for having the "del e", and that's to
> prevent reference loops 

So, I'm not sure if I made that clear, but the issue is that currently
the generated code tries to break the reference loop *twice*. But once
should be enough. And which one of "N = None" or "del N" is better is
something to consider.

-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From steve at pearwood.info  Sun Jan  6 07:10:24 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 6 Jan 2019 23:10:24 +1100
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <20190106131048.6e0a3dd8@x230>
References: <20190106131048.6e0a3dd8@x230>
Message-ID: <20190106121011.GX13616@ando.pearwood.info>

On Sun, Jan 06, 2019 at 01:10:48PM +0200, Paul Sokolovsky wrote:

[...]
> P.S.
> I actually wanted to argue that such an implementation is hardly ideal
> at all. Consider:
> 
> ----
> e = "my precious data"
> try:
>     1/0
> except Exception as e:
>     pass
> 
> # Where's my *global* variable?
> # Worse, my variable can be gone or not, depending on whether exception
> # triggered or not.
> print(e)

That's not "worse", that's BETTER.

With e deleted, you get an immediate exception the moment you try to use 
it, and you *know* something has gone wrong, instead of the program 
continuing, either to fail much later, where it isn't clear why the code 
fails, or worse, not fail at all, but instead silently do the wrong 
thing.

Incorrect code that fails sooner is better than incorrect code that 
fails later, and MUCH better than incorrect code that doesn't fail at 
all.

The real problem here is that if e holds your precious data, why are you 
assigning an exception to e? You probably wouldn't complain that this 
breaks your program:

e = "my precious data"
import sys as e
# where is my data???

or this:

e = "my precious data"
e = None

so why do you expect `except Exception as e` to be different? Its just a 
name binding operation, the same as all the others.

Regardless of the mechanism, if you don't want to lose your precious 
data, then don't use the same name. But if you truly, really must 
overwrite e, then there is a simple work-around:

e = "my precious data"
try:
    1/0
except Exception as err:
    e = err


Having the exception name unbound at the end of the block is a weird and 
unfortunate wart on the language, but not as unfortunate as having 
exceptions cause long-lasting reference cycles.


-- 
Steve

From pmiscml at gmail.com  Sun Jan  6 08:03:03 2019
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Sun, 6 Jan 2019 15:03:03 +0200
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <20190106121011.GX13616@ando.pearwood.info>
References: <20190106131048.6e0a3dd8@x230>
 <20190106121011.GX13616@ando.pearwood.info>
Message-ID: <20190106150303.57b77c5a@x230>

Hello,

On Sun, 6 Jan 2019 23:10:24 +1100
Steven D'Aprano <steve at pearwood.info> wrote:

[]

> > # Where's my *global* variable?
> > # Worse, my variable can be gone or not, depending on whether
> > exception # triggered or not.
> > print(e)  
> 
> That's not "worse", that's BETTER.
> 
> With e deleted, you get an immediate exception the moment you try to
> use it, and you *know* something has gone wrong

Ack. Such an explanation makes sense, though semantics behind it is
"magic". And given that variable deletion happens conditionally on
whether exception happened or not, it's still pretty strange. I would
definitely find a behavior of merely clearing an "except" target
variable to be more understandable than magic fiddling with namespaces.

[]

> The real problem here is that if e holds your precious data, why are
> you assigning an exception to e?

Because I got an idea that "except Exc as var" introduces a nested
lexical scope for "var"? I.e., a completely new variable is introduced,
which may shadow, but not override, any on the enclosing scope.

Why would I get that idea? Well, because lexical scoping is a
well-known concept in many programming languages, Python included. For
example, list/set/etc. comprehensions have proper lexical scope, how
would I imagine that specifically in except clauses, it's not a proper
lexical scoping, but a leaky "emulation" of it? 

That said, the description in
https://docs.python.org/3/reference/compound_stmts.html#the-try-statement
doesn't give any false promises regarding lexical scoping. Instead, it
effectively says that the behavior is CPython implementation detail:

"Exceptions are cleared because with the traceback attached to them,
they form a reference cycle"


So, the only thing to take away from this might be indeed an opportunity
to save 4 bytes of bytecode/2 dispatches.

> Having the exception name unbound at the end of the block is a weird
> and unfortunate wart on the language, but not as unfortunate as
> having exceptions cause long-lasting reference cycles.

Thanks, that summarizes it well. And well, my interest is also how
non-compliant would be for another Python implementation to act
differently, specifically to skip wrapping an except handler body in
try-finally (i.e., go back to Python2 behavior). I'm keen to add such
an option to my fork of MicroPython.

> 
> 
> -- 
> Steve

[]

-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From steve at pearwood.info  Sun Jan  6 09:14:55 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 7 Jan 2019 01:14:55 +1100
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <20190106150303.57b77c5a@x230>
References: <20190106131048.6e0a3dd8@x230>
 <20190106121011.GX13616@ando.pearwood.info> <20190106150303.57b77c5a@x230>
Message-ID: <20190106141453.GY13616@ando.pearwood.info>

On Sun, Jan 06, 2019 at 03:03:03PM +0200, Paul Sokolovsky wrote:

[...]
> Ack. Such an explanation makes sense, though semantics behind it is
> "magic". And given that variable deletion happens conditionally on
> whether exception happened or not, it's still pretty strange.

I don't think anyone *likes* the current behaviour. Speaking for myself, 
I think it is definitely a wart on the language, but a justifiable one, 
since the alternative is worse.

So yes, it is strange, and is something that Python programmers have to 
learn. Probably through experience.


> I would
> definitely find a behavior of merely clearing an "except" target
> variable to be more understandable than magic fiddling with namespaces.

I don't understand what this means. What is "clearing an except target"? 
Do you mean deleting the target name? Names are either bound to an 
object, or they aren't bound at all, so I don't know what it means to 
"clear" a name if it doesn't mean delete it.


> []
> 
> > The real problem here is that if e holds your precious data, why are
> > you assigning an exception to e?
> 
> Because I got an idea that "except Exc as var" introduces a nested
> lexical scope for "var"? I.e., a completely new variable is introduced,
> which may shadow, but not override, any on the enclosing scope.
> 
> Why would I get that idea? Well, because lexical scoping is a
> well-known concept in many programming languages, Python included. For
> example, list/set/etc. comprehensions have proper lexical scope, how
> would I imagine that specifically in except clauses, it's not a proper
> lexical scoping, but a leaky "emulation" of it? 

It isn't an emulation of lexical scope at all. Only the exception target 
is deleted, not other variables bound inside the block.

But a better question is, why would you (generic, not you personally) 
imagine that, alone out of all flow control statements, ONLY "except" 
clauses introduce a new scope? Every other flow control statement (for, 
while, if, elif, else, try, with) runs in the current scope. The only 
statements which create a new scope are def and class. (Did I miss any?)


[...]
> Thanks, that summarizes it well. And well, my interest is also how
> non-compliant would be for another Python implementation to act
> differently, specifically to skip wrapping an except handler body in
> try-finally (i.e., go back to Python2 behavior). I'm keen to add such
> an option to my fork of MicroPython.

Wouldn't that mean that MicroPython suffers from the exception/traceback 
reference cycle problem? How do you propose to solve that?



-- 
Steve

From rosuav at gmail.com  Sun Jan  6 09:44:23 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 7 Jan 2019 01:44:23 +1100
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <20190106141453.GY13616@ando.pearwood.info>
References: <20190106131048.6e0a3dd8@x230>
 <20190106121011.GX13616@ando.pearwood.info>
 <20190106150303.57b77c5a@x230> <20190106141453.GY13616@ando.pearwood.info>
Message-ID: <CAPTjJmrTe_dqaiho+TAiyBvGXQ+mNSg2AQOvEx6=4rk=7qi-Mw@mail.gmail.com>

On Mon, Jan 7, 2019 at 1:21 AM Steven D'Aprano <steve at pearwood.info> wrote:
> > I would
> > definitely find a behavior of merely clearing an "except" target
> > variable to be more understandable than magic fiddling with namespaces.
>
> I don't understand what this means. What is "clearing an except target"?
> Do you mean deleting the target name? Names are either bound to an
> object, or they aren't bound at all, so I don't know what it means to
> "clear" a name if it doesn't mean delete it.

Setting it to None before unbinding it. It's apparently not enough to
say "del e" - you have to do "e = None; del e". Check the OP.

ChrisA

From storchaka at gmail.com  Sun Jan  6 10:26:09 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 6 Jan 2019 17:26:09 +0200
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <20190106131048.6e0a3dd8@x230>
References: <20190106131048.6e0a3dd8@x230>
Message-ID: <q0t6id$a79$1@blaine.gmane.org>

06.01.19 13:10, Paul Sokolovsky ????:
> Storing None looks superfluous. For example, DELETE_FAST explicitly
> stores NULL on delete.
> https://github.com/python/cpython/blob/master/Python/ceval.c#L2249
> 
> Likewise, for DELETE_NAME/DELETE_GLOBAL, PyObject_DelItem() is
> called which translates to:
> 
> m->mp_ass_subscript(o, key, (PyObject*)NULL);
> 
> So hopefully any compliant mapping implementation actually clears
> stored value, not leaving it dangling.
> 
> The "store None" behavior can be traced down to introduction of the
> "del except target var" behavior back in 2007:
> https://github.com/python/cpython/commit/b940e113bf90ff71b0ef57414ea2beea9d2a4bc0#diff-cb296cc5109f5640ff3f6d7198a6abeeR1999
> 
> There's no clear explanation why it's done like that, so probably an
> artifact of the initial implementation. Note that even
> https://github.com/python/cpython/commit/520b7ae27e39d1c77ea74ccd1b184d7cb43f9dcb
> which did quite a bunch of refactoring to "except" implementation, and
> reformatted this code, otherwise left it in place.

Because there is a reason for such code.

See issue1631942 [1] and the thread with the subject "self-contained 
exceptions" on the Python-3000 mailing list [2] for the rationale.

In short, the code

     try:
         1/0
     except Exception as e:
         del e

should work.

I do not see a problem with storing None before deleting a variable. 
This is not a performance critical code, because it is executed only 
when an exception was raised and caught.

[1] https://bugs.python.org/issue1631942
[2] https://mail.python.org/pipermail/python-3000/2007-January/005294.html


From paul at ganssle.io  Sun Jan  6 10:32:11 2019
From: paul at ganssle.io (Paul Ganssle)
Date: Sun, 6 Jan 2019 10:32:11 -0500
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <20190106141453.GY13616@ando.pearwood.info>
References: <20190106131048.6e0a3dd8@x230>
 <20190106121011.GX13616@ando.pearwood.info> <20190106150303.57b77c5a@x230>
 <20190106141453.GY13616@ando.pearwood.info>
Message-ID: <4d012c3d-fc57-3fdc-1721-18f5384e20e3@ganssle.io>

On 1/6/19 9:14 AM, Steven D'Aprano wrote:
> [...]
> But a better question is, why would you (generic, not you personally) 
> imagine that, alone out of all flow control statements, ONLY "except" 
> clauses introduce a new scope? Every other flow control statement (for, 
> while, if, elif, else, try, with) runs in the current scope. The only 
> statements which create a new scope are def and class. (Did I miss any?)

To be fair except is already unique in that it /does/ "pseudo-scope" the
binding to the variable. The other obvious comparisons are to for loops
and context managers, both of which bind a value to a name that survives
after the exit of the control flow statement.

Given the reference counting reasons for exceptions *not* to outlive
their control flow statement, they are the "odd man out" in that they
delete the exception after the control statement's body exits. To me,
the natural and intuitive way to do this would be to have the exception
live in its own scope where it shadows existing variables, rather than
replacing and then completely removing them. The way it works now is
halfway between the behavior of existing control flow statements and
having a proper nested scope.

Not saying that anything has to change - in the end this is one of the
more minor "gotchas" about Python, and there may be practical reasons
for leaving it as it is - but I do think it's worth noting that for many
people this /will/ be surprising behavior, even if you know that
exceptions don't survive outside of the "except" block.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/69d9425d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/69d9425d/attachment-0001.sig>

From rosuav at gmail.com  Sun Jan  6 10:35:37 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 7 Jan 2019 02:35:37 +1100
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <q0t6id$a79$1@blaine.gmane.org>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
Message-ID: <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>

On Mon, Jan 7, 2019 at 2:27 AM Serhiy Storchaka <storchaka at gmail.com> wrote:
> > The "store None" behavior can be traced down to introduction of the
> > "del except target var" behavior back in 2007:
> > https://github.com/python/cpython/commit/b940e113bf90ff71b0ef57414ea2beea9d2a4bc0#diff-cb296cc5109f5640ff3f6d7198a6abeeR1999
> >
> > There's no clear explanation why it's done like that, so probably an
> > artifact of the initial implementation. Note that even
> > https://github.com/python/cpython/commit/520b7ae27e39d1c77ea74ccd1b184d7cb43f9dcb
> > which did quite a bunch of refactoring to "except" implementation, and
> > reformatted this code, otherwise left it in place.
>
> Because there is a reason for such code.

What reason though??

> See issue1631942 [1] and the thread with the subject "self-contained
> exceptions" on the Python-3000 mailing list [2] for the rationale.
>
> In short, the code
>
>      try:
>          1/0
>      except Exception as e:
>          del e
>
> should work.

But obviously not, because the code was specifically written to set it
to None first. It's not the sort of thing that can be done
accidentally, is it?

ChrisA

From storchaka at gmail.com  Sun Jan  6 10:50:46 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 6 Jan 2019 17:50:46 +0200
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
 <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
Message-ID: <q0t80n$m9u$1@blaine.gmane.org>

06.01.19 17:35, Chris Angelico ????:
> On Mon, Jan 7, 2019 at 2:27 AM Serhiy Storchaka <storchaka at gmail.com> wrote:
>> Because there is a reason for such code.
> 
> What reason though??

I added references and excerpts in the previous message.

>> In short, the code
>>
>>       try:
>>           1/0
>>       except Exception as e:
>>           del e
>>
>> should work.
> 
> But obviously not, because the code was specifically written to set it
> to None first.
Did you tried it? It works.


From paul at ganssle.io  Sun Jan  6 11:02:43 2019
From: paul at ganssle.io (Paul Ganssle)
Date: Sun, 6 Jan 2019 11:02:43 -0500
Subject: [Python-Dev] Return type of datetime subclasses added to
 timedelta
In-Reply-To: <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>
References: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
 <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>
Message-ID: <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>

I can think of many reasons why datetime is different from builtins,
though to be honest I'm not sure that consistency for its own sake is
really a strong argument for keeping a counter-intuitive behavior - and
to be honest I'm open to the idea that /all/ arithmetic types /should/
have some form of this change.

That said, I would say that the biggest difference between datetime and
builtins (other than the fact that datetime is /not/ a builtin, and as
such doesn't necessarily need to be categorized in this group), is that
unlike almost all other arithmetic types, /datetime/ has a special,
dedicated type for describing differences in datetimes. Using your
example of a float subclass, consider that without the behavior of
"addition of floats returns floats", it would be hard to predict what
would happen in this situation:

>>> F(1.2) + 3.4

Would that always return a float, even though F(1.2) + F(3.4) returns an
F? Would that return an F because F is the left-hand operand? Would it
return a float because float is the right-hand operand? Would you walk
the MROs and find the lowest type in common between the operands and
return that? It's not entirely clear which subtype predominates. With
datetime, you have:

datetime - datetime -> timedelta
datetime ? timedelta -> datetime
timedelta ? timedelta -> timedelta

There's no operation between two datetime objects that would return a
datetime object, so it's always clear: operations between datetime
subclasses return timedelta, operations between a datetime object and a
timedelta return the subclass of the datetime that it was added to or
subtracted from.

Of course, the real way to resolve whether datetime should be different
from int/float/string/etc is to look at why this choice was actually
made for those types in the first place, and decide whether datetime is
like them /in this respect/. The heterogeneous operations problem may be
a reasonable justification for leaving the other builtins alone but
changing datetime, but if someone knows of other fundamental reasons why
the decision to have arithmetic operations always create the base class
was chosen, please let me know.

Best,
Paul

On 1/5/19 3:55 AM, Alexander Belopolsky wrote:
>
>
> On Wed, Jan 2, 2019 at 10:18 PM Paul Ganssle <paul at ganssle.io
> <mailto:paul at ganssle.io>> wrote:
>
>     .. the original objection was that this implementation assumes
>     that the datetime subclass has a constructor with the same (or a
>     sufficiently similar) signature as datetime.
>
> While this was used as a possible rationale for the way standard types
> behave, the main objection to changing datetime classes is that it
> will make them behave differently from builtins.? For example:
>
> >>> class F(float):
> ...? ? ?pass
> ...
> >>> type(F.fromhex('AA'))
> <class '__main__.F'>
> >>> type(F(1) + F(2))
> <class 'float'>
>
>     This may be a legitimate gripe, but unfortunately that ship has
>     sailed long ago. All of datetime's alternate constructors make
>     this assumption. Any subclass that does not meet this requirement
>     must have worked around it long ago (or they don't care about
>     alternate constructors).
>
>
> This is right, but the same argument is equally applicable to int,
> float, etc. subclasses.? If you want to limit your change to datetime
> types you should explain what makes these types special.??
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/8f7a75a7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/8f7a75a7/attachment.sig>

From rosuav at gmail.com  Sun Jan  6 11:05:27 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 7 Jan 2019 03:05:27 +1100
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <q0t80n$m9u$1@blaine.gmane.org>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
 <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
 <q0t80n$m9u$1@blaine.gmane.org>
Message-ID: <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>

On Mon, Jan 7, 2019 at 2:52 AM Serhiy Storchaka <storchaka at gmail.com> wrote:
>
> 06.01.19 17:35, Chris Angelico ????:
> > On Mon, Jan 7, 2019 at 2:27 AM Serhiy Storchaka <storchaka at gmail.com> wrote:
> >> Because there is a reason for such code.
> >
> > What reason though??
>
> I added references and excerpts in the previous message.

Which I read, and they do not explain the assignment of None. The only
reference is that the tracker issue mentions having a test case to
ensure that it's happening, which is further proof that it's
intentional, but still fails to explain *why*.

ChrisA

From paul at ganssle.io  Sun Jan  6 11:11:42 2019
From: paul at ganssle.io (Paul G)
Date: Sun, 06 Jan 2019 16:11:42 +0000
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
 <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
 <q0t80n$m9u$1@blaine.gmane.org>
 <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>
Message-ID: <5B21C86E-F904-44E8-BC8D-9E17BB31A9F0@ganssle.io>

I think Serhiy is saying that if you delete exc before the except body ends, when the compiled code goes to delete it it will fail. Binding to None guarantees that there is something to delete.

On January 6, 2019 4:05:27 PM UTC, Chris Angelico <rosuav at gmail.com> wrote:
>On Mon, Jan 7, 2019 at 2:52 AM Serhiy Storchaka <storchaka at gmail.com>
>wrote:
>>
>> 06.01.19 17:35, Chris Angelico ????:
>> > On Mon, Jan 7, 2019 at 2:27 AM Serhiy Storchaka
><storchaka at gmail.com> wrote:
>> >> Because there is a reason for such code.
>> >
>> > What reason though??
>>
>> I added references and excerpts in the previous message.
>
>Which I read, and they do not explain the assignment of None. The only
>reference is that the tracker issue mentions having a test case to
>ensure that it's happening, which is further proof that it's
>intentional, but still fails to explain *why*.
>
>ChrisA
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/fd150079/attachment.html>

From rosuav at gmail.com  Sun Jan  6 11:18:00 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 7 Jan 2019 03:18:00 +1100
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <5B21C86E-F904-44E8-BC8D-9E17BB31A9F0@ganssle.io>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
 <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
 <q0t80n$m9u$1@blaine.gmane.org>
 <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>
 <5B21C86E-F904-44E8-BC8D-9E17BB31A9F0@ganssle.io>
Message-ID: <CAPTjJmpsz_+0teVETV6YzZzMZ18rjtA1tVStCGoYxPzZs1jhCg@mail.gmail.com>

On Mon, Jan 7, 2019 at 3:13 AM Paul G <paul at ganssle.io> wrote:
>
> I think Serhiy is saying that if you delete exc before the except body ends, when the compiled code goes to delete it it will fail. Binding to None guarantees that there is something to delete.

Ahh! Thank you. I interpreted that example as a "this is what it
compiles to", rather than actual source code that might fail. Can we
get this example, suitably annotated, into the documentation
somewhere?

ChrisA

From paul at ganssle.io  Sun Jan  6 11:37:34 2019
From: paul at ganssle.io (Paul G)
Date: Sun, 06 Jan 2019 16:37:34 +0000
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <CAPTjJmpsz_+0teVETV6YzZzMZ18rjtA1tVStCGoYxPzZs1jhCg@mail.gmail.com>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
 <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
 <q0t80n$m9u$1@blaine.gmane.org>
 <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>
 <5B21C86E-F904-44E8-BC8D-9E17BB31A9F0@ganssle.io>
 <CAPTjJmpsz_+0teVETV6YzZzMZ18rjtA1tVStCGoYxPzZs1jhCg@mail.gmail.com>
Message-ID: <2457F573-1F15-4312-BBFF-0CB96452FC88@ganssle.io>

I am not familiar enough with the compiler logic, but would it be possible to optimize this away by detecting whether the bound name has already been deleted during the body of the except statement?


On January 6, 2019 4:18:00 PM UTC, Chris Angelico <rosuav at gmail.com> wrote:
>On Mon, Jan 7, 2019 at 3:13 AM Paul G <paul at ganssle.io> wrote:
>>
>> I think Serhiy is saying that if you delete exc before the except
>body ends, when the compiled code goes to delete it it will fail.
>Binding to None guarantees that there is something to delete.
>
>Ahh! Thank you. I interpreted that example as a "this is what it
>compiles to", rather than actual source code that might fail. Can we
>get this example, suitably annotated, into the documentation
>somewhere?
>
>ChrisA
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/eeb269de/attachment.html>

From guido at python.org  Sun Jan  6 13:30:42 2019
From: guido at python.org (Guido van Rossum)
Date: Sun, 6 Jan 2019 11:30:42 -0700
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <2457F573-1F15-4312-BBFF-0CB96452FC88@ganssle.io>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
 <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
 <q0t80n$m9u$1@blaine.gmane.org>
 <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>
 <5B21C86E-F904-44E8-BC8D-9E17BB31A9F0@ganssle.io>
 <CAPTjJmpsz_+0teVETV6YzZzMZ18rjtA1tVStCGoYxPzZs1jhCg@mail.gmail.com>
 <2457F573-1F15-4312-BBFF-0CB96452FC88@ganssle.io>
Message-ID: <CAP7+vJJE6Wx_Fd2_CeTfUtJuUyEaH_42_twHiKL47hRwviHr7Q@mail.gmail.com>

On Sun, Jan 6, 2019 at 9:40 AM Paul G <paul at ganssle.io> wrote:

> I am not familiar enough with the compiler logic, but would it be possible
> to optimize this away by detecting whether the bound name has already been
> deleted during the body of the except statement?
>

As was said before, it's not useful to optimize code that only runs on an
exception. (Though if there was a

On January 6, 2019 4:18:00 PM UTC, Chris Angelico <rosuav at gmail.com> wrote:
>
> On Mon, Jan 7, 2019 at 3:13 AM Paul G <paul at ganssle.io> wrote:
>
>>
>>  I think Serhiy is saying that if you delete exc before the except body ends, when the compiled code goes to delete it it will fail. Binding to None guarantees that there is something to delete.
>>
>
> Ahh! Thank you. I interpreted that example as a "this is what it
> compiles to", rather than actual source code that might fail. Can we
> get this example, suitably annotated, into the documentation
> somewhere?
>
>
I don't see a reason to document this behavior -- it would be strange if it
*didn't* work.

If we had an opcode to delete a variable but didn't raise if it is unset,
we could use that, but otherwise I see no action needed here.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/cd808d6b/attachment.html>

From andrew.svetlov at gmail.com  Sun Jan  6 13:29:55 2019
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sun, 6 Jan 2019 20:29:55 +0200
Subject: [Python-Dev] Return type of datetime subclasses added to
 timedelta
In-Reply-To: <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>
References: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
 <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>
 <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>
Message-ID: <CAL3CFcXCr-tJC==+a4VUav1-A6vsDZu5RMFB=Kyfm0x2yqm+Gg@mail.gmail.com>

>From my perspective datetime classes are even more complex than int/float.
Let's assume we have

class DT(datetime.datetime): ...
class TD(datetime.timedelta): ...

What is the result type for the following expressions?
DT - datetime
DT - DT
DT + TD
DT + timedelta

I have a feeling that the question has no generic answer.
For *particular* implementation you can override all __add__, __sub__
and other arithmetic operations, and you can do it right now with the
current datetime module implementation.

P.S.
I think inheritance from datetime classes is a very rare thing, 99.99% of
users don't need it.


On Sun, Jan 6, 2019 at 6:03 PM Paul Ganssle <paul at ganssle.io> wrote:

> I can think of many reasons why datetime is different from builtins,
> though to be honest I'm not sure that consistency for its own sake is
> really a strong argument for keeping a counter-intuitive behavior - and to
> be honest I'm open to the idea that *all* arithmetic types *should* have
> some form of this change.
>
> That said, I would say that the biggest difference between datetime and
> builtins (other than the fact that datetime is *not* a builtin, and as
> such doesn't necessarily need to be categorized in this group), is that
> unlike almost all other arithmetic types, *datetime* has a special,
> dedicated type for describing differences in datetimes. Using your example
> of a float subclass, consider that without the behavior of "addition of
> floats returns floats", it would be hard to predict what would happen in
> this situation:
>
> >>> F(1.2) + 3.4
>
> Would that always return a float, even though F(1.2) + F(3.4) returns an
> F? Would that return an F because F is the left-hand operand? Would it
> return a float because float is the right-hand operand? Would you walk the
> MROs and find the lowest type in common between the operands and return
> that? It's not entirely clear which subtype predominates. With datetime,
> you have:
>
> datetime - datetime -> timedelta
> datetime ? timedelta -> datetime
> timedelta ? timedelta -> timedelta
>
> There's no operation between two datetime objects that would return a
> datetime object, so it's always clear: operations between datetime
> subclasses return timedelta, operations between a datetime object and a
> timedelta return the subclass of the datetime that it was added to or
> subtracted from.
>
> Of course, the real way to resolve whether datetime should be different
> from int/float/string/etc is to look at why this choice was actually made
> for those types in the first place, and decide whether datetime is like
> them *in this respect*. The heterogeneous operations problem may be a
> reasonable justification for leaving the other builtins alone but changing
> datetime, but if someone knows of other fundamental reasons why the
> decision to have arithmetic operations always create the base class was
> chosen, please let me know.
>
> Best,
> Paul
> On 1/5/19 3:55 AM, Alexander Belopolsky wrote:
>
>
>
> On Wed, Jan 2, 2019 at 10:18 PM Paul Ganssle <paul at ganssle.io> wrote:
>
>> .. the original objection was that this implementation assumes that the
>> datetime subclass has a constructor with the same (or a sufficiently
>> similar) signature as datetime.
>>
> While this was used as a possible rationale for the way standard types
> behave, the main objection to changing datetime classes is that it will
> make them behave differently from builtins.  For example:
>
> >>> class F(float):
> ...     pass
> ...
> >>> type(F.fromhex('AA'))
> <class '__main__.F'>
> >>> type(F(1) + F(2))
> <class 'float'>
>
> This may be a legitimate gripe, but unfortunately that ship has sailed
>> long ago. All of datetime's alternate constructors make this assumption.
>> Any subclass that does not meet this requirement must have worked around it
>> long ago (or they don't care about alternate constructors).
>>
>
> This is right, but the same argument is equally applicable to int, float,
> etc. subclasses.  If you want to limit your change to datetime types you
> should explain what makes these types special.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>


-- 
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/63d736ae/attachment.html>

From guido at python.org  Sun Jan  6 13:43:18 2019
From: guido at python.org (Guido van Rossum)
Date: Sun, 6 Jan 2019 11:43:18 -0700
Subject: [Python-Dev] Return type of datetime subclasses added to
 timedelta
In-Reply-To: <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>
References: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
 <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>
 <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>
Message-ID: <CAP7+vJL3JrRFstg_QNdq5C4kHkxdsbKyACJBxWSPUbyuTjao3g@mail.gmail.com>

I don't think datetime and builtins like int necessarily need to be
aligned. But I do see a problem -- the __new__ and __init__ methods defined
in the subclass (if any) should allow for being called with the same
signature as the base datetime class. Currently you can have a subclass of
datetime whose __new__ has no arguments (or, more realistically, interprets
its arguments differently). Instances of such a class can still be added to
a timedelta. The proposal would cause this to break (since such an addition
has to create a new instance, which calls __new__ and __init__). Since this
is a backwards incompatibility, I don't see how it can be done -- and I
also don't see many use cases, so I think it's not worth pursuing further.

Note that the same problem already happens with the .fromordinal() class
method, though it doesn't happen with .fromdatetime() or .now():

>>> class D(datetime.datetime):
...   def __new__(cls): return cls.now()
...
>>> D()
D(2019, 1, 6, 10, 33, 37, 161606)
>>> D.fromordinal(100)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __new__() takes 1 positional argument but 4 were given
>>> D.fromtimestamp(123456789)
D(1973, 11, 29, 13, 33, 9)
>>>

On Sun, Jan 6, 2019 at 9:05 AM Paul Ganssle <paul at ganssle.io> wrote:

> I can think of many reasons why datetime is different from builtins,
> though to be honest I'm not sure that consistency for its own sake is
> really a strong argument for keeping a counter-intuitive behavior - and to
> be honest I'm open to the idea that *all* arithmetic types *should* have
> some form of this change.
>
> That said, I would say that the biggest difference between datetime and
> builtins (other than the fact that datetime is *not* a builtin, and as
> such doesn't necessarily need to be categorized in this group), is that
> unlike almost all other arithmetic types, *datetime* has a special,
> dedicated type for describing differences in datetimes. Using your example
> of a float subclass, consider that without the behavior of "addition of
> floats returns floats", it would be hard to predict what would happen in
> this situation:
>
> >>> F(1.2) + 3.4
>
> Would that always return a float, even though F(1.2) + F(3.4) returns an
> F? Would that return an F because F is the left-hand operand? Would it
> return a float because float is the right-hand operand? Would you walk the
> MROs and find the lowest type in common between the operands and return
> that? It's not entirely clear which subtype predominates. With datetime,
> you have:
>
> datetime - datetime -> timedelta
> datetime ? timedelta -> datetime
> timedelta ? timedelta -> timedelta
>
> There's no operation between two datetime objects that would return a
> datetime object, so it's always clear: operations between datetime
> subclasses return timedelta, operations between a datetime object and a
> timedelta return the subclass of the datetime that it was added to or
> subtracted from.
>
> Of course, the real way to resolve whether datetime should be different
> from int/float/string/etc is to look at why this choice was actually made
> for those types in the first place, and decide whether datetime is like
> them *in this respect*. The heterogeneous operations problem may be a
> reasonable justification for leaving the other builtins alone but changing
> datetime, but if someone knows of other fundamental reasons why the
> decision to have arithmetic operations always create the base class was
> chosen, please let me know.
>
> Best,
> Paul
> On 1/5/19 3:55 AM, Alexander Belopolsky wrote:
>
>
>
> On Wed, Jan 2, 2019 at 10:18 PM Paul Ganssle <paul at ganssle.io> wrote:
>
>> .. the original objection was that this implementation assumes that the
>> datetime subclass has a constructor with the same (or a sufficiently
>> similar) signature as datetime.
>>
> While this was used as a possible rationale for the way standard types
> behave, the main objection to changing datetime classes is that it will
> make them behave differently from builtins.  For example:
>
> >>> class F(float):
> ...     pass
> ...
> >>> type(F.fromhex('AA'))
> <class '__main__.F'>
> >>> type(F(1) + F(2))
> <class 'float'>
>
> This may be a legitimate gripe, but unfortunately that ship has sailed
>> long ago. All of datetime's alternate constructors make this assumption.
>> Any subclass that does not meet this requirement must have worked around it
>> long ago (or they don't care about alternate constructors).
>>
>
> This is right, but the same argument is equally applicable to int, float,
> etc. subclasses.  If you want to limit your change to datetime types you
> should explain what makes these types special.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/62f9bcb1/attachment.html>

From paul at ganssle.io  Sun Jan  6 13:42:58 2019
From: paul at ganssle.io (Paul Ganssle)
Date: Sun, 6 Jan 2019 13:42:58 -0500
Subject: [Python-Dev] Return type of datetime subclasses added to
 timedelta
In-Reply-To: <CAL3CFcXCr-tJC==+a4VUav1-A6vsDZu5RMFB=Kyfm0x2yqm+Gg@mail.gmail.com>
References: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
 <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>
 <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>
 <CAL3CFcXCr-tJC==+a4VUav1-A6vsDZu5RMFB=Kyfm0x2yqm+Gg@mail.gmail.com>
Message-ID: <e7dab004-f861-93f7-970a-5ec5e97be6fc@ganssle.io>


On 1/6/19 1:29 PM, Andrew Svetlov wrote:
> From my perspective datetime?classes are even more complex than int/float.
> Let's assume we have
>
> class DT(datetime.datetime): ...
> class TD(datetime.timedelta): ...
>
> What is the result type for the following expressions?
> DT - datetime
> DT - DT
> DT?+ TD
> DT?+ timedelta
>
It is not really complicated, the default "difference between two
datetimes" returns a `timedelta`, you can change that by overriding
`__sub__` or `__rsub__` as desired, but there's no reason to think that
the fact that just because DT is a subclass of datetime that it would be
coupled to a specific timedelta subclass *by default*.

Similarly, DT + TD by default will do whatever "datetime" and
"timedelta" do unless you specifically override them. In my proposal,
adding some time to a datetime subclass would return an object of the
datetime subclass, so unless __radd__ or __rsub__ were overriden in
`timedelta`, that's what would happen, the defaults would be (sensibly):

DT - datetime -> timedelta
DT - DT -> timedelta
DT + TD -> DT
DT + timedelta -> timedelta

The only time it would be more complicated is if datetime were defined
like this:

class datetime:
??? TIMEDELTA_CLASS = datetime.timedelta
??? ...

In which case you'd have the same problem you have with float/int/etc
(not a particularly more complicated one. But that's not the case, and
there /is/ one obviously right answer. This is not the case with float
subclasses, because the intuitive rule is "adding together two objects
of the same class gives the same class", which fails when you have two
different subclasses. With datetime, you have "adding a delta type to a
value type returns an object of the value type", which makes perfect
sense, as opposed to "adding a delta type to a value type returns the
base value type, even if the base value type was never used".


> I have a feeling that the question has no generic answer.
> For *particular* implementation you can override all __add__, __sub__
> and other arithmetic operations, and you can do it right now with the
> current datetime?module implementation.
> P.S.
> I think inheritance from datetime?classes is a very rare thing, 99.99%
> of users don't need it.
>
Both of these points are addressed in my original post, IIRC, but both
of these arguments cut both ways. Assuming it's true that this is very
rare - the 0.01% of people who /are/ subclassing datetime either don't
care about this behavior or want timedelta arithmetic to return their
subclass. It's rare enough that there should be no problem giving them
what they want.

Similarly, the rarest group - people who are creating datetime
subclasses /and/ want the original behavior - can simply implement
__add__ and __sub__ to get what they want, so there's no real conflict,
it's just a matter of setting a sane default that also solves the
problem that datetime alternate constructors tend to leak their
implementation details because of the arithmetic return type issue.


Best, Paul

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/e7ce698c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/e7ce698c/attachment-0001.sig>

From storchaka at gmail.com  Sun Jan  6 13:44:23 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 6 Jan 2019 20:44:23 +0200
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
 <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
 <q0t80n$m9u$1@blaine.gmane.org>
 <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>
Message-ID: <q0ti64$cci$1@blaine.gmane.org>

06.01.19 18:05, Chris Angelico ????:
> Which I read, and they do not explain the assignment of None. The only
> reference is that the tracker issue mentions having a test case to
> ensure that it's happening, which is further proof that it's
> intentional, but still fails to explain *why*.

This is explained on a discussion on the Python-3000 mailing list. I 
provided a code example which would fail if do not perform such 
assignment. The test, mentioned in the tracker issue, contains similar code.


From pmiscml at gmail.com  Sun Jan  6 13:48:39 2019
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Sun, 6 Jan 2019 20:48:39 +0200
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <q0t6id$a79$1@blaine.gmane.org>
References: <20190106131048.6e0a3dd8@x230>
	<q0t6id$a79$1@blaine.gmane.org>
Message-ID: <20190106204839.2dccb050@x230>

Hello,

On Sun, 6 Jan 2019 17:26:09 +0200
Serhiy Storchaka <storchaka at gmail.com> wrote:

[]

> Because there is a reason for such code.
> 
> See issue1631942 [1] and the thread with the subject "self-contained 
> exceptions" on the Python-3000 mailing list [2] for the rationale.
> 
> In short, the code
> 
>      try:
>          1/0
>      except Exception as e:
>          del e
> 
> should work.

(Dark) Magic. Live and learn/think. So, we allow user to delete a var,
then recreate it, just to delete again. Thanks for both the hint and
references, Serhiy!

I can only agree with what Chris wrote in the initial reply - it
would nice if there would be an explanation of these tricks somewhere.
I myself wouldn't know a better place than the source code comments.
Well, at least I did trace it to the source code where it's handled.

> I do not see a problem with storing None before deleting a variable. 
> This is not a performance critical code, because it is executed only 
> when an exception was raised and caught.

From CPython's points of view, I might (well, have to) agree. But I was
pretty surprised/disappointed that MicroPython follows the same
routine, I'm glad I now understand ins and outs of the choices made.

> 
> [1] https://bugs.python.org/issue1631942
> [2]
> https://mail.python.org/pipermail/python-3000/2007-January/005294.html


-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From paul at ganssle.io  Sun Jan  6 13:59:06 2019
From: paul at ganssle.io (Paul Ganssle)
Date: Sun, 6 Jan 2019 13:59:06 -0500
Subject: [Python-Dev] Return type of datetime subclasses added to
 timedelta
In-Reply-To: <CAP7+vJL3JrRFstg_QNdq5C4kHkxdsbKyACJBxWSPUbyuTjao3g@mail.gmail.com>
References: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
 <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>
 <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>
 <CAP7+vJL3JrRFstg_QNdq5C4kHkxdsbKyACJBxWSPUbyuTjao3g@mail.gmail.com>
Message-ID: <2415dd60-b6b4-30b0-90d2-c0c8b22314c7@ganssle.io>

I did address this in the original post - the assumption that the
subclass constructor will have the same arguments as the base
constructor is baked into many alternate constructors of datetime. I
acknowledge that this is a breaking change, but it is a small one -
anyone creating such a subclass that /cannot/ handled the class being
created this way would be broken in myriad ways.

We have also in recent years changed several alternate constructors
(including `replace`) to retain the original subclass, which by your
same standard would be a breaking change. I believe there have been no
complaints. In fact, between Python 3.6 and 3.7, the very example you
showed broke:

Python 3.6.6:

>>> class D(datetime.datetime):
...???? def __new__(cls):
...???????? return cls.now()
...
>>> D()
D(2019, 1, 6, 13, 49, 38, 842033)

Python 3.7.2:

>>> class D(datetime.datetime):
...???? def __new__(cls):
...???????? return cls.now()
...
>>> D()
Traceback (most recent call last):
? File "<stdin>", line 1, in <module>
? File "<stdin>", line 3, in __new__
TypeError: __new__() takes 1 positional argument but 9 were given


We haven't seen any bug reports about this sort of thing; what we /have/
been getting is bug reports that subclassing datetime doesn't retain the
subclass in various ways (because people /are/ using datetime
subclasses). This is likely to cause very little in the way of problems,
but it will improve convenience for people making datetime subclasses
and almost certainly performance for people using them (e.g. pendulum
and arrow, which now need to take a slow pure python route in many
situations to work around this problem).

If we're /really/ concerned with this backward compatibility breaking,
we could do the equivalent of:

try:
??? return new_behavior(...)
except TypeError:
??? warnings.warn("The semantics of timedelta addition have "
????????????????? "changed in a way that raises an error in "
????????????????? "this subclass. Please implement __add__ "
????????????????? "if you need the old behavior.", DeprecationWarning)

Then after a suitable notice period drop the warning and turn it to a
hard error.

Best,

Paul

On 1/6/19 1:43 PM, Guido van Rossum wrote:
> I don't think datetime and builtins like int necessarily need to be
> aligned. But I do see a problem -- the __new__ and __init__ methods
> defined in the subclass (if any) should allow for being called with
> the same signature as the base datetime class. Currently you can have
> a subclass of datetime whose __new__ has no arguments (or, more
> realistically, interprets its arguments differently). Instances of
> such a class can still be added to a timedelta. The proposal would
> cause this to break (since such an addition has to create a new
> instance, which calls __new__ and __init__). Since this is a backwards
> incompatibility, I don't see how it can be done -- and I also don't
> see many use cases, so I think it's not worth pursuing further.
>
> Note that the same problem already happens with the .fromordinal()
> class method, though it doesn't happen with .fromdatetime() or .now():
>
> >>> class D(datetime.datetime):
> ...?? def __new__(cls): return cls.now()
> ...
> >>> D()
> D(2019, 1, 6, 10, 33, 37, 161606)
> >>> D.fromordinal(100)
> Traceback (most recent call last):
> ? File "<stdin>", line 1, in <module>
> TypeError: __new__() takes 1 positional argument but 4 were given
> >>> D.fromtimestamp(123456789)
> D(1973, 11, 29, 13, 33, 9)
> >>>
>
> On Sun, Jan 6, 2019 at 9:05 AM Paul Ganssle <paul at ganssle.io
> <mailto:paul at ganssle.io>> wrote:
>
>     I can think of many reasons why datetime is different from
>     builtins, though to be honest I'm not sure that consistency for
>     its own sake is really a strong argument for keeping a
>     counter-intuitive behavior - and to be honest I'm open to the idea
>     that /all/ arithmetic types /should/ have some form of this change.
>
>     That said, I would say that the biggest difference between
>     datetime and builtins (other than the fact that datetime is /not/
>     a builtin, and as such doesn't necessarily need to be categorized
>     in this group), is that unlike almost all other arithmetic types,
>     /datetime/ has a special, dedicated type for describing
>     differences in datetimes. Using your example of a float subclass,
>     consider that without the behavior of "addition of floats returns
>     floats", it would be hard to predict what would happen in this
>     situation:
>
>     >>> F(1.2) + 3.4
>
>     Would that always return a float, even though F(1.2) + F(3.4)
>     returns an F? Would that return an F because F is the left-hand
>     operand? Would it return a float because float is the right-hand
>     operand? Would you walk the MROs and find the lowest type in
>     common between the operands and return that? It's not entirely
>     clear which subtype predominates. With datetime, you have:
>
>     datetime - datetime -> timedelta
>     datetime ? timedelta -> datetime
>     timedelta ? timedelta -> timedelta
>
>     There's no operation between two datetime objects that would
>     return a datetime object, so it's always clear: operations between
>     datetime subclasses return timedelta, operations between a
>     datetime object and a timedelta return the subclass of the
>     datetime that it was added to or subtracted from.
>
>     Of course, the real way to resolve whether datetime should be
>     different from int/float/string/etc is to look at why this choice
>     was actually made for those types in the first place, and decide
>     whether datetime is like them /in this respect/. The heterogeneous
>     operations problem may be a reasonable justification for leaving
>     the other builtins alone but changing datetime, but if someone
>     knows of other fundamental reasons why the decision to have
>     arithmetic operations always create the base class was chosen,
>     please let me know.
>
>     Best,
>     Paul
>
>     On 1/5/19 3:55 AM, Alexander Belopolsky wrote:
>>
>>
>>     On Wed, Jan 2, 2019 at 10:18 PM Paul Ganssle <paul at ganssle.io
>>     <mailto:paul at ganssle.io>> wrote:
>>
>>         .. the original objection was that this implementation
>>         assumes that the datetime subclass has a constructor with the
>>         same (or a sufficiently similar) signature as datetime.
>>
>>     While this was used as a possible rationale for the way standard
>>     types behave, the main objection to changing datetime classes is
>>     that it will make them behave differently from builtins.? For
>>     example:
>>
>>     >>> class F(float):
>>     ...? ? ?pass
>>     ...
>>     >>> type(F.fromhex('AA'))
>>     <class '__main__.F'>
>>     >>> type(F(1) + F(2))
>>     <class 'float'>
>>
>>         This may be a legitimate gripe, but unfortunately that ship
>>         has sailed long ago. All of datetime's alternate constructors
>>         make this assumption. Any subclass that does not meet this
>>         requirement must have worked around it long ago (or they
>>         don't care about alternate constructors).
>>
>>
>>     This is right, but the same argument is equally applicable to
>>     int, float, etc. subclasses.? If you want to limit your change to
>>     datetime types you should explain what makes these types special.??
>     _______________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     https://mail.python.org/mailman/listinfo/python-dev
>     Unsubscribe:
>     https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> -- 
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/a8b25999/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/a8b25999/attachment.sig>

From pmiscml at gmail.com  Sun Jan  6 15:11:45 2019
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Sun, 6 Jan 2019 22:11:45 +0200
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <20190106141453.GY13616@ando.pearwood.info>
References: <20190106131048.6e0a3dd8@x230>
 <20190106121011.GX13616@ando.pearwood.info>
 <20190106150303.57b77c5a@x230>
 <20190106141453.GY13616@ando.pearwood.info>
Message-ID: <20190106221145.43021319@x230>

Hello,

On Mon, 7 Jan 2019 01:14:55 +1100
Steven D'Aprano <steve at pearwood.info> wrote:

[]

> > Thanks, that summarizes it well. And well, my interest is also how
> > non-compliant would be for another Python implementation to act
> > differently, specifically to skip wrapping an except handler body in
> > try-finally (i.e., go back to Python2 behavior). I'm keen to add
> > such an option to my fork of MicroPython.  
> 
> Wouldn't that mean that MicroPython suffers from the
> exception/traceback reference cycle problem? How do you propose to
> solve that?

To answer this, MicroPython doesn't have reference counting and relies
solely on garbage collection, so breaking cycles early wouldn't have
much effect. Breaking stray links to should-be-dead data structures
might (to allow them to be collected sooner rather than later), but
probably not at the expense of introducing extra entry in the block
stack. (That may seems like breadcrumb saving, but saving a few
percents of overhead in many places is how MicroPython achieves
its promise of programming small embedded systems in a very high level
language.)

[]

-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From brett at python.org  Sun Jan  6 15:17:47 2019
From: brett at python.org (Brett Cannon)
Date: Sun, 6 Jan 2019 12:17:47 -0800
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <CAP7+vJJE6Wx_Fd2_CeTfUtJuUyEaH_42_twHiKL47hRwviHr7Q@mail.gmail.com>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
 <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
 <q0t80n$m9u$1@blaine.gmane.org>
 <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>
 <5B21C86E-F904-44E8-BC8D-9E17BB31A9F0@ganssle.io>
 <CAPTjJmpsz_+0teVETV6YzZzMZ18rjtA1tVStCGoYxPzZs1jhCg@mail.gmail.com>
 <2457F573-1F15-4312-BBFF-0CB96452FC88@ganssle.io>
 <CAP7+vJJE6Wx_Fd2_CeTfUtJuUyEaH_42_twHiKL47hRwviHr7Q@mail.gmail.com>
Message-ID: <CAP1=2W4LvbgutAfsuB=XCxeD1R8Jf9LYsiC+ORaP81UAfRhocA@mail.gmail.com>

On Sun, 6 Jan 2019 at 10:26, Guido van Rossum <guido at python.org> wrote:

> On Sun, Jan 6, 2019 at 9:40 AM Paul G <paul at ganssle.io> wrote:
>
>> I am not familiar enough with the compiler logic, but would it be
>> possible to optimize this away by detecting whether the bound name has
>> already been deleted during the body of the except statement?
>>
>
> As was said before, it's not useful to optimize code that only runs on an
> exception. (Though if there was a
>
> On January 6, 2019 4:18:00 PM UTC, Chris Angelico <rosuav at gmail.com>
> wrote:
>>
>> On Mon, Jan 7, 2019 at 3:13 AM Paul G <paul at ganssle.io> wrote:
>>
>>>
>>>  I think Serhiy is saying that if you delete exc before the except body ends, when the compiled code goes to delete it it will fail. Binding to None guarantees that there is something to delete.
>>>
>>
>> Ahh! Thank you. I interpreted that example as a "this is what it
>> compiles to", rather than actual source code that might fail. Can we
>> get this example, suitably annotated, into the documentation
>> somewhere?
>>
>>
> I don't see a reason to document this behavior -- it would be strange if
> it *didn't* work.
>
> If we had an opcode to delete a variable but didn't raise if it is unset,
> we could use that, but otherwise I see no action needed here.
>

Maybe someone can propose a code comment to explain this instead? I agree
that this doesn't need to be explained in the language spec, but I don't
see any harm in a comment where the opcodes are emitted to explain why it
is the way it is .
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/df49faca/attachment.html>

From rosuav at gmail.com  Sun Jan  6 15:22:19 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 7 Jan 2019 07:22:19 +1100
Subject: [Python-Dev] Compilation of "except FooExc as var" adds useless
 store
In-Reply-To: <CAP1=2W4LvbgutAfsuB=XCxeD1R8Jf9LYsiC+ORaP81UAfRhocA@mail.gmail.com>
References: <20190106131048.6e0a3dd8@x230> <q0t6id$a79$1@blaine.gmane.org>
 <CAPTjJmoUbOH_8CRaUeUQ2mKBQPH_29WRZBrxidaOkYfT6Jt+2Q@mail.gmail.com>
 <q0t80n$m9u$1@blaine.gmane.org>
 <CAPTjJmr7EvUemFakEMto8fCO_5AbaW-ebqNbqQq3U=BJ3Xkqdg@mail.gmail.com>
 <5B21C86E-F904-44E8-BC8D-9E17BB31A9F0@ganssle.io>
 <CAPTjJmpsz_+0teVETV6YzZzMZ18rjtA1tVStCGoYxPzZs1jhCg@mail.gmail.com>
 <2457F573-1F15-4312-BBFF-0CB96452FC88@ganssle.io>
 <CAP7+vJJE6Wx_Fd2_CeTfUtJuUyEaH_42_twHiKL47hRwviHr7Q@mail.gmail.com>
 <CAP1=2W4LvbgutAfsuB=XCxeD1R8Jf9LYsiC+ORaP81UAfRhocA@mail.gmail.com>
Message-ID: <CAPTjJmrPoeGXdM4dbE0YpnXb4m5UR98-vcTsKgOzor+eT9uihQ@mail.gmail.com>

On Mon, Jan 7, 2019 at 7:18 AM Brett Cannon <brett at python.org> wrote:
>
> Maybe someone can propose a code comment to explain this instead? I agree that this doesn't need to be explained in the language spec, but I don't see any harm in a comment where the opcodes are emitted to explain why it is the way it is .
>

No BPO as yet but here's a tracker issue to get started.

https://github.com/python/cpython/pull/11448

ChrisA

From brett at python.org  Sun Jan  6 15:24:54 2019
From: brett at python.org (Brett Cannon)
Date: Sun, 6 Jan 2019 12:24:54 -0800
Subject: [Python-Dev] Return type of datetime subclasses added to
 timedelta
In-Reply-To: <2415dd60-b6b4-30b0-90d2-c0c8b22314c7@ganssle.io>
References: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
 <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>
 <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>
 <CAP7+vJL3JrRFstg_QNdq5C4kHkxdsbKyACJBxWSPUbyuTjao3g@mail.gmail.com>
 <2415dd60-b6b4-30b0-90d2-c0c8b22314c7@ganssle.io>
Message-ID: <CAP1=2W7RnpW7HgpuDQPzZzfrMMzPXAGj8G91xQyRMU4qLV1btg@mail.gmail.com>

On Sun, 6 Jan 2019 at 11:00, Paul Ganssle <paul at ganssle.io> wrote:

> I did address this in the original post - the assumption that the subclass
> constructor will have the same arguments as the base constructor is baked
> into many alternate constructors of datetime. I acknowledge that this is a
> breaking change, but it is a small one - anyone creating such a subclass
> that *cannot* handled the class being created this way would be broken in
> myriad ways.
>
> We have also in recent years changed several alternate constructors
> (including `replace`) to retain the original subclass, which by your same
> standard would be a breaking change. I believe there have been no
> complaints. In fact, between Python 3.6 and 3.7, the very example you
> showed broke:
>
> Python 3.6.6:
>
> >>> class D(datetime.datetime):
> ...     def __new__(cls):
> ...         return cls.now()
> ...
> >>> D()
> D(2019, 1, 6, 13, 49, 38, 842033)
>
> Python 3.7.2:
>
> >>> class D(datetime.datetime):
> ...     def __new__(cls):
> ...         return cls.now()
> ...
> >>> D()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 3, in __new__
> TypeError: __new__() takes 1 positional argument but 9 were given
>
>
> We haven't seen any bug reports about this sort of thing; what we *have*
> been getting is bug reports that subclassing datetime doesn't retain the
> subclass in various ways (because people *are* using datetime subclasses).
>

To help set expectations, the current semantics are not a bug and so the
proposal isn't fixing a bug but proposing a change in semantics.


> This is likely to cause very little in the way of problems, but it will
> improve convenience for people making datetime subclasses and almost
> certainly performance for people using them (e.g. pendulum and arrow, which
> now need to take a slow pure python route in many situations to work around
> this problem).
>
> If we're *really* concerned with this backward compatibility breaking,
>

We very much do care. Because this isn't a bug but a voluntary semantic
change you're proposing to change we can't blindly break people who are
relying on the current semantics. We need to have a justification for those
people as to why we have decided to change the semantics now after all of
these years as well as provide an upgrade path.

-Brett


> we could do the equivalent of:
>
> try:
>     return new_behavior(...)
> except TypeError:
>     warnings.warn("The semantics of timedelta addition have "
>                   "changed in a way that raises an error in "
>                   "this subclass. Please implement __add__ "
>                   "if you need the old behavior.", DeprecationWarning)
>

> Then after a suitable notice period drop the warning and turn it to a hard
> error.
>
> Best,
>
> Paul
> On 1/6/19 1:43 PM, Guido van Rossum wrote:
>
> I don't think datetime and builtins like int necessarily need to be
> aligned. But I do see a problem -- the __new__ and __init__ methods defined
> in the subclass (if any) should allow for being called with the same
> signature as the base datetime class. Currently you can have a subclass of
> datetime whose __new__ has no arguments (or, more realistically, interprets
> its arguments differently). Instances of such a class can still be added to
> a timedelta. The proposal would cause this to break (since such an addition
> has to create a new instance, which calls __new__ and __init__). Since this
> is a backwards incompatibility, I don't see how it can be done -- and I
> also don't see many use cases, so I think it's not worth pursuing further.
>
> Note that the same problem already happens with the .fromordinal() class
> method, though it doesn't happen with .fromdatetime() or .now():
>
> >>> class D(datetime.datetime):
> ...   def __new__(cls): return cls.now()
> ...
> >>> D()
> D(2019, 1, 6, 10, 33, 37, 161606)
> >>> D.fromordinal(100)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: __new__() takes 1 positional argument but 4 were given
> >>> D.fromtimestamp(123456789)
> D(1973, 11, 29, 13, 33, 9)
> >>>
>
> On Sun, Jan 6, 2019 at 9:05 AM Paul Ganssle <paul at ganssle.io> wrote:
>
>> I can think of many reasons why datetime is different from builtins,
>> though to be honest I'm not sure that consistency for its own sake is
>> really a strong argument for keeping a counter-intuitive behavior - and to
>> be honest I'm open to the idea that *all* arithmetic types *should* have
>> some form of this change.
>>
>> That said, I would say that the biggest difference between datetime and
>> builtins (other than the fact that datetime is *not* a builtin, and as
>> such doesn't necessarily need to be categorized in this group), is that
>> unlike almost all other arithmetic types, *datetime* has a special,
>> dedicated type for describing differences in datetimes. Using your example
>> of a float subclass, consider that without the behavior of "addition of
>> floats returns floats", it would be hard to predict what would happen in
>> this situation:
>>
>> >>> F(1.2) + 3.4
>>
>> Would that always return a float, even though F(1.2) + F(3.4) returns an
>> F? Would that return an F because F is the left-hand operand? Would it
>> return a float because float is the right-hand operand? Would you walk the
>> MROs and find the lowest type in common between the operands and return
>> that? It's not entirely clear which subtype predominates. With datetime,
>> you have:
>>
>> datetime - datetime -> timedelta
>> datetime ? timedelta -> datetime
>> timedelta ? timedelta -> timedelta
>>
>> There's no operation between two datetime objects that would return a
>> datetime object, so it's always clear: operations between datetime
>> subclasses return timedelta, operations between a datetime object and a
>> timedelta return the subclass of the datetime that it was added to or
>> subtracted from.
>>
>> Of course, the real way to resolve whether datetime should be different
>> from int/float/string/etc is to look at why this choice was actually made
>> for those types in the first place, and decide whether datetime is like
>> them *in this respect*. The heterogeneous operations problem may be a
>> reasonable justification for leaving the other builtins alone but changing
>> datetime, but if someone knows of other fundamental reasons why the
>> decision to have arithmetic operations always create the base class was
>> chosen, please let me know.
>>
>> Best,
>> Paul
>> On 1/5/19 3:55 AM, Alexander Belopolsky wrote:
>>
>>
>>
>> On Wed, Jan 2, 2019 at 10:18 PM Paul Ganssle <paul at ganssle.io> wrote:
>>
>>> .. the original objection was that this implementation assumes that the
>>> datetime subclass has a constructor with the same (or a sufficiently
>>> similar) signature as datetime.
>>>
>> While this was used as a possible rationale for the way standard types
>> behave, the main objection to changing datetime classes is that it will
>> make them behave differently from builtins.  For example:
>>
>> >>> class F(float):
>> ...     pass
>> ...
>> >>> type(F.fromhex('AA'))
>> <class '__main__.F'>
>> >>> type(F(1) + F(2))
>> <class 'float'>
>>
>> This may be a legitimate gripe, but unfortunately that ship has sailed
>>> long ago. All of datetime's alternate constructors make this assumption.
>>> Any subclass that does not meet this requirement must have worked around it
>>> long ago (or they don't care about alternate constructors).
>>>
>>
>> This is right, but the same argument is equally applicable to int, float,
>> etc. subclasses.  If you want to limit your change to datetime types you
>> should explain what makes these types special.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/d2e2d630/attachment.html>

From paul at ganssle.io  Sun Jan  6 15:50:32 2019
From: paul at ganssle.io (Paul Ganssle)
Date: Sun, 6 Jan 2019 15:50:32 -0500
Subject: [Python-Dev] Return type of datetime subclasses added to
 timedelta
In-Reply-To: <CAP1=2W7RnpW7HgpuDQPzZzfrMMzPXAGj8G91xQyRMU4qLV1btg@mail.gmail.com>
References: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
 <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>
 <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>
 <CAP7+vJL3JrRFstg_QNdq5C4kHkxdsbKyACJBxWSPUbyuTjao3g@mail.gmail.com>
 <2415dd60-b6b4-30b0-90d2-c0c8b22314c7@ganssle.io>
 <CAP1=2W7RnpW7HgpuDQPzZzfrMMzPXAGj8G91xQyRMU4qLV1btg@mail.gmail.com>
Message-ID: <ba6bdb89-9a60-e46f-8037-7b013a478db3@ganssle.io>

Brett,

Thank you for bringing this up, but I think you /may/ have misunderstood
my position - though maybe you understood the thrust and wanted to
clarify for people coming in halfway, which I applaud.

I proposed this change /knowing/ that it was a breaking change - it's
why I brought it to the attention of datetime-SIG and now python-dev -
and I believe that there are several factors that lead this to being a
smaller compatibility problem than it seems.

One such factor is the fact that /many/ other features of `datetime`,
including the implementation of `datetime.now()` are /already broken/ in
the current implementation for anyone who would be broken by this
particular aspect of the semantic change. That is not saying that it's
impossible that there is code out there that will break if this change
goes through, it's just saying that the scope of the breakage is
necessarily very limited.

The reason I brought up the bug tracker is because between Python 3.6
and Python 3.7, we in fact made a similar breaking change to the one I'm
proposing here without thinking that anyone might be relying on the fact
that they could do something like:

class D(datetime.datetime):
??? def __new__(cls):
??????? return cls.now()

My point was that there have been no bug reports about the /existing
change/ that Guido was bringing up (his example itself does not work on
Python 3.7!), which leads me to believe that few if any people are
relying on the fact that it is possible to define a datetime subclass
with a different default constructor.

As I mentioned, it is likely possible to have a transition period where
this would still work even if the subclassers have not created their own
__add__ method.

There is no way to create a similar deprecation/transition period for
people relying on the fact that `type(datetime_obj + timedelta_obj) ==
datetime.datetime`, but I think this is honestly a sufficiently minor
breakage that the good outweighs the harm. I will note that we have
already made several such changes with respect to alternate constructors
even though technically someone could have been relying on the fact that
`MyDateTime(*args).replace(month=3)` returns a `datetime` object.

This is not to say that we should lightly make the change (hence my
canvassing for opinions), it is just that there is a good amount of
evidence that, practically speaking, no one is relying on this, and in
fact it is likely that people are writing code that assumes that adding
`timedelta` to a datetime subclass returns the original subclass, either
directly or indirectly - I think we're likely to fix more people than we
break if we make this change.

Best,
Paul


On 1/6/19 3:24 PM, Brett Cannon wrote:
>
>
> On Sun, 6 Jan 2019 at 11:00, Paul Ganssle <paul at ganssle.io
> <mailto:paul at ganssle.io>> wrote:
>
>     I did address this in the original post - the assumption that the
>     subclass constructor will have the same arguments as the base
>     constructor is baked into many alternate constructors of datetime.
>     I acknowledge that this is a breaking change, but it is a small
>     one - anyone creating such a subclass that /cannot/ handled the
>     class being created this way would be broken in myriad ways.
>
>     We have also in recent years changed several alternate
>     constructors (including `replace`) to retain the original
>     subclass, which by your same standard would be a breaking change.
>     I believe there have been no complaints. In fact, between Python
>     3.6 and 3.7, the very example you showed broke:
>
>     Python 3.6.6:
>
>     >>> class D(datetime.datetime):
>     ...???? def __new__(cls):
>     ...???????? return cls.now()
>     ...
>     >>> D()
>     D(2019, 1, 6, 13, 49, 38, 842033)
>
>     Python 3.7.2:
>
>     >>> class D(datetime.datetime):
>     ...???? def __new__(cls):
>     ...???????? return cls.now()
>     ...
>     >>> D()
>     Traceback (most recent call last):
>     ? File "<stdin>", line 1, in <module>
>     ? File "<stdin>", line 3, in __new__
>     TypeError: __new__() takes 1 positional argument but 9 were given
>
>
>     We haven't seen any bug reports about this sort of thing; what we
>     /have/ been getting is bug reports that subclassing datetime
>     doesn't retain the subclass in various ways (because people /are/
>     using datetime subclasses).
>
>
> To help set expectations, the current semantics are not a bug and so
> the proposal isn't fixing a bug but proposing a change in semantics.
> ?
>
>     This is likely to cause very little in the way of problems, but it
>     will improve convenience for people making datetime subclasses and
>     almost certainly performance for people using them (e.g. pendulum
>     and arrow, which now need to take a slow pure python route in many
>     situations to work around this problem).
>
>     If we're /really/ concerned with this backward compatibility
>     breaking,
>
>
> We very much do care. Because this isn't a bug but a voluntary
> semantic change you're proposing to change we can't blindly break
> people who are relying on the current semantics. We need to have a
> justification for those people as to why we have decided to change the
> semantics now after all of these years as well as provide an upgrade path.
>
> -Brett
> ?
>
>     we could do the equivalent of:
>
>     try:
>     ??? return new_behavior(...)
>     except TypeError:
>     ??? warnings.warn("The semantics of timedelta addition have "
>     ????????????????? "changed in a way that raises an error in "
>     ????????????????? "this subclass. Please implement __add__ "
>     ????????????????? "if you need the old behavior.",
>     DeprecationWarning)?
>
>
>     Then after a suitable notice period drop the warning and turn it
>     to a hard error.
>
>     Best,
>
>     Paul
>
>     On 1/6/19 1:43 PM, Guido van Rossum wrote:
>>     I don't think datetime and builtins like int necessarily need to
>>     be aligned. But I do see a problem -- the __new__ and __init__
>>     methods defined in the subclass (if any) should allow for being
>>     called with the same signature as the base datetime class.
>>     Currently you can have a subclass of datetime whose __new__ has
>>     no arguments (or, more realistically, interprets its arguments
>>     differently). Instances of such a class can still be added to a
>>     timedelta. The proposal would cause this to break (since such an
>>     addition has to create a new instance, which calls __new__ and
>>     __init__). Since this is a backwards incompatibility, I don't see
>>     how it can be done -- and I also don't see many use cases, so I
>>     think it's not worth pursuing further.
>>
>>     Note that the same problem already happens with the
>>     .fromordinal() class method, though it doesn't happen with
>>     .fromdatetime() or .now():
>>
>>     >>> class D(datetime.datetime):
>>     ...?? def __new__(cls): return cls.now()
>>     ...
>>     >>> D()
>>     D(2019, 1, 6, 10, 33, 37, 161606)
>>     >>> D.fromordinal(100)
>>     Traceback (most recent call last):
>>     ? File "<stdin>", line 1, in <module>
>>     TypeError: __new__() takes 1 positional argument but 4 were given
>>     >>> D.fromtimestamp(123456789)
>>     D(1973, 11, 29, 13, 33, 9)
>>     >>>
>>
>>     On Sun, Jan 6, 2019 at 9:05 AM Paul Ganssle <paul at ganssle.io
>>     <mailto:paul at ganssle.io>> wrote:
>>
>>         I can think of many reasons why datetime is different from
>>         builtins, though to be honest I'm not sure that consistency
>>         for its own sake is really a strong argument for keeping a
>>         counter-intuitive behavior - and to be honest I'm open to the
>>         idea that /all/ arithmetic types /should/ have some form of
>>         this change.
>>
>>         That said, I would say that the biggest difference between
>>         datetime and builtins (other than the fact that datetime is
>>         /not/ a builtin, and as such doesn't necessarily need to be
>>         categorized in this group), is that unlike almost all other
>>         arithmetic types, /datetime/ has a special, dedicated type
>>         for describing differences in datetimes. Using your example
>>         of a float subclass, consider that without the behavior of
>>         "addition of floats returns floats", it would be hard to
>>         predict what would happen in this situation:
>>
>>         >>> F(1.2) + 3.4
>>
>>         Would that always return a float, even though F(1.2) + F(3.4)
>>         returns an F? Would that return an F because F is the
>>         left-hand operand? Would it return a float because float is
>>         the right-hand operand? Would you walk the MROs and find the
>>         lowest type in common between the operands and return that?
>>         It's not entirely clear which subtype predominates. With
>>         datetime, you have:
>>
>>         datetime - datetime -> timedelta
>>         datetime ? timedelta -> datetime
>>         timedelta ? timedelta -> timedelta
>>
>>         There's no operation between two datetime objects that would
>>         return a datetime object, so it's always clear: operations
>>         between datetime subclasses return timedelta, operations
>>         between a datetime object and a timedelta return the subclass
>>         of the datetime that it was added to or subtracted from.
>>
>>         Of course, the real way to resolve whether datetime should be
>>         different from int/float/string/etc is to look at why this
>>         choice was actually made for those types in the first place,
>>         and decide whether datetime is like them /in this respect/.
>>         The heterogeneous operations problem may be a reasonable
>>         justification for leaving the other builtins alone but
>>         changing datetime, but if someone knows of other fundamental
>>         reasons why the decision to have arithmetic operations always
>>         create the base class was chosen, please let me know.
>>
>>         Best,
>>         Paul
>>
>>         On 1/5/19 3:55 AM, Alexander Belopolsky wrote:
>>>
>>>
>>>         On Wed, Jan 2, 2019 at 10:18 PM Paul Ganssle
>>>         <paul at ganssle.io <mailto:paul at ganssle.io>> wrote:
>>>
>>>             .. the original objection was that this implementation
>>>             assumes that the datetime subclass has a constructor
>>>             with the same (or a sufficiently similar) signature as
>>>             datetime.
>>>
>>>         While this was used as a possible rationale for the way
>>>         standard types behave, the main objection to changing
>>>         datetime classes is that it will make them behave
>>>         differently from builtins.? For example:
>>>
>>>         >>> class F(float):
>>>         ...? ? ?pass
>>>         ...
>>>         >>> type(F.fromhex('AA'))
>>>         <class '__main__.F'>
>>>         >>> type(F(1) + F(2))
>>>         <class 'float'>
>>>
>>>             This may be a legitimate gripe, but unfortunately that
>>>             ship has sailed long ago. All of datetime's alternate
>>>             constructors make this assumption. Any subclass that
>>>             does not meet this requirement must have worked around
>>>             it long ago (or they don't care about alternate
>>>             constructors).
>>>
>>>
>>>         This is right, but the same argument is equally applicable
>>>         to int, float, etc. subclasses.? If you want to limit your
>>>         change to datetime types you should explain what makes these
>>>         types special.??
>>         _______________________________________________
>>         Python-Dev mailing list
>>         Python-Dev at python.org <mailto:Python-Dev at python.org>
>>         https://mail.python.org/mailman/listinfo/python-dev
>>         Unsubscribe:
>>         https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>>
>>     -- 
>>     --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>     _______________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     https://mail.python.org/mailman/listinfo/python-dev
>     Unsubscribe:
>     https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/3463db7e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/3463db7e/attachment-0001.sig>

From guido at python.org  Sun Jan  6 19:17:14 2019
From: guido at python.org (Guido van Rossum)
Date: Sun, 6 Jan 2019 16:17:14 -0800
Subject: [Python-Dev] Return type of datetime subclasses added to
 timedelta
In-Reply-To: <2415dd60-b6b4-30b0-90d2-c0c8b22314c7@ganssle.io>
References: <1059740e-cc65-205d-5986-a9397463a315@ganssle.io>
 <CAP7h-xa-Dp8TXrm7Oa=jDwf0wyBfVy-6zK08w219q0jVaYACiQ@mail.gmail.com>
 <bb5120e0-159e-106d-04c6-2a15f7398966@ganssle.io>
 <CAP7+vJL3JrRFstg_QNdq5C4kHkxdsbKyACJBxWSPUbyuTjao3g@mail.gmail.com>
 <2415dd60-b6b4-30b0-90d2-c0c8b22314c7@ganssle.io>
Message-ID: <CAP7+vJKbQeTgROAtNLvXMwK5xjO8V=pCCs-CHpGAnt+G5TZcyg@mail.gmail.com>

OK, I concede your point (and indeed I only tested this on 3.6). If we
could break the backward compatibility for now() we presumably can break it
for this purpose.

On Sun, Jan 6, 2019 at 11:02 AM Paul Ganssle <paul at ganssle.io> wrote:

> I did address this in the original post - the assumption that the subclass
> constructor will have the same arguments as the base constructor is baked
> into many alternate constructors of datetime. I acknowledge that this is a
> breaking change, but it is a small one - anyone creating such a subclass
> that *cannot* handled the class being created this way would be broken in
> myriad ways.
>
> We have also in recent years changed several alternate constructors
> (including `replace`) to retain the original subclass, which by your same
> standard would be a breaking change. I believe there have been no
> complaints. In fact, between Python 3.6 and 3.7, the very example you
> showed broke:
>
> Python 3.6.6:
>
> >>> class D(datetime.datetime):
> ...     def __new__(cls):
> ...         return cls.now()
> ...
> >>> D()
> D(2019, 1, 6, 13, 49, 38, 842033)
>
> Python 3.7.2:
>
> >>> class D(datetime.datetime):
> ...     def __new__(cls):
> ...         return cls.now()
> ...
> >>> D()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 3, in __new__
> TypeError: __new__() takes 1 positional argument but 9 were given
>
>
> We haven't seen any bug reports about this sort of thing; what we *have*
> been getting is bug reports that subclassing datetime doesn't retain the
> subclass in various ways (because people *are* using datetime
> subclasses). This is likely to cause very little in the way of problems,
> but it will improve convenience for people making datetime subclasses and
> almost certainly performance for people using them (e.g. pendulum and
> arrow, which now need to take a slow pure python route in many situations
> to work around this problem).
>
> If we're *really* concerned with this backward compatibility breaking, we
> could do the equivalent of:
>
> try:
>     return new_behavior(...)
> except TypeError:
>     warnings.warn("The semantics of timedelta addition have "
>                   "changed in a way that raises an error in "
>                   "this subclass. Please implement __add__ "
>                   "if you need the old behavior.", DeprecationWarning)
>
> Then after a suitable notice period drop the warning and turn it to a hard
> error.
>
> Best,
>
> Paul
> On 1/6/19 1:43 PM, Guido van Rossum wrote:
>
> I don't think datetime and builtins like int necessarily need to be
> aligned. But I do see a problem -- the __new__ and __init__ methods defined
> in the subclass (if any) should allow for being called with the same
> signature as the base datetime class. Currently you can have a subclass of
> datetime whose __new__ has no arguments (or, more realistically, interprets
> its arguments differently). Instances of such a class can still be added to
> a timedelta. The proposal would cause this to break (since such an addition
> has to create a new instance, which calls __new__ and __init__). Since this
> is a backwards incompatibility, I don't see how it can be done -- and I
> also don't see many use cases, so I think it's not worth pursuing further.
>
> Note that the same problem already happens with the .fromordinal() class
> method, though it doesn't happen with .fromdatetime() or .now():
>
> >>> class D(datetime.datetime):
> ...   def __new__(cls): return cls.now()
> ...
> >>> D()
> D(2019, 1, 6, 10, 33, 37, 161606)
> >>> D.fromordinal(100)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: __new__() takes 1 positional argument but 4 were given
> >>> D.fromtimestamp(123456789)
> D(1973, 11, 29, 13, 33, 9)
> >>>
>
> On Sun, Jan 6, 2019 at 9:05 AM Paul Ganssle <paul at ganssle.io> wrote:
>
>> I can think of many reasons why datetime is different from builtins,
>> though to be honest I'm not sure that consistency for its own sake is
>> really a strong argument for keeping a counter-intuitive behavior - and to
>> be honest I'm open to the idea that *all* arithmetic types *should* have
>> some form of this change.
>>
>> That said, I would say that the biggest difference between datetime and
>> builtins (other than the fact that datetime is *not* a builtin, and as
>> such doesn't necessarily need to be categorized in this group), is that
>> unlike almost all other arithmetic types, *datetime* has a special,
>> dedicated type for describing differences in datetimes. Using your example
>> of a float subclass, consider that without the behavior of "addition of
>> floats returns floats", it would be hard to predict what would happen in
>> this situation:
>>
>> >>> F(1.2) + 3.4
>>
>> Would that always return a float, even though F(1.2) + F(3.4) returns an
>> F? Would that return an F because F is the left-hand operand? Would it
>> return a float because float is the right-hand operand? Would you walk the
>> MROs and find the lowest type in common between the operands and return
>> that? It's not entirely clear which subtype predominates. With datetime,
>> you have:
>>
>> datetime - datetime -> timedelta
>> datetime ? timedelta -> datetime
>> timedelta ? timedelta -> timedelta
>>
>> There's no operation between two datetime objects that would return a
>> datetime object, so it's always clear: operations between datetime
>> subclasses return timedelta, operations between a datetime object and a
>> timedelta return the subclass of the datetime that it was added to or
>> subtracted from.
>>
>> Of course, the real way to resolve whether datetime should be different
>> from int/float/string/etc is to look at why this choice was actually made
>> for those types in the first place, and decide whether datetime is like
>> them *in this respect*. The heterogeneous operations problem may be a
>> reasonable justification for leaving the other builtins alone but changing
>> datetime, but if someone knows of other fundamental reasons why the
>> decision to have arithmetic operations always create the base class was
>> chosen, please let me know.
>>
>> Best,
>> Paul
>> On 1/5/19 3:55 AM, Alexander Belopolsky wrote:
>>
>>
>>
>> On Wed, Jan 2, 2019 at 10:18 PM Paul Ganssle <paul at ganssle.io> wrote:
>>
>>> .. the original objection was that this implementation assumes that the
>>> datetime subclass has a constructor with the same (or a sufficiently
>>> similar) signature as datetime.
>>>
>> While this was used as a possible rationale for the way standard types
>> behave, the main objection to changing datetime classes is that it will
>> make them behave differently from builtins.  For example:
>>
>> >>> class F(float):
>> ...     pass
>> ...
>> >>> type(F.fromhex('AA'))
>> <class '__main__.F'>
>> >>> type(F(1) + F(2))
>> <class 'float'>
>>
>> This may be a legitimate gripe, but unfortunately that ship has sailed
>>> long ago. All of datetime's alternate constructors make this assumption.
>>> Any subclass that does not meet this requirement must have worked around it
>>> long ago (or they don't care about alternate constructors).
>>>
>>
>> This is right, but the same argument is equally applicable to int, float,
>> etc. subclasses.  If you want to limit your change to datetime types you
>> should explain what makes these types special.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190106/fa3ba704/attachment.html>

From ltian6 at eng.ua.edu  Mon Jan  7 19:11:23 2019
From: ltian6 at eng.ua.edu (Tian, Liang)
Date: Tue, 8 Jan 2019 00:11:23 +0000
Subject: [Python-Dev] python subprocess module to submit a list of slurm
 sbatch jobs,
 each job use multiprocessing.Pool to run simulation on single compute node
 in cluster
Message-ID: <ede904dacb9e40a880f8a8c52c58e463@eng.ua.edu>

Dear Developers,
I have been working on a piece of code development that need your sincere help (three code file attached here).

I use run_slurm.py to use python subprocess module to submit multiple jobs to slurm cluster by invoking a sbatch file with a for loop to reach the following target:

1) create some environmental variables for job_slurm.py to run simulations (values of these environmental variables will change for each job with the for loop)

2) invoke submit_slurm.sh to submit a sbatch job that will run job_slurm.py

3) each job_slurm.py will use python multiprocess.Pool to run parallized simulations on each ./mod_bart.sh exeutable file (will run for a few hours) on a single compute node in cluster using all cores of this compute node

I get the following error all the time, could you provide some insights on how our implementation is wrong to achieve the desired goal:


Exception in thread Thread-3:
Traceback (most recent call last):
  File "/share/apps/python/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/share/apps/python/2.7/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/share/apps/python/2.7/lib/python2.7/multiprocessing/pool.py", line 347, in _handle_results
    task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190108/7117a0e8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: run_slurm.py
Type: text/x-python-script
Size: 2268 bytes
Desc: run_slurm.py
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190108/7117a0e8/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: submit_slurm.sh
Type: text/x-sh
Size: 133 bytes
Desc: submit_slurm.sh
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190108/7117a0e8/attachment.sh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: job_slurm.py
Type: text/x-python-script
Size: 1055 bytes
Desc: job_slurm.py
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190108/7117a0e8/attachment-0001.bin>

From steve at pearwood.info  Mon Jan  7 20:06:03 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 8 Jan 2019 12:06:03 +1100
Subject: [Python-Dev] python subprocess module to submit a list of slurm
 sbatch jobs,
 each job use multiprocessing.Pool to run simulation on single compute node
 in cluster
In-Reply-To: <ede904dacb9e40a880f8a8c52c58e463@eng.ua.edu>
References: <ede904dacb9e40a880f8a8c52c58e463@eng.ua.edu>
Message-ID: <20190108010603.GN13616@ando.pearwood.info>

This mailing list is for the development of the Python interpreter, not 
for asking questions about your own code.

Did you sign up using the Python-Dev mailing list website? At the top of 
the page, it says

    Do not post general Python questions to this list. For help with 
    Python please see the Python help page.

in bold and highlighted in red. Is this message not clear enough? How 
can we make it clearer?

For help debugging your code, you can try the resources here:

https://www.python.org/about/help/

or this mailing list: 

https://mail.python.org/mailman/listinfo/python-list

or here:

https://www.reddit.com/r/learnpython/


Thank you.


-- 
Steve

From aman30894 at gmail.com  Tue Jan  8 03:05:29 2019
From: aman30894 at gmail.com (Aman Agrawal)
Date: Tue, 8 Jan 2019 13:35:29 +0530
Subject: [Python-Dev] How can i insert more than 32K data to a column type
 clob in db2 using python programming
Message-ID: <CAB_3DMrLd++-LeWCdXYSK9m=vUu1XLmJQdaDFGfch7h8GCm_pg@mail.gmail.com>

 Hi Team,

I have tried to insert in clob data type more than 32k byte data but
getting error:-

Statement Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0102N  The
string constant beginning with "\'<p>Hi good day team:<br /> We had been
requested to Add Anniel to all" is too long.  SQLSTATE=54002 SQLCODE=-102

could any one please help me is there any way i can achieve it?

Please find my DDL statement:-

CREATE TABLE "DB2IDEV "."CHANGE"  (
          "TICKET_ID" INTEGER NOT NULL ,
          "SHORTDESC" VARCHAR(5000 OCTETS) NOT NULL ,
          "LONGDESC" CLOB(2147483647 OCTETS) LOGGED NOT COMPACT ,
          "RELEASENUM" VARCHAR(12 OCTETS) WITH DEFAULT NULL ,
          "ISSOFTDELETED" INTEGER NOT NULL WITH DEFAULT 0 ,
          "CREATETIMESTAMP" TIMESTAMP NOT NULL WITH DEFAULT CURRENT
TIMESTAMP ,
          "LASTUPDATETIMESTAMP" TIMESTAMP WITH DEFAULT NULL )
         IN "TABLESPACE"
         ORGANIZE BY ROW  ;

 conn =
db.connect("DATABASE=%s;HOSTNAME=%s;PORT=60001;PROTOCOL=TCPIP;UID=%s;PWD=%s;"%(DB2_DATABASE,DB2_HOSTNAME,DB2_UID,DB2_PWD),
"", "")
cursor = conn.cursor()

sql="INSERT INTO Change
(Ticket_ID,shortDesc,longDesc,releaseNum,isSoftDeleted,createTimeStamp,lastUpdateTimeStamp)
VALUES ('296129','High Cisco Adaptive Security Appliance Remote Code
Execution and Denial of Service Vulnerabilit','<Some long data around
50KB>',NULL,'0','2018-02-07 02:11:50',NULL)"

cursor.execute(sql)
conn.commit()

Thanks & Regards

*Aman Agrawal*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190108/122fd3fe/attachment.html>

From steve at pearwood.info  Tue Jan  8 11:09:22 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 9 Jan 2019 03:09:22 +1100
Subject: [Python-Dev] How can i insert more than 32K data to a column
 type clob in db2 using python programming
In-Reply-To: <CAB_3DMrLd++-LeWCdXYSK9m=vUu1XLmJQdaDFGfch7h8GCm_pg@mail.gmail.com>
References: <CAB_3DMrLd++-LeWCdXYSK9m=vUu1XLmJQdaDFGfch7h8GCm_pg@mail.gmail.com>
Message-ID: <20190108160919.GR13616@ando.pearwood.info>

On Tue, Jan 08, 2019 at 01:35:29PM +0530, Aman Agrawal wrote:
>  Hi Team,
> 
> I have tried to insert in clob data type more than 32k byte data but
> getting error:-

This mailing list is for the development of the Python interpreter, not
for asking questions about your own code.

Did you sign up using the Python-Dev mailing list website? At the top of
the page, it says

    Do not post general Python questions to this list. For help with
    Python please see the Python help page.

in bold and highlighted in red. Is this message not clear enough? How
can we make it clearer?

For help debugging your code, you can try the resources here:

https://www.python.org/about/help/

or this mailing list:

https://mail.python.org/mailman/listinfo/python-list

or here:

https://www.reddit.com/r/learnpython/


Thank you.


--
Steve

From brian.kuhl at windriver.com  Wed Jan  9 11:06:56 2019
From: brian.kuhl at windriver.com (Kuhl, Brian)
Date: Wed, 9 Jan 2019 16:06:56 +0000
Subject: [Python-Dev] VxWorks and cpython?
Message-ID: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>

Hi Python Developers,
I'm Brian Kuhl, I've spent about  28 years working with embedded software. Since 2000 I've worked for Wind River. I'm currently a manager of documentation and customer training in our Ottawa, Canada office. Throughout my career I've had an interest in the use of open source software in embedded systems, have ported many libraries to Wind River's flagship product the VxWorks RTOS.

The safe and secure embedded market where VxWorks is the leader is evolving, and devices that use to be made up of multiple processors running multiple operating systems are now being consolidated. Device security and IoT trends mean a device must be more configurable and updateable, and a better host for portable code. In this evolving space our  customers have asked us to add Python support to VxWorks.

Wind River would like cpython to officially support VxWorks.  I've been ask by my colleagues to volunteer as a maintainer for cpython in order to support this effort.  Wind River will also provide the needed buildbot clients required to verify continuing VxWorks support.

Myself and an intern were able to get the majority of Python tests suite passing with a POC last year.
An engineering group in Beijing  have taken that POC and are improving and cleaning up that effort now with the hopes of up-steaming their efforts.

My Chinese colleagues have suggested that the first step to gaining your support would be to summit a PEP detailing the changes?
If you agree, I will proceed with authoring a PEP.

Many thanks in advance for your responses,

Brian


P.S. I can be found on  github (and many other places) as kuhlenough.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190109/4a79cab3/attachment.html>

From guido at python.org  Wed Jan  9 11:52:19 2019
From: guido at python.org (Guido van Rossum)
Date: Wed, 9 Jan 2019 08:52:19 -0800
Subject: [Python-Dev] VxWorks and cpython?
In-Reply-To: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
References: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
Message-ID: <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>

Hi Brian,

I am glad that this is happening!

I don't think you need a PEP to motivate your request -- however you need
to submit the upstream patches, work with the CPython buildbot managers
(Victor?) to connect your buildbots, and work with reviewers (Serhiy?) to
get your patches integrated in CPython. Hopefully the patches are small and
focused -- we generally don't take patches that refactor large swaths of
code. It's also better not to submit everything in a single giant patch
(reviewers hate mega-patches). And the patch submitters need to be
responsive when the code review comments are coming.

This is not a one-time effort -- your team will need to be "on call" to
handle VxWorks issues for the foreseeable future (a minimum commitment of 5
years?), and your team members needs to be able to do this in their
capacity as Wind River engineers, not as open source volunteers.

I believe there is a PEP with a list of supported operating systems that
needs to be updated, but I don't recall which one it is -- hopefully you
can track it down with a little Googling. (Or was it the devguide?)

Good luck,

--Guido

On Wed, Jan 9, 2019 at 8:42 AM Kuhl, Brian <brian.kuhl at windriver.com> wrote:

> Hi Python Developers,
>
> I?m Brian Kuhl, I?ve spent about  28 years working with embedded software.
> Since 2000 I?ve worked for Wind River. I?m currently a manager of
> documentation and customer training in our Ottawa, Canada office.
> Throughout my career I?ve had an interest in the use of open source
> software in embedded systems, have ported many libraries to Wind River?s
> flagship product the VxWorks RTOS.
>
>
>
> The safe and secure embedded market where VxWorks is the leader is
> evolving, and devices that use to be made up of multiple processors running
> multiple operating systems are now being consolidated. Device security and
> IoT trends mean a device must be more configurable and updateable, and a
> better host for portable code. In this evolving space our  customers have
> asked us to add Python support to VxWorks.
>
>
>
> Wind River would like cpython to officially support VxWorks.  I?ve been
> ask by my colleagues to volunteer as a maintainer for cpython in order to
> support this effort.  Wind River will also provide the needed buildbot
> clients required to verify continuing VxWorks support.
>
>
>
> Myself and an intern were able to get the majority of Python tests suite
> passing with a POC last year.
>
> An engineering group in Beijing  have taken that POC and are improving and
> cleaning up that effort now with the hopes of up-steaming their efforts.
>
>
>
> My Chinese colleagues have suggested that the first step to gaining your
> support would be to summit a PEP detailing the changes?
>
> If you agree, I will proceed with authoring a PEP.
>
>
>
> Many thanks in advance for your responses,
>
>
>
> Brian
>
>
>
>
>
> P.S. I can be found on  github (and many other places) as kuhlenough.
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190109/0a95f150/attachment.html>

From vstinner at redhat.com  Wed Jan  9 12:09:47 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 9 Jan 2019 18:09:47 +0100
Subject: [Python-Dev] AMD64 Windows8.1 Refleaks 3.x buildbot is back to
 green!
Message-ID: <CA+3bQGG3kqX3z_GCeEQNAOa=WVh1Nc9iopZyT305cFztZCAM1w@mail.gmail.com>

Hi,

The "AMD64 Windows 8.1 Refleaks 3.x" buildbot (which hunts reference
leaks and memory leaks) was failing on test_asyncio for 1 year:

   https://bugs.python.org/issue32710

It was a leak of a single reference:

   test_aiosend leaked [1, 1, 1] references, sum=3

I tried multiple times since last year (January 2018) to understand
the leak: it didn't make any sense (well, as any bug at the beginning,
no?). I checked several times the complex asyncio code: all transports
are cleanly closed, the event loop is properly closed, etc. The code
looks correct.

After a long sleepness night... I still failed to reproduce the bug
:-( But I succeeded to get a way shorter reproducer script. Thanks to
this script, I was able to loop to get 100 reference leaks instead of
leaking a single reference. Using tracemalloc, I found the faulty
line... but it still didn't make sense to me. After additional several
hours of debug, I found that an overlapped object wasn't released
properly: an asynchronous WSASend().

The problem was when an overlapped WSASend() failed immediately, the
internal buffer was not released, whereas it holds a reference to the
input byte string. **It means that an asyncio send() failure using the
ProactorEventLoop can cause a memory leak**... I'm very surprised that
nobody noticed such **huge** memory leak previously!

The _overlapped bugfix:

https://github.com/python/cpython/commit/a234e148394c2c7419372ab65b773d53a57f3625

Eventually, the "AMD64 Windows 8.1 Refleaks 3.x" buildbot is back to green!

   https://buildbot.python.org/all/#/builders/80

It means that it will be easier and faster to spot reference or memory
leak regressions (specific to Windows, the Gentoo Refleaks buildbot
was already green for several months!).

Since ProactorEventLoop became the default event loop in Python 3.8
(on Windows, it's specific to Windows), I hope that we fixed all most
obvious bugs!

This story also means that any very tiny buildbot failure (a single
test method failure on a single very specific buildbot) can hide a
giant bug ;-) Sadly, we have to fix *all* buildbots failures to find
them... Hopefully, almost all buildbots are green currently.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From christian at python.org  Wed Jan  9 12:13:32 2019
From: christian at python.org (Christian Heimes)
Date: Wed, 9 Jan 2019 18:13:32 +0100
Subject: [Python-Dev] VxWorks and cpython?
In-Reply-To: <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>
References: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
 <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>
Message-ID: <c33aa093-81cb-1c97-bb18-f86d78752d49@python.org>

On 09/01/2019 17.52, Guido van Rossum wrote:
> Hi Brian,
> 
> I am glad that this is happening!
> 
> I don't think you need a PEP to motivate your request -- however you
> need to submit the upstream patches, work with the CPython buildbot
> managers (Victor?) to connect your buildbots, and work with reviewers
> (Serhiy?) to get your patches integrated in CPython. Hopefully the
> patches are small and focused -- we generally don't take patches that
> refactor large swaths of code. It's also better not to submit everything
> in a single giant patch (reviewers hate mega-patches). And the patch
> submitters need to be responsive when the code review comments are coming.
> 
> This is not a one-time effort -- your team will need to be "on call" to
> handle VxWorks issues for the foreseeable future (a minimum commitment
> of 5 years?), and your team members needs to be able to do this in their
> capacity as Wind River engineers, not as open source volunteers.
> 
> I believe there is a PEP with a list of supported operating systems that
> needs to be updated, but I don't recall which one it is -- hopefully you
> can track it down with a little Googling. (Or was it the devguide?)

Hi,

It's a PEP. The process and expectations for platform are explained in
PEP 11, https://www.python.org/dev/peps/pep-0011/

If possible it would also be helpful to get SSH access to some VxWorks
machines for core devs. I know that Victor likes to dig into rare corner
cases and help to debug exotic platforms.

Christian

From brett at python.org  Wed Jan  9 12:51:04 2019
From: brett at python.org (Brett Cannon)
Date: Wed, 9 Jan 2019 09:51:04 -0800
Subject: [Python-Dev] AMD64 Windows8.1 Refleaks 3.x buildbot is back to
 green!
In-Reply-To: <CA+3bQGG3kqX3z_GCeEQNAOa=WVh1Nc9iopZyT305cFztZCAM1w@mail.gmail.com>
References: <CA+3bQGG3kqX3z_GCeEQNAOa=WVh1Nc9iopZyT305cFztZCAM1w@mail.gmail.com>
Message-ID: <CAP1=2W7tCBB=wySih9FJ20UDkxp5R+YgSkU3ZcvCk_6DDz6TwA@mail.gmail.com>

Thanks for tracking that down! Been bugging me as well and the one time I
tried to figure it out I got no where, so kudos for sticking with it!

On Wed, 9 Jan 2019 at 09:13, Victor Stinner <vstinner at redhat.com> wrote:

> Hi,
>
> The "AMD64 Windows 8.1 Refleaks 3.x" buildbot (which hunts reference
> leaks and memory leaks) was failing on test_asyncio for 1 year:
>
>    https://bugs.python.org/issue32710
>
> It was a leak of a single reference:
>
>    test_aiosend leaked [1, 1, 1] references, sum=3
>
> I tried multiple times since last year (January 2018) to understand
> the leak: it didn't make any sense (well, as any bug at the beginning,
> no?). I checked several times the complex asyncio code: all transports
> are cleanly closed, the event loop is properly closed, etc. The code
> looks correct.
>
> After a long sleepness night... I still failed to reproduce the bug
> :-( But I succeeded to get a way shorter reproducer script. Thanks to
> this script, I was able to loop to get 100 reference leaks instead of
> leaking a single reference. Using tracemalloc, I found the faulty
> line... but it still didn't make sense to me. After additional several
> hours of debug, I found that an overlapped object wasn't released
> properly: an asynchronous WSASend().
>
> The problem was when an overlapped WSASend() failed immediately, the
> internal buffer was not released, whereas it holds a reference to the
> input byte string. **It means that an asyncio send() failure using the
> ProactorEventLoop can cause a memory leak**... I'm very surprised that
> nobody noticed such **huge** memory leak previously!
>
> The _overlapped bugfix:
>
>
> https://github.com/python/cpython/commit/a234e148394c2c7419372ab65b773d53a57f3625
>
> Eventually, the "AMD64 Windows 8.1 Refleaks 3.x" buildbot is back to green!
>
>    https://buildbot.python.org/all/#/builders/80
>
> It means that it will be easier and faster to spot reference or memory
> leak regressions (specific to Windows, the Gentoo Refleaks buildbot
> was already green for several months!).
>
> Since ProactorEventLoop became the default event loop in Python 3.8
> (on Windows, it's specific to Windows), I hope that we fixed all most
> obvious bugs!
>
> This story also means that any very tiny buildbot failure (a single
> test method failure on a single very specific buildbot) can hide a
> giant bug ;-) Sadly, we have to fix *all* buildbots failures to find
> them... Hopefully, almost all buildbots are green currently.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190109/9d259a9b/attachment.html>

From andrew.svetlov at gmail.com  Wed Jan  9 13:05:12 2019
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Wed, 9 Jan 2019 20:05:12 +0200
Subject: [Python-Dev] AMD64 Windows8.1 Refleaks 3.x buildbot is back to
 green!
In-Reply-To: <CAP1=2W7tCBB=wySih9FJ20UDkxp5R+YgSkU3ZcvCk_6DDz6TwA@mail.gmail.com>
References: <CA+3bQGG3kqX3z_GCeEQNAOa=WVh1Nc9iopZyT305cFztZCAM1w@mail.gmail.com>
 <CAP1=2W7tCBB=wySih9FJ20UDkxp5R+YgSkU3ZcvCk_6DDz6TwA@mail.gmail.com>
Message-ID: <CAL3CFcUGaxZNAOJEOnoLXwD=ttArt8Kb0-HT0Hh2n+K3LQ+LAA@mail.gmail.com>

That's great!

I would add that many Windows Proactor bugs were found after implementing
sendfile support.
If sendfile is not available for some reason asyncio uses a fallback to
send a file content chunk-by-chunk reading it in memory.
The fallback code was easy and straightforward, it worked just fine on
Linux but the same code failed on proactor.

Thank you very much, Victor, for finally pinning the problem down.

On Wed, Jan 9, 2019 at 7:51 PM Brett Cannon <brett at python.org> wrote:

> Thanks for tracking that down! Been bugging me as well and the one time I
> tried to figure it out I got no where, so kudos for sticking with it!
>
> On Wed, 9 Jan 2019 at 09:13, Victor Stinner <vstinner at redhat.com> wrote:
>
>> Hi,
>>
>> The "AMD64 Windows 8.1 Refleaks 3.x" buildbot (which hunts reference
>> leaks and memory leaks) was failing on test_asyncio for 1 year:
>>
>>    https://bugs.python.org/issue32710
>>
>> It was a leak of a single reference:
>>
>>    test_aiosend leaked [1, 1, 1] references, sum=3
>>
>> I tried multiple times since last year (January 2018) to understand
>> the leak: it didn't make any sense (well, as any bug at the beginning,
>> no?). I checked several times the complex asyncio code: all transports
>> are cleanly closed, the event loop is properly closed, etc. The code
>> looks correct.
>>
>> After a long sleepness night... I still failed to reproduce the bug
>> :-( But I succeeded to get a way shorter reproducer script. Thanks to
>> this script, I was able to loop to get 100 reference leaks instead of
>> leaking a single reference. Using tracemalloc, I found the faulty
>> line... but it still didn't make sense to me. After additional several
>> hours of debug, I found that an overlapped object wasn't released
>> properly: an asynchronous WSASend().
>>
>> The problem was when an overlapped WSASend() failed immediately, the
>> internal buffer was not released, whereas it holds a reference to the
>> input byte string. **It means that an asyncio send() failure using the
>> ProactorEventLoop can cause a memory leak**... I'm very surprised that
>> nobody noticed such **huge** memory leak previously!
>>
>> The _overlapped bugfix:
>>
>>
>> https://github.com/python/cpython/commit/a234e148394c2c7419372ab65b773d53a57f3625
>>
>> Eventually, the "AMD64 Windows 8.1 Refleaks 3.x" buildbot is back to
>> green!
>>
>>    https://buildbot.python.org/all/#/builders/80
>>
>> It means that it will be easier and faster to spot reference or memory
>> leak regressions (specific to Windows, the Gentoo Refleaks buildbot
>> was already green for several months!).
>>
>> Since ProactorEventLoop became the default event loop in Python 3.8
>> (on Windows, it's specific to Windows), I hope that we fixed all most
>> obvious bugs!
>>
>> This story also means that any very tiny buildbot failure (a single
>> test method failure on a single very specific buildbot) can hide a
>> giant bug ;-) Sadly, we have to fix *all* buildbots failures to find
>> them... Hopefully, almost all buildbots are green currently.
>>
>> Victor
>> --
>> Night gathers, and now my watch begins. It shall not end until my death.
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>


-- 
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190109/e9a0de65/attachment.html>

From brian.kuhl at windriver.com  Wed Jan  9 12:03:45 2019
From: brian.kuhl at windriver.com (Kuhl, Brian)
Date: Wed, 9 Jan 2019 17:03:45 +0000
Subject: [Python-Dev] FW:  VxWorks and cpython?
In-Reply-To: <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>
References: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
 <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>
Message-ID: <6C486AB80682E24095E3E77A408D68F201499C9C3A@ALA-MBD.corp.ad.wrs.com>

From: Guido van Rossum [mailto:guido at python.org]
Sent: Wednesday, January 09, 2019 11:52 AM
To: Kuhl, Brian
Cc: python-dev at python.org
Subject: Re: [Python-Dev] VxWorks and cpython?

Hi Brian,

I am glad that this is happening!

I don't think you need a PEP to motivate your request -- however you need to submit the upstream patches, work with the CPython buildbot managers (Victor?) to connect your buildbots, and work with reviewers (Serhiy?) to get your patches integrated in CPython. Hopefully the patches are small and focused -- we generally don't take patches that refactor large swaths of code. It's also better not to submit everything in a single giant patch (reviewers hate mega-patches). And the patch submitters need to be responsive when the code review comments are coming.

This is not a one-time effort -- your team will need to be "on call" to handle VxWorks issues for the foreseeable future (a minimum commitment of 5 years?), and your team members needs to be able to do this in their capacity as Wind River engineers, not as open source volunteers.

I believe there is a PEP with a list of supported operating systems that needs to be updated, but I don't recall which one it is -- hopefully you can track it down with a little Googling. (Or was it the devguide?)

Good luck,

--Guido

On Wed, Jan 9, 2019 at 8:42 AM Kuhl, Brian <brian.kuhl at windriver.com<mailto:brian.kuhl at windriver.com>> wrote:
Hi Python Developers,
I?m Brian Kuhl, I?ve spent about  28 years working with embedded software. Since 2000 I?ve worked for Wind River. I?m currently a manager of documentation and customer training in our Ottawa, Canada office. Throughout my career I?ve had an interest in the use of open source software in embedded systems, have ported many libraries to Wind River?s flagship product the VxWorks RTOS.

The safe and secure embedded market where VxWorks is the leader is evolving, and devices that use to be made up of multiple processors running multiple operating systems are now being consolidated. Device security and IoT trends mean a device must be more configurable and updateable, and a better host for portable code. In this evolving space our  customers have asked us to add Python support to VxWorks.

Wind River would like cpython to officially support VxWorks.  I?ve been ask by my colleagues to volunteer as a maintainer for cpython in order to support this effort.  Wind River will also provide the needed buildbot clients required to verify continuing VxWorks support.

Myself and an intern were able to get the majority of Python tests suite passing with a POC last year.
An engineering group in Beijing  have taken that POC and are improving and cleaning up that effort now with the hopes of up-steaming their efforts.

My Chinese colleagues have suggested that the first step to gaining your support would be to summit a PEP detailing the changes?
If you agree, I will proceed with authoring a PEP.

Many thanks in advance for your responses,

Brian


P.S. I can be found on  github (and many other places) as kuhlenough.


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org<mailto:Python-Dev at python.org>
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org


--
--Guido van Rossum (python.org/~guido<http://python.org/~guido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190109/f3d40c1d/attachment-0001.html>

From vstinner at redhat.com  Wed Jan  9 18:43:20 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 10 Jan 2019 00:43:20 +0100
Subject: [Python-Dev] VxWorks and cpython?
In-Reply-To: <c33aa093-81cb-1c97-bb18-f86d78752d49@python.org>
References: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
 <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>
 <c33aa093-81cb-1c97-bb18-f86d78752d49@python.org>
Message-ID: <CA+3bQGHAAYOOu88JErN8eTrOvkvW1GSr8E2QvHL2nu2Nkn3d8g@mail.gmail.com>

Le mer. 9 janv. 2019 ? 18:16, Christian Heimes <christian at python.org> a ?crit :
> It's a PEP. The process and expectations for platform are explained in
> PEP 11, https://www.python.org/dev/peps/pep-0011/

I also wrote some information in my website:
https://pythondev.readthedocs.io/platforms.html

> If possible it would also be helpful to get SSH access to some VxWorks
> machines for core devs. I know that Victor likes to dig into rare corner
> cases and help to debug exotic platforms.

The best case to get a full support is to have one core developer
responsible of handling all bugs specific to the platform.

As a core developer, I'm never comfortable to merge a change specific
to a platform if I'm not able to validate it manually myself. I trust
no one, not even myself (I know well that I do frequently mistakes!),
so I prefer to always double check changes before merging them ;-)

In the meanwhile, I would say that we can only offer "best effort"
support. Fix reports bugs and do our best in our limited time.

Someone has to take the work done to port Python on VxWorks and write
small pull requests. These PRs should be reviewed one by one to make
sure that it's the correct way to fix an issue. Be prepared that it
can take several months even if all these changes look obvious to
*you*. Core developers have limited time and many prefer to focus on
the platforms that they are using, not worry about a platform they
never heard about... I can have a look at such PRs.

It would also help to have a documentation somewhere about "Python on
VxWorks". Pointers to VxWorks (general info, developers info),
pointers to your port, current status of the port, etc.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From chris.jerdonek at gmail.com  Wed Jan  9 20:32:44 2019
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Wed, 9 Jan 2019 17:32:44 -0800
Subject: [Python-Dev] Interested in serving on Steering Council
In-Reply-To: <20190104213605.57fba25b@fsol>
References: <CAEbHw4bhjt3k-Q57BnSWw150B2Pint1465ick+775i-eaYeVBA@mail.gmail.com>
 <20190102191517.7fc2094c@fsol>
 <cb08756a-cf4b-f6ad-5147-7ef3e41f4c95@python.org>
 <CAEbHw4b-dWKeYWaYoGBS1HMxJjuN0gquM99mKfuNvvU3wmwzGQ@mail.gmail.com>
 <20190104213605.57fba25b@fsol>
Message-ID: <CAOTb1wcQB-ZQG63Pri06bT06zFtbHdqMpd7iS14QPiFznd-Wkg@mail.gmail.com>

Just to close (or continue) the loop on this thread, I just nominated
David for the steering council:
https://discuss.python.org/t/steering-council-nomination-david-mertz/647

Thanks for stepping forward with your interest and willingness to serve, David!

--Chris

On Fri, Jan 4, 2019 at 12:37 PM Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>
> Hi David,
>
> On Fri, 4 Jan 2019 15:24:20 -0500
> David Mertz <mertz at gnosis.cx> wrote:
> >
> > I've been part of the Python community since 1998, but really active in it
> > since about 2001.  During the early 2000s, I wrote a large number of widely
> > read articles promoting Python, often delving into explaining semi-obscure
> > features and language design issues.  Most of these were with in my column
> > _Charming Python_.  I believe that several changes in Python itself?such as
> > making coroutines easier to use and the design of metaclasses and class
> > decorators?were significantly influenced by things I wrote on the topics.
> > [snip]
>
> Those are useful things to know, thank you.
>
> > If the core developers feel that the overwhelming qualification for the
> > Steering Committee is familiarity with the C code base of CPython, then
> > indeed I am not the best candidate for that.
>
> Obviously not the overwhelming qualification (though at least _some_ of
> the committee members would have to be familiar with the C code base, I
> think).
>
> > If language design issues are
> > more important?and especially if thinking about Python's place among users
> > and industry are important, then I think I'm a very good candidate for the
> > role.
>
> That, but I think also familiarity with the development and
> contribution process, will definitely play a role.  In other words, if
> some external candidate gets elected I would hope they take the time to
> become familiar with how things work in that regard, and try to
> contribute themselves (not necessarily to make important contributions
> to the codebase but to understand the daily routine).
>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com

From hodgestar+pythondev at gmail.com  Thu Jan 10 03:03:18 2019
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Thu, 10 Jan 2019 10:03:18 +0200
Subject: [Python-Dev] VxWorks and cpython?
In-Reply-To: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
References: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
Message-ID: <CAD5NRCEiw8yxNp3RsewXYbn1A_+vzYH2ZVG+64A-EMX2GtUC0w@mail.gmail.com>

It feels like it would be a good idea for Brian or someone else from
Wind River to work towards becoming a core developer? If Brian, et al
are interested, is there a core developer who would be willing to
mentor them?

From brian.kuhl at windriver.com  Thu Jan 10 11:54:46 2019
From: brian.kuhl at windriver.com (Kuhl, Brian)
Date: Thu, 10 Jan 2019 16:54:46 +0000
Subject: [Python-Dev] VxWorks and cpython?
In-Reply-To: <CA+3bQGHAAYOOu88JErN8eTrOvkvW1GSr8E2QvHL2nu2Nkn3d8g@mail.gmail.com>
References: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
 <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>
 <c33aa093-81cb-1c97-bb18-f86d78752d49@python.org>
 <CA+3bQGHAAYOOu88JErN8eTrOvkvW1GSr8E2QvHL2nu2Nkn3d8g@mail.gmail.com>
Message-ID: <6C486AB80682E24095E3E77A408D68F201499CAF79@ALA-MBD.corp.ad.wrs.com>

Hi Victor,
I think we can come up with some sort of strategy that will work for everyone. 
I'll ask about the SSH access; and if that runs up against a corporate roadblock, I will explore some other alternative. 

Is there a good place to document "Python on VxWorks" ?
The changes to Python are not large, I've kept the pull request from last year's POC active. The changed files provide a good summary of the scope.
https://github.com/python/cpython/pull/4184/files
However, based on the POC we've gone back and improved VxWorks, so the some of the uglier bits, in libraries like submodule, won't be needed.
These will be in the next release of VxWorks. 

However we let automake and setup.py do much of the work for us, so where VxWorks  does not have support for something, it's not obvious.
A public document would go a long way to filling in those details, something much more detailed than  my glib "VxWorks is almost POSIX" in the pull request.  

Other challenges; 
* VxWorks is cross-compiled on both Linux and Windows. ( currently with clang and gcc)
* Supported on ARM,  PowerPC and Intel processors
* 32bit and 64bit builds 
* A constantly evolving set of reference boards (or BSPs)   
https://marketplace.windriver.com/index.php?bsp&on=list&type=platform&value=VxWorks:%207%20-%20Wind%20River%20Workbench%204.0

I don't think we need a buildbot for every board.  I'm thinking a 1/2 dozen to cover ARM, PPC and IA with both a 32bit and 64 bit build?
We have a bit of chicken and egg problem right now, a buildbot will always fail until there's some basic VxWorks support added. 

Do we set them up, and just let them fail, till enough PRs are accepted to make it build?

Brian

> -----Original Message-----
> From: Victor Stinner [mailto:vstinner at redhat.com]
> Sent: Wednesday, January 09, 2019 6:43 PM
> To: Christian Heimes
> Cc: Guido van Rossum; Kuhl, Brian; python-dev at python.org
> Subject: Re: [Python-Dev] VxWorks and cpython?
> 
> Le mer. 9 janv. 2019 ? 18:16, Christian Heimes <christian at python.org> a ?crit :
> > It's a PEP. The process and expectations for platform are explained in
> > PEP 11, https://www.python.org/dev/peps/pep-0011/
> 
> I also wrote some information in my website:
> https://pythondev.readthedocs.io/platforms.html
> 
> > If possible it would also be helpful to get SSH access to some VxWorks
> > machines for core devs. I know that Victor likes to dig into rare
> > corner cases and help to debug exotic platforms.
> 
> The best case to get a full support is to have one core developer responsible of
> handling all bugs specific to the platform.
> 
> As a core developer, I'm never comfortable to merge a change specific to a
> platform if I'm not able to validate it manually myself. I trust no one, not even
> myself (I know well that I do frequently mistakes!), so I prefer to always double
> check changes before merging them ;-)
> 
> In the meanwhile, I would say that we can only offer "best effort"
> support. Fix reports bugs and do our best in our limited time.
> 
> Someone has to take the work done to port Python on VxWorks and write small
> pull requests. These PRs should be reviewed one by one to make sure that it's
> the correct way to fix an issue. Be prepared that it can take several months even
> if all these changes look obvious to *you*. Core developers have limited time
> and many prefer to focus on the platforms that they are using, not worry about
> a platform they never heard about... I can have a look at such PRs.
> 
> It would also help to have a documentation somewhere about "Python on
> VxWorks". Pointers to VxWorks (general info, developers info), pointers to your
> port, current status of the port, etc.
> 
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.

From vstinner at redhat.com  Thu Jan 10 12:19:52 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Thu, 10 Jan 2019 18:19:52 +0100
Subject: [Python-Dev] VxWorks and cpython?
In-Reply-To: <6C486AB80682E24095E3E77A408D68F201499CAF79@ALA-MBD.corp.ad.wrs.com>
References: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
 <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>
 <c33aa093-81cb-1c97-bb18-f86d78752d49@python.org>
 <CA+3bQGHAAYOOu88JErN8eTrOvkvW1GSr8E2QvHL2nu2Nkn3d8g@mail.gmail.com>
 <6C486AB80682E24095E3E77A408D68F201499CAF79@ALA-MBD.corp.ad.wrs.com>
Message-ID: <CA+3bQGEbOz4X7RbccRrXZn9fgJ3_ZagjV7hD-Aq-J605_rcJ7Q@mail.gmail.com>

Le jeu. 10 janv. 2019 ? 17:54, Kuhl, Brian <brian.kuhl at windriver.com> a ?crit :
> Is there a good place to document "Python on VxWorks" ?

Anywhere. If you don't know, you might use
https://wiki.python.org/moin/ ... But I'm not sure that the wiki is
still widely used. Many pages may be outdated.

> The changes to Python are not large, I've kept the pull request from last year's POC active. The changed files provide a good summary of the scope.
> https://github.com/python/cpython/pull/4184/files

That's a giant PR. Sorry, I'm unable to review that. Usually, I simply
ignore such PR.

> However we let automake and setup.py do much of the work for us, so where VxWorks  does not have support for something, it's not obvious.
> A public document would go a long way to filling in those details, something much more detailed than  my glib "VxWorks is almost POSIX" in the pull request.

Cross-compilation is complex. So yeah, any documentation is more than
welcome. Many people are still fighting to try to get a working Python
on Android... My notes:
https://pythondev.readthedocs.io/android.html
(is there still someone working on that?)

> Other challenges;
> * VxWorks is cross-compiled on both Linux and Windows. ( currently with clang and gcc)
> * Supported on ARM,  PowerPC and Intel processors

I don't think that it should be an issue for Python. Very few parts of
CPython are architecture specific. I'm aware of ctypes which is more
or less optional. ctypes rely on libffi which supports a wide range of
architectures.

> * 32bit and 64bit builds

We have a wide range of 32 and 64 bits buildbot workers. I don't
expect any issue here.

Which C compiler do you use?

> * A constantly evolving set of reference boards (or BSPs)
> https://marketplace.windriver.com/index.php?bsp&on=list&type=platform&value=VxWorks:%207%20-%20Wind%20River%20Workbench%204.0

I'm not sure how it's supposed to impact Python?

> I don't think we need a buildbot for every board.  I'm thinking a 1/2 dozen to cover ARM, PPC and IA with both a 32bit and 64 bit build?
> We have a bit of chicken and egg problem right now, a buildbot will always fail until there's some basic VxWorks support added.

No please. A *single* buildbot worker in total is enough. When you
will have a very good support and a full test suite coverage, we can
discuss about adding more flavors.

> Do we set them up, and just let them fail, till enough PRs are accepted to make it build?

Multiple buildbot workers are failing since many years. *I* would
prefer to see the full test suite passing (even if some tests are
skipped on your platform) before adding a buildbot, but it seems like
some people have a different opinion on that. For example, there is an
AIX buildbot and some tests are still failing (it was always red,
failing, no?).

You're right that it's a chicken-and-egg issue, except that we don't
have a very strong policy for buildbots.

Note: I cannot promise that I will review your PR. I can only promise
that I will have a look :-) VxWorks is not really a priority for me.
(I prefer to not give false promise here.)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.

From storchaka at gmail.com  Thu Jan 10 12:38:37 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 10 Jan 2019 19:38:37 +0200
Subject: [Python-Dev] VxWorks and cpython?
In-Reply-To: <6C486AB80682E24095E3E77A408D68F201499CAF79@ALA-MBD.corp.ad.wrs.com>
References: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
 <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>
 <c33aa093-81cb-1c97-bb18-f86d78752d49@python.org>
 <CA+3bQGHAAYOOu88JErN8eTrOvkvW1GSr8E2QvHL2nu2Nkn3d8g@mail.gmail.com>
 <6C486AB80682E24095E3E77A408D68F201499CAF79@ALA-MBD.corp.ad.wrs.com>
Message-ID: <q17vqq$39g$1@blaine.gmane.org>

10.01.19 18:54, Kuhl, Brian ????:
> The changes to Python are not large, I've kept the pull request from last year's POC active. The changed files provide a good summary of the scope.
> https://github.com/python/cpython/pull/4184/files

To prepare this PR for review the following changes are required:

* Update it to the current master. Currently there are many conflicts.
* Remove unrelated changes, like: adding/removing empty lines, 
adding/removing spaces in unrelated lines.
* Remove unrelated changes in 'configure'. It depends on the used 
version of autoconf.
* Ensure that you do not use tabs, unless they are necessary (in Makefile).
* Make the code conforming PEP 7 and PEP 8. In particular, spaces should 
be added between 'if' and '(', between ')' and '{', opening '{' should 
be on the same line for statements, but on a new line if it is defines a 
function body, parenthesis around conditions of 'if' and 'while' are not 
needed in Python, long lines should be wrapped, etc, etc.


From status at bugs.python.org  Fri Jan 11 13:07:44 2019
From: status at bugs.python.org (Python tracker)
Date: Fri, 11 Jan 2019 18:07:44 +0000
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190111180744.1.B3C927E2606B4686@roundup.psfhosted.org>


ACTIVITY SUMMARY (2019-01-04 - 2019-01-11)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6932 (+10)
  closed 40537 (+50)
  total  47469 (+60)

Open issues with patches: 2762 


Issues opened (46)
==================

#8538: Add FlagAction to argparse
https://bugs.python.org/issue8538  reopened by vstinner

#32592: Drop support of Windows Vista in Python 3.7
https://bugs.python.org/issue32592  reopened by vstinner

#35661: Store the venv prompt in pyvenv.cfg
https://bugs.python.org/issue35661  opened by brett.cannon

#35662: Windows #define _PY_EMULATED_WIN_CV 0 bug
https://bugs.python.org/issue35662  opened by jeffr at livedata.com

#35663: webbrowser.py firefox bug [python3, windows 10]
https://bugs.python.org/issue35663  opened by codextj

#35666: Update design FAQ about assignment expression
https://bugs.python.org/issue35666  opened by carlbordum

#35667: activate for venv containing apostrophe doesn't work in powers
https://bugs.python.org/issue35667  opened by cheryl.sabella

#35668: Improve test coverage for idlelib
https://bugs.python.org/issue35668  opened by anthony shaw

#35669: tar symlink
https://bugs.python.org/issue35669  opened by Yilmaz

#35671: reserved identifier violation
https://bugs.python.org/issue35671  opened by elfring

#35673: Loader for namespace packages
https://bugs.python.org/issue35673  opened by ronaldoussoren

#35674: Expose os.posix_spawnp()
https://bugs.python.org/issue35674  opened by vstinner

#35675: IDLE: Refactor config_key module.
https://bugs.python.org/issue35675  opened by terry.reedy

#35676: unittest assert helper methods have incorrect signature in doc
https://bugs.python.org/issue35676  opened by ????????

#35677: Do not automount in stat() by default
https://bugs.python.org/issue35677  opened by serhiy.storchaka

#35679: pdb restart hooks
https://bugs.python.org/issue35679  opened by Hernot

#35680: [2.7] Coverity scan: Passing freed pointer "name" as an argume
https://bugs.python.org/issue35680  opened by cstratak

#35681: urllib.request.HTTPPasswordMgr.add_password requires more info
https://bugs.python.org/issue35681  opened by tsekine

#35683: Enable manylinux1 builds on Pipelines for CI testing
https://bugs.python.org/issue35683  opened by steve.dower

#35685: Add samples on patch.dict of the use of decorator and in class
https://bugs.python.org/issue35685  opened by eamanu

#35686: BufferError with memory.release()
https://bugs.python.org/issue35686  opened by Thomas.Waldmann

#35687: The unittest module diff is missing/forgetting/not putting new
https://bugs.python.org/issue35687  opened by addons_zz

#35688: "pip install --user numpy" fails on Python from the Windows St
https://bugs.python.org/issue35688  opened by mattip

#35689: IDLE: Docstrings and test for colorizer
https://bugs.python.org/issue35689  opened by cheryl.sabella

#35690: IDLE: Fix and test debugger.
https://bugs.python.org/issue35690  opened by terry.reedy

#35691: cpython3.7.2 make test failed
https://bugs.python.org/issue35691  opened by Wencan Deng

#35692: pathlib.Path.exists() on non-existent drive raises WinError in
https://bugs.python.org/issue35692  opened by Jordan Hueckstaedt

#35693: test_httpservers fails
https://bugs.python.org/issue35693  opened by neyuru

#35696: remove unnecessary operation in long_compare()
https://bugs.python.org/issue35696  opened by Ma Lin

#35697: _decimal: Implement the previously rejected changes from #7442
https://bugs.python.org/issue35697  opened by vstinner

#35698: [statistics] Division by 2 in statistics.median
https://bugs.python.org/issue35698  opened by jfine2358

#35699: distutils cannot find Build Tools 2017 since 3.7.2
https://bugs.python.org/issue35699  opened by schlamar

#35701: [uuid] 3.8 breaks weak references for UUIDs
https://bugs.python.org/issue35701  opened by josh.r

#35704: On AIX, test_unpack_archive_xztar fails with default MAXDATA s
https://bugs.python.org/issue35704  opened by Michael.Felt

#35705: libffi support is not there for windows on ARM64
https://bugs.python.org/issue35705  opened by ossdev07

#35706: Make it easier to use a venv with an embedded Python interpret
https://bugs.python.org/issue35706  opened by Dieter Weber

#35707: time.sleep() should support objects with __float__
https://bugs.python.org/issue35707  opened by jdemeyer

#35708: lib2to3 failed to convert as refactor's fixes not search.pyc f
https://bugs.python.org/issue35708  opened by njhsio

#35710: Make dataclasses.field() accept another name for __init__ fiel
https://bugs.python.org/issue35710  opened by remi.lapeyre

#35711: Print information about an unexpectedly pending error before c
https://bugs.python.org/issue35711  opened by sfreilich

#35712: Make NotImplemented unusable in boolean context
https://bugs.python.org/issue35712  opened by josh.r

#35713: Fatal Python error: _PySys_BeginInit: can't initialize sys mod
https://bugs.python.org/issue35713  opened by Tasy

#35714: Document that the null character '\0' terminates a struct form
https://bugs.python.org/issue35714  opened by bup

#35715: ProcessPool workers hold onto return value of last task in mem
https://bugs.python.org/issue35715  opened by dchevell

#35717: enum.Enum error on sys._getframe(2)
https://bugs.python.org/issue35717  opened by vstinner

#35718: Cannot initialize the "force" command-option
https://bugs.python.org/issue35718  opened by Opher Shachar



Most recent 15 issues with no replies (15)
==========================================

#35718: Cannot initialize the "force" command-option
https://bugs.python.org/issue35718

#35715: ProcessPool workers hold onto return value of last task in mem
https://bugs.python.org/issue35715

#35710: Make dataclasses.field() accept another name for __init__ fiel
https://bugs.python.org/issue35710

#35708: lib2to3 failed to convert as refactor's fixes not search.pyc f
https://bugs.python.org/issue35708

#35706: Make it easier to use a venv with an embedded Python interpret
https://bugs.python.org/issue35706

#35705: libffi support is not there for windows on ARM64
https://bugs.python.org/issue35705

#35704: On AIX, test_unpack_archive_xztar fails with default MAXDATA s
https://bugs.python.org/issue35704

#35701: [uuid] 3.8 breaks weak references for UUIDs
https://bugs.python.org/issue35701

#35696: remove unnecessary operation in long_compare()
https://bugs.python.org/issue35696

#35692: pathlib.Path.exists() on non-existent drive raises WinError in
https://bugs.python.org/issue35692

#35690: IDLE: Fix and test debugger.
https://bugs.python.org/issue35690

#35689: IDLE: Docstrings and test for colorizer
https://bugs.python.org/issue35689

#35688: "pip install --user numpy" fails on Python from the Windows St
https://bugs.python.org/issue35688

#35685: Add samples on patch.dict of the use of decorator and in class
https://bugs.python.org/issue35685

#35683: Enable manylinux1 builds on Pipelines for CI testing
https://bugs.python.org/issue35683



Most recent 15 issues waiting for review (15)
=============================================

#35717: enum.Enum error on sys._getframe(2)
https://bugs.python.org/issue35717

#35715: ProcessPool workers hold onto return value of last task in mem
https://bugs.python.org/issue35715

#35711: Print information about an unexpectedly pending error before c
https://bugs.python.org/issue35711

#35707: time.sleep() should support objects with __float__
https://bugs.python.org/issue35707

#35705: libffi support is not there for windows on ARM64
https://bugs.python.org/issue35705

#35704: On AIX, test_unpack_archive_xztar fails with default MAXDATA s
https://bugs.python.org/issue35704

#35699: distutils cannot find Build Tools 2017 since 3.7.2
https://bugs.python.org/issue35699

#35697: _decimal: Implement the previously rejected changes from #7442
https://bugs.python.org/issue35697

#35696: remove unnecessary operation in long_compare()
https://bugs.python.org/issue35696

#35690: IDLE: Fix and test debugger.
https://bugs.python.org/issue35690

#35689: IDLE: Docstrings and test for colorizer
https://bugs.python.org/issue35689

#35685: Add samples on patch.dict of the use of decorator and in class
https://bugs.python.org/issue35685

#35683: Enable manylinux1 builds on Pipelines for CI testing
https://bugs.python.org/issue35683

#35680: [2.7] Coverity scan: Passing freed pointer "name" as an argume
https://bugs.python.org/issue35680

#35677: Do not automount in stat() by default
https://bugs.python.org/issue35677



Top 10 most discussed issues (10)
=================================

#35582: Argument Clinic: inline parsing code for functions with only p
https://bugs.python.org/issue35582  17 msgs

#35693: test_httpservers fails
https://bugs.python.org/issue35693  14 msgs

#35686: BufferError with memory.release()
https://bugs.python.org/issue35686  12 msgs

#35697: _decimal: Implement the previously rejected changes from #7442
https://bugs.python.org/issue35697  10 msgs

#35698: [statistics] Division by 2 in statistics.median
https://bugs.python.org/issue35698  10 msgs

#35668: Improve test coverage for idlelib
https://bugs.python.org/issue35668   9 msgs

#19974: tarfile doesn't overwrite symlink by directory
https://bugs.python.org/issue19974   8 msgs

#35674: Expose os.posix_spawnp()
https://bugs.python.org/issue35674   7 msgs

#35687: The unittest module diff is missing/forgetting/not putting new
https://bugs.python.org/issue35687   7 msgs

#35717: enum.Enum error on sys._getframe(2)
https://bugs.python.org/issue35717   7 msgs



Issues closed (50)
==================

#4696: email module does not unfold headers
https://bugs.python.org/issue4696  closed by r.david.murray

#14614: PyTuple_SET_ITEM could check bounds in debug mode
https://bugs.python.org/issue14614  closed by vstinner

#24746: doctest 'fancy diff' formats incorrectly strip trailing whites
https://bugs.python.org/issue24746  closed by orsenthil

#25412: __floordiv__ in module fraction fails with TypeError instead o
https://bugs.python.org/issue25412  closed by ShashkovS

#26239: distutils link-objects is not normalized
https://bugs.python.org/issue26239  closed by cheryl.sabella

#31450: Subprocess exceptions re-raised in parent process do not have 
https://bugs.python.org/issue31450  closed by vstinner

#31887: docs for email.generator are missing a comment on special mult
https://bugs.python.org/issue31887  closed by cheryl.sabella

#32660: Solaris should support constants like termios' FIONREAD
https://bugs.python.org/issue32660  closed by cheryl.sabella

#32710: test_asyncio: ProactorEventLoopTests sendfile tests leak refer
https://bugs.python.org/issue32710  closed by vstinner

#33437: Defining __init__ in enums
https://bugs.python.org/issue33437  closed by cheryl.sabella

#33498: pathlib.Path wants an rmtree method
https://bugs.python.org/issue33498  closed by pitrou

#33834: test_asyncio: test_sendfile_close_peer_in_the_middle_of_receiv
https://bugs.python.org/issue33834  closed by vstinner

#34173: [3.7] deadlock in /usr/lib/python3.7/concurrent/futures/thread
https://bugs.python.org/issue34173  closed by xtreak

#34439: Expose venv --prompt value to an environment value
https://bugs.python.org/issue34439  closed by vinay.sajip

#34855: batch file variables
https://bugs.python.org/issue34855  closed by steve.dower

#35156: Consider revising documentation on Python Builds from source
https://bugs.python.org/issue35156  closed by neyuru

#35157: Missing pyconfig.h when building from source and pgo flag is e
https://bugs.python.org/issue35157  closed by neyuru

#35374: Windows doc build does not find autodetected hhc.exe
https://bugs.python.org/issue35374  closed by steve.dower

#35399: Sysconfig bug
https://bugs.python.org/issue35399  closed by neyuru

#35400: PGOMGR : warning PG0188:
https://bugs.python.org/issue35400  closed by neyuru

#35432: str.format and string.Formatter bug with French (and other) lo
https://bugs.python.org/issue35432  closed by vstinner

#35470: A deadly decref in _PyImport_FindExtensionObjectEx()
https://bugs.python.org/issue35470  closed by serhiy.storchaka

#35483: tarfile.extractall on existing symlink in Ubuntu overwrites ta
https://bugs.python.org/issue35483  closed by martin.panter

#35530: Counter-intuitive logging API
https://bugs.python.org/issue35530  closed by vinay.sajip

#35550: Some define guards for Solaris are wrong
https://bugs.python.org/issue35550  closed by vstinner

#35560: format(float(123), "00") causes segfault in debug builds
https://bugs.python.org/issue35560  closed by vstinner

#35568: Expose the C raise() function in the signal module, for use on
https://bugs.python.org/issue35568  closed by asvetlov

#35596: Fatal Python error: initfsencoding: unable to load the file sy
https://bugs.python.org/issue35596  closed by ned.deily

#35598: IDLE: Modernize config_key module
https://bugs.python.org/issue35598  closed by terry.reedy

#35616: Change references to '4.0'.
https://bugs.python.org/issue35616  closed by terry.reedy

#35629: hang and/or leaked processes with multiprocessing.Pool(...).im
https://bugs.python.org/issue35629  closed by pitrou

#35636: remove redundant check in unicode_hash(PyObject *self)
https://bugs.python.org/issue35636  closed by serhiy.storchaka

#35639: Lowecasing Unicode Characters
https://bugs.python.org/issue35639  closed by terry.reedy

#35651: PEP 257 (active) references PEP 258 (rejected) as if it were a
https://bugs.python.org/issue35651  closed by gvanrossum

#35659: Add heapremove() function to heapq
https://bugs.python.org/issue35659  closed by Wanja Chresta

#35660: IDLE: Fix imports in window.py
https://bugs.python.org/issue35660  closed by terry.reedy

#35664: Optimize itemgetter()
https://bugs.python.org/issue35664  closed by rhettinger

#35665: Function ssl.create_default_context raises exception on Window
https://bugs.python.org/issue35665  closed by pervlad

#35670: os functions return '??' for unicode characters in paths on wi
https://bugs.python.org/issue35670  closed by serhiy.storchaka

#35672: Error on divide
https://bugs.python.org/issue35672  closed by steven.daprano

#35678: subprocess.check_output(): OSError: [WinError 87]
https://bugs.python.org/issue35678  closed by MaximilianSP

#35682: asyncio: bug in _ProactorBasePipeTransport._force_close()
https://bugs.python.org/issue35682  closed by vstinner

#35684: Windows "embedded" Python downloads are malformed
https://bugs.python.org/issue35684  closed by serhiy.storchaka

#35694: missing modules on test suite
https://bugs.python.org/issue35694  closed by zach.ware

#35695: missing attributes
https://bugs.python.org/issue35695  closed by zach.ware

#35700: Place, Pack and Grid should return the widget
https://bugs.python.org/issue35700  closed by josh.r

#35702: clock_gettime: Add new identifier CLOCK_UPTIME_RAW for Darwin
https://bugs.python.org/issue35702  closed by vstinner

#35703: Underscores in numeric literals cannot be before or after deci
https://bugs.python.org/issue35703  closed by ronaldoussoren

#35709: test_ssl fails on Fedora 29: test_min_max_version()
https://bugs.python.org/issue35709  closed by vstinner

#35716: CLOCK_MONOTONIC_RAW available on macOS
https://bugs.python.org/issue35716  closed by vstinner

From ncoghlan at gmail.com  Mon Jan 14 07:39:46 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 14 Jan 2019 22:39:46 +1000
Subject: [Python-Dev] Interested in serving on Steering Council
In-Reply-To: <CAEbHw4b-dWKeYWaYoGBS1HMxJjuN0gquM99mKfuNvvU3wmwzGQ@mail.gmail.com>
References: <CAEbHw4bhjt3k-Q57BnSWw150B2Pint1465ick+775i-eaYeVBA@mail.gmail.com>
 <20190102191517.7fc2094c@fsol>
 <cb08756a-cf4b-f6ad-5147-7ef3e41f4c95@python.org>
 <CAEbHw4b-dWKeYWaYoGBS1HMxJjuN0gquM99mKfuNvvU3wmwzGQ@mail.gmail.com>
Message-ID: <CADiSq7eH5vkzhkiQN2pQWhKKq7GETqY9sf9JDY6b+iZ7kGDr3A@mail.gmail.com>

On Sat, 5 Jan 2019 at 06:28, David Mertz <mertz at gnosis.cx> wrote:
> It is interesting to me that whereas when I started volunteering for the PSF, there was significant overlap between the PSF board and the core-committers, I think there is little or no overlap today.  For better or worse, PSF is much more community than technical today.

(tangent) For those that are curious, I went and looked this up [1],
and the current PSF board has 2 of 11 members being core devs, whereas
the early days of the board look like they had a ratio that was more
often on the order of 4-6 core devs out of 7 members.

I take that as a sign of healthy community growth :)

Cheers,
Nick.

[1]  https://www.python.org/psf/records/board/history/

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Jan 14 07:52:06 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 14 Jan 2019 22:52:06 +1000
Subject: [Python-Dev] VxWorks and cpython?
In-Reply-To: <CA+3bQGEbOz4X7RbccRrXZn9fgJ3_ZagjV7hD-Aq-J605_rcJ7Q@mail.gmail.com>
References: <6C486AB80682E24095E3E77A408D68F201499C9AF5@ALA-MBD.corp.ad.wrs.com>
 <CAP7+vJLH2rBaqoMwYAJtC_gizPDqjWz=1ZUrEUyFYAVQ3NHrsQ@mail.gmail.com>
 <c33aa093-81cb-1c97-bb18-f86d78752d49@python.org>
 <CA+3bQGHAAYOOu88JErN8eTrOvkvW1GSr8E2QvHL2nu2Nkn3d8g@mail.gmail.com>
 <6C486AB80682E24095E3E77A408D68F201499CAF79@ALA-MBD.corp.ad.wrs.com>
 <CA+3bQGEbOz4X7RbccRrXZn9fgJ3_ZagjV7hD-Aq-J605_rcJ7Q@mail.gmail.com>
Message-ID: <CADiSq7dKbaj3MaLOCEomK_G_pjr=-A5LdHSoiTorhiOOrm1HMA@mail.gmail.com>

On Fri, 11 Jan 2019 at 03:23, Victor Stinner <vstinner at redhat.com> wrote:
> Le jeu. 10 janv. 2019 ? 17:54, Kuhl, Brian <brian.kuhl at windriver.com> a ?crit :
> > Do we set them up, and just let them fail, till enough PRs are accepted to make it build?
>
> Multiple buildbot workers are failing since many years. *I* would
> prefer to see the full test suite passing (even if some tests are
> skipped on your platform) before adding a buildbot, but it seems like
> some people have a different opinion on that. For example, there is an
> AIX buildbot and some tests are still failing (it was always red,
> failing, no?).

Michael Felt has been working towards getting that AIX bot green, and
I merged several PRs over the end of year break to help make progress
towards that goal.

He hit a similar challenge to the one Brian will be facing: putting
all the changes in one PR can make it overwhelming to review (so the
PR stalls), but splitting them up into individual PRs can result in
changes that don't appear sufficiently well motivated on their own (so
those PRs also stall).

The AIX case is currently an example where some of the buildbot
"fixes" are actually test skips, as the goal is to get to a point
where "things that currently work on AIX keep working, while folks
interested in AIX can work towards getting the not-working things
indicated by skipped tests also working".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From aixtools at felt.demon.nl  Tue Jan 15 04:06:42 2019
From: aixtools at felt.demon.nl (Michael)
Date: Tue, 15 Jan 2019 10:06:42 +0100
Subject: [Python-Dev] Thanks - come a long way with getting tests resolved!
Message-ID: <7d5a4f47-0c56-020e-1857-184bf78bc858@felt.demon.nl>

Many thanks to all who assisted with feedback. Back at the start of
August my make tests had a nasty block like this:

29 tests failed:
??? test__xxsubinterpreters test_array test_asyncio test_cmath
??? test_compile test_complex test_ctypes test_distutils test_embed
??? test_float test_fractions test_getargs2 test_httplib
??? test_httpservers test_imaplib test_importlib test_math test_poplib
??? test_shutil test_signal test_socket test_ssl test_statistics
??? test_strtod test_struct test_subprocess test_time test_timeout
??? test_utf8_mode

And now that is down to:


4 tests failed:
test_eintr test_importlib test_multiprocessing_forkserver
test_multiprocessing_spawn
While I am still trying to figure out where the "multiprocessing" errors
come from - there is
only one "test" left from the original list - and that one, plus
test_eintr have PR's waiting for your approval.
I could not have gotten this far without help!
Sincerely,
Michael aka aixtools

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190115/34187608/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190115/34187608/attachment.sig>

From solipsis at pitrou.net  Wed Jan 16 05:11:06 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 16 Jan 2019 11:11:06 +0100
Subject: [Python-Dev] pickle5 backport updated
Message-ID: <20190116111106.193088f4@fsol>


Hello,

For the record, the pickle5 backport (PEP 574) was updated to include
the latest pickle changes from CPython git master.

pickle5 is available for Python 3.6 and 3.7.
https://pypi.org/project/pickle5/

Regards

Antoine.



From vstinner at redhat.com  Wed Jan 16 05:25:04 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 16 Jan 2019 11:25:04 +0100
Subject: [Python-Dev] pickle5 backport updated
In-Reply-To: <20190116111106.193088f4@fsol>
References: <20190116111106.193088f4@fsol>
Message-ID: <CA+3bQGF1f8N7wMdz64RDh6kh2Vh-TUQ9rHn-T-g+BV_uYrKGRQ@mail.gmail.com>

I see that the PEP is still a draft. What's the status?

Do you need someone to review your PEP? Do you have anyone in mind who
can review it? I don't feel able to review such PEP, sorry :-(

Victor

Le mer. 16 janv. 2019 ? 11:14, Antoine Pitrou <solipsis at pitrou.net> a ?crit :
>
>
> Hello,
>
> For the record, the pickle5 backport (PEP 574) was updated to include
> the latest pickle changes from CPython git master.
>
> pickle5 is available for Python 3.6 and 3.7.
> https://pypi.org/project/pickle5/
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From wes.turner at gmail.com  Wed Jan 16 05:26:13 2019
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 16 Jan 2019 05:26:13 -0500
Subject: [Python-Dev] pickle5 backport updated
In-Reply-To: <20190116111106.193088f4@fsol>
References: <20190116111106.193088f4@fsol>
Message-ID: <CACfEFw8KH3p-BvYnONDX1zyX6gaFMPmnJqVaA+QXQNJjo8gyTw@mail.gmail.com>

Thanks! Apache Arrow may also be worth a look:

> Apache Arrow is a cross-language development platform for in-memory data.
It specifies a standardized language-independent columnar memory format for
flat and hierarchical data, organized for efficient analytic operations on
modern hardware. It also provides computational libraries and zero-copy
streaming messaging and interprocess communication.

> The Arrow Python bindings have first-class integration with NumPy,
pandas, and built-in Python objects

https://arrow.apache.org/docs/python/

Pickle supports arbitrary Python objects without any schema definition
(other than that what's defined in the object's __reduce__()
and __reduce_ex__())

On Wednesday, January 16, 2019, Antoine Pitrou <solipsis at pitrou.net> wrote:

>
> Hello,
>
> For the record, the pickle5 backport (PEP 574) was updated to include
> the latest pickle changes from CPython git master.
>
> pickle5 is available for Python 3.6 and 3.7.
> https://pypi.org/project/pickle5/
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> wes.turner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190116/431e45b3/attachment.html>

From wes.turner at gmail.com  Wed Jan 16 05:27:39 2019
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 16 Jan 2019 05:27:39 -0500
Subject: [Python-Dev] pickle5 backport updated
In-Reply-To: <CACfEFw8KH3p-BvYnONDX1zyX6gaFMPmnJqVaA+QXQNJjo8gyTw@mail.gmail.com>
References: <20190116111106.193088f4@fsol>
 <CACfEFw8KH3p-BvYnONDX1zyX6gaFMPmnJqVaA+QXQNJjo8gyTw@mail.gmail.com>
Message-ID: <CACfEFw86RjqQPPT9cjwQzuqPvkj02_zzhEG6k=qdrtnXaAM-kA@mail.gmail.com>

Maybe a bit OT: is there a way to *not pickle any callables*?

On Wednesday, January 16, 2019, Wes Turner <wes.turner at gmail.com> wrote:

> Thanks! Apache Arrow may also be worth a look:
>
> > Apache Arrow is a cross-language development platform for in-memory
> data. It specifies a standardized language-independent columnar memory
> format for flat and hierarchical data, organized for efficient analytic
> operations on modern hardware. It also provides computational libraries and
> zero-copy streaming messaging and interprocess communication.
>
> > The Arrow Python bindings have first-class integration with NumPy,
> pandas, and built-in Python objects
>
> https://arrow.apache.org/docs/python/
>
> Pickle supports arbitrary Python objects without any schema definition
> (other than that what's defined in the object's __reduce__()
> and __reduce_ex__())
>
> On Wednesday, January 16, 2019, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
>
>>
>> Hello,
>>
>> For the record, the pickle5 backport (PEP 574) was updated to include
>> the latest pickle changes from CPython git master.
>>
>> pickle5 is available for Python 3.6 and 3.7.
>> https://pypi.org/project/pickle5/
>>
>> Regards
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.
>> turner%40gmail.com
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190116/7ed0108e/attachment.html>

From solipsis at pitrou.net  Wed Jan 16 05:30:42 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 16 Jan 2019 11:30:42 +0100
Subject: [Python-Dev] pickle5 backport updated
In-Reply-To: <CACfEFw8KH3p-BvYnONDX1zyX6gaFMPmnJqVaA+QXQNJjo8gyTw@mail.gmail.com>
References: <20190116111106.193088f4@fsol>
 <CACfEFw8KH3p-BvYnONDX1zyX6gaFMPmnJqVaA+QXQNJjo8gyTw@mail.gmail.com>
Message-ID: <20190116113042.511b9631@fsol>

On Wed, 16 Jan 2019 05:26:13 -0500
Wes Turner <wes.turner at gmail.com> wrote:
> Thanks! Apache Arrow may also be worth a look:

From PEP 574, "implementation section":

? Support for pickle protocol 5 and out-of-band buffers was added to the
Apache Arrow Python bindings [10]. ?

So Arrow is very much in the scope for PEP 574.

Regards

Antoine.


PS: I work on Arrow.

From solipsis at pitrou.net  Wed Jan 16 05:29:21 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 16 Jan 2019 11:29:21 +0100
Subject: [Python-Dev] pickle5 backport updated
In-Reply-To: <CA+3bQGF1f8N7wMdz64RDh6kh2Vh-TUQ9rHn-T-g+BV_uYrKGRQ@mail.gmail.com>
References: <20190116111106.193088f4@fsol>
 <CA+3bQGF1f8N7wMdz64RDh6kh2Vh-TUQ9rHn-T-g+BV_uYrKGRQ@mail.gmail.com>
Message-ID: <20190116112921.40ba838c@fsol>

On Wed, 16 Jan 2019 11:25:04 +0100
Victor Stinner <vstinner at redhat.com> wrote:
> I see that the PEP is still a draft. What's the status?

PEP 574 is ready for pronouncement.  It's waiting for an authority to
approve it (or decide on a PEP delegate, who IMHO should probably
be Nick Coghlan).

Regards

Antoine.



> 
> Do you need someone to review your PEP? Do you have anyone in mind who
> can review it? I don't feel able to review such PEP, sorry :-(
> 
> Victor
> 
> Le mer. 16 janv. 2019 ? 11:14, Antoine Pitrou <solipsis at pitrou.net> a ?crit :
> >
> >
> > Hello,
> >
> > For the record, the pickle5 backport (PEP 574) was updated to include
> > the latest pickle changes from CPython git master.
> >
> > pickle5 is available for Python 3.6 and 3.7.
> > https://pypi.org/project/pickle5/
> >
> > Regards
> >
> > Antoine.
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com  
> 
> 
> 


From steve at pearwood.info  Thu Jan 17 05:26:06 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 17 Jan 2019 21:26:06 +1100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only way to
 get the address of an object?
Message-ID: <20190117102605.GG13616@ando.pearwood.info>

Disclaimer: I'm not a ctypes expert, so I might have this completely 
wrong. If so, I apologise for the noise.

The id() function is documented as returning an abstract ID number. In 
CPython, that happens to have been implemented as the address of the 
object.

I understand that the only way to pass the address of an object to 
ctypes is to use that id. Is that intentional?

As I see it, there is a conflict between two facts:

- that id() returns a memory address is an implementation detail; as 
such users should not rely on it, as the implementation could (in 
principle) change without notice;

- but users using ctypes have no choice but to rely on id() returning 
the object memory address, as of it were an offical part of the API.

Implementations like PyPy which emulate ctypes, while objects don't have 
fixed memory locations, will surely have a problem here. I don't know 
how PyPy solves this.

Have I misunderstood something here?



-- 
Steve

From solipsis at pitrou.net  Thu Jan 17 05:37:13 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 17 Jan 2019 11:37:13 +0100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
References: <20190117102605.GG13616@ando.pearwood.info>
Message-ID: <20190117113713.02a99b4d@fsol>

On Thu, 17 Jan 2019 21:26:06 +1100
Steven D'Aprano <steve at pearwood.info> wrote:
> Disclaimer: I'm not a ctypes expert, so I might have this completely 
> wrong. If so, I apologise for the noise.
> 
> The id() function is documented as returning an abstract ID number. In 
> CPython, that happens to have been implemented as the address of the 
> object.
> 
> I understand that the only way to pass the address of an object to 
> ctypes is to use that id. Is that intentional?

Can you explain in detail what you're doing?
If you're calling a C API taking a PyObject*, it seems like you should
be using ctypes.py_object as argument type specifier.  Various examples
can be found with Google.

Regards

Antoine.



From ncoghlan at gmail.com  Thu Jan 17 06:08:29 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 17 Jan 2019 21:08:29 +1000
Subject: [Python-Dev] pickle5 backport updated
In-Reply-To: <20190116112921.40ba838c@fsol>
References: <20190116111106.193088f4@fsol>
 <CA+3bQGF1f8N7wMdz64RDh6kh2Vh-TUQ9rHn-T-g+BV_uYrKGRQ@mail.gmail.com>
 <20190116112921.40ba838c@fsol>
Message-ID: <CADiSq7c2=dK2nTGemLnn9sCTrD_beL3ze_wDqbMkGvhO=tCx7A@mail.gmail.com>

On Wed, 16 Jan 2019 at 20:38, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> On Wed, 16 Jan 2019 11:25:04 +0100
> Victor Stinner <vstinner at redhat.com> wrote:
> > I see that the PEP is still a draft. What's the status?
>
> PEP 574 is ready for pronouncement.  It's waiting for an authority to
> approve it (or decide on a PEP delegate, who IMHO should probably
> be Nick Coghlan).

Aye, and I had already taken on that role unofficially - IIRC, the
only major change I requested was the one to allow buffer handling
callbacks to ask the pickler to include the buffer in-line after all
[1].

It was just that the BDFL-Delegate appointment process was put on hold
after Guido stepped down.

Cheers,
Nick.

[1] https://github.com/python/peps/commit/1ed3043948527893f277d6451a315a05bde123ec#diff-4576f13d94059bcccb9205ad5005f96c

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From eryksun at gmail.com  Thu Jan 17 08:50:51 2019
From: eryksun at gmail.com (eryk sun)
Date: Thu, 17 Jan 2019 07:50:51 -0600
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <20190117102605.GG13616@ando.pearwood.info>
References: <20190117102605.GG13616@ando.pearwood.info>
Message-ID: <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>

On 1/17/19, Steven D'Aprano <steve at pearwood.info> wrote:
>
> I understand that the only way to pass the address of an object to
> ctypes is to use that id. Is that intentional?

It's kind of dangerous to pass an object to C without an increment of
its reference count. The proper way is to use a simple pointer of type
"O" (object), which is already created for you as the "py_object"
type.

    >>> ctypes.py_object._type_
    'O'
    >>> ctypes.py_object.__bases__
    (<class '_ctypes._SimpleCData'>,)

It keeps a reference in the readonly _objects attribute. For example:

    >>> b = bytearray(b'spam')
    >>> sys.getrefcount(b)
    2
    >>> cb = ctypes.py_object(b)
    >>> sys.getrefcount(b)
    3
    >>> cb._objects
    bytearray(b'spam')
    >>> del cb
    >>> sys.getrefcount(b)
    2

If you need the address without relying on id(), cast to a void pointer:

    >>> ctypes.POINTER(ctypes.c_void_p)(cb)[0] == id(b)
    True

Or instantiate a c_void_p from the py_object as a buffer:

    >>> ctypes.c_void_p.from_buffer(cb).value == id(b)
    True

Note that ctypes.cast() doesn't work in this case. It's implemented as
an FFI function that takes the object address as a void pointer. The
from_param method of c_void_p doesn't support py_object:

    >>> ctypes.c_void_p.from_param(cb)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: wrong type

From greg at krypto.org  Thu Jan 17 19:48:38 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Thu, 17 Jan 2019 16:48:38 -0800
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <20190117102605.GG13616@ando.pearwood.info>
References: <20190117102605.GG13616@ando.pearwood.info>
Message-ID: <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>

I've heard that libraries using ctypes, cffi, or cython code of various
sorts in the real world wild today does abuse the unfortunate side effect
of CPython's implementation of id(). I don't have specific instances of
this in mind but trust what I've heard: that it is happening.

id() should never be considered to be the PyObject*.  In as much as code
shouldn't assume it is running on top of a specific CPython implementation.
If there is a _need_ to get a pointer to a C struct handle referencing a
CPython C API PyObject, we should make an explicit API for that rather than
the id() hack.  That way code can be explicit about its need, and code that
is just doing a funky form of identity tracking without using is and is not
can continue using id() without triggering regressive behavior on VMs that
don't have a CPython compatible PyObject under the hood by default.

[who uses id() anyways?]

-gps


On Thu, Jan 17, 2019 at 2:26 AM Steven D'Aprano <steve at pearwood.info> wrote:

> Disclaimer: I'm not a ctypes expert, so I might have this completely
> wrong. If so, I apologise for the noise.
>
> The id() function is documented as returning an abstract ID number. In
> CPython, that happens to have been implemented as the address of the
> object.
>
> I understand that the only way to pass the address of an object to
> ctypes is to use that id. Is that intentional?
>
> As I see it, there is a conflict between two facts:
>
> - that id() returns a memory address is an implementation detail; as
> such users should not rely on it, as the implementation could (in
> principle) change without notice;
>
> - but users using ctypes have no choice but to rely on id() returning
> the object memory address, as of it were an offical part of the API.
>
> Implementations like PyPy which emulate ctypes, while objects don't have
> fixed memory locations, will surely have a problem here. I don't know
> how PyPy solves this.
>
> Have I misunderstood something here?
>
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190117/388ca470/attachment.html>

From rosuav at gmail.com  Thu Jan 17 19:57:23 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 18 Jan 2019 11:57:23 +1100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
Message-ID: <CAPTjJmo5GM=HDxwyZDg7nK5wc82ROR425Wqgedow1eipwR5DQA@mail.gmail.com>

On Fri, Jan 18, 2019 at 11:50 AM Gregory P. Smith <greg at krypto.org> wrote:
>
> I've heard that libraries using ctypes, cffi, or cython code of various sorts in the real world wild today does abuse the unfortunate side effect of CPython's implementation of id(). I don't have specific instances of this in mind but trust what I've heard: that it is happening.
>
> id() should never be considered to be the PyObject*.  In as much as code shouldn't assume it is running on top of a specific CPython implementation.
> If there is a _need_ to get a pointer to a C struct handle referencing a CPython C API PyObject, we should make an explicit API for that rather than the id() hack.  That way code can be explicit about its need, and code that is just doing a funky form of identity tracking without using is and is not can continue using id() without triggering regressive behavior on VMs that don't have a CPython compatible PyObject under the hood by default.
>

I would be strongly in favour of ctypes gaining a "get address of
object" function, which happens (in current CPythons) to return the
same value as id() does, but is specifically tied to ctypes.

ChrisA

From python at mrabarnett.plus.com  Thu Jan 17 22:00:54 2019
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 18 Jan 2019 03:00:54 +0000
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
Message-ID: <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>

On 2019-01-18 00:48, Gregory P. Smith wrote:
> I've heard that libraries using ctypes, cffi, or cython code of various 
> sorts in the real world wild today does abuse the unfortunate side 
> effect of CPython's implementation of id(). I don't have specific 
> instances of this in mind but trust what I've heard: that it is happening.
> 
> id() should never be considered to be the PyObject*.? In as much as code 
> shouldn't assume it is running on top of a specific CPython implementation.
> If there is a _need_ to get a pointer to a C struct handle referencing a 
> CPython C API PyObject, we should make an explicit API for that rather 
> than the id() hack.? That way code can be explicit about its need, and 
> code that is just doing a funky form of identity tracking without using 
> is and is not can continue using id() without triggering regressive 
> behavior on VMs that don't have a CPython compatible PyObject under the 
> hood by default.
> 
> [who uses id() anyways?]
> 
I use it in some of my code.

If I want to cache some objects, I put them in a dict, using the id as 
the key. If I wanted to locate an object in a cache and didn't have 
id(), I'd have to do a linear search for it.

From njs at pobox.com  Thu Jan 17 22:18:37 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 17 Jan 2019 19:18:37 -0800
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
Message-ID: <CAPJVwBkA3OBVHgi=VyS0fd+1+V2u=6OmBxwzZ7OpfyQB4fAdTg@mail.gmail.com>

n Thu, Jan 17, 2019 at 4:51 PM Gregory P. Smith <greg at krypto.org> wrote:
>
> I've heard that libraries using ctypes, cffi, or cython code of various sorts in the real world wild today does abuse the unfortunate side effect of CPython's implementation of id(). I don't have specific instances of this in mind but trust what I've heard: that it is happening.

IME it's reasonably common with ctypes, for cases where you need to do
some gross hack and there's no other option. Here's an example in
jinja2:

  https://github.com/pallets/jinja/blob/9fe9520f2daa1df6079b188adba758d6e03d6af2/jinja2/debug.py#L350

I haven't seen it with cffi or cython. (cffi explicitly doesn't
provide any way to access the CPython C API, and in cython you can
just cast an object to a pointer.)

> id() should never be considered to be the PyObject*.  In as much as code shouldn't assume it is running on top of a specific CPython implementation.
> If there is a _need_ to get a pointer to a C struct handle referencing a CPython C API PyObject, we should make an explicit API for that rather than the id() hack.  That way code can be explicit about its need, and code that is just doing a funky form of identity tracking without using is and is not can continue using id() without triggering regressive behavior on VMs that don't have a CPython compatible PyObject under the hood by default.

Using id() like this is certainly offensive to our sensibilities, but
in practice I don't see how it causes much harm. If you are doing
*anything* with PyObject*, then you're tying yourself to
implementation details of CPython (and usually a specific version of
CPython). That's not great, but at that point relying on CPython's
implementation of id() is the least of your worries, and it tends to
be a self-correcting problem.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From steve.dower at python.org  Fri Jan 18 01:09:36 2019
From: steve.dower at python.org (Steve Dower)
Date: Thu, 17 Jan 2019 22:09:36 -0800
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
Message-ID: <c070073d-cc15-3583-2458-14e25e95c5ca@python.org>

For everyone who managed to reply *hours* after Eryk Sun posted the
correct answer and still get it wrong, here it is again in full.

As a bonus, here's a link to the place where this answer appears in the
documentation:
https://docs.python.org/3/library/ctypes.html#ctypes.py_object

Cheers,
Steve

On 17Jan.2019 0550, eryk sun wrote:
> On 1/17/19, Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> I understand that the only way to pass the address of an object to
>> ctypes is to use that id. Is that intentional?
> 
> It's kind of dangerous to pass an object to C without an increment of
> its reference count. The proper way is to use a simple pointer of type
> "O" (object), which is already created for you as the "py_object"
> type.
> 
>     >>> ctypes.py_object._type_
>     'O'
>     >>> ctypes.py_object.__bases__
>     (<class '_ctypes._SimpleCData'>,)
> 
> It keeps a reference in the readonly _objects attribute. For example:
> 
>     >>> b = bytearray(b'spam')
>     >>> sys.getrefcount(b)
>     2
>     >>> cb = ctypes.py_object(b)
>     >>> sys.getrefcount(b)
>     3
>     >>> cb._objects
>     bytearray(b'spam')
>     >>> del cb
>     >>> sys.getrefcount(b)
>     2
> 
> If you need the address without relying on id(), cast to a void pointer:
> 
>     >>> ctypes.POINTER(ctypes.c_void_p)(cb)[0] == id(b)
>     True
> 
> Or instantiate a c_void_p from the py_object as a buffer:
> 
>     >>> ctypes.c_void_p.from_buffer(cb).value == id(b)
>     True
> 
> Note that ctypes.cast() doesn't work in this case. It's implemented as
> an FFI function that takes the object address as a void pointer. The
> from_param method of c_void_p doesn't support py_object:
> 
>     >>> ctypes.c_void_p.from_param(cb)
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>     TypeError: wrong type

From steve.dower at python.org  Fri Jan 18 01:18:13 2019
From: steve.dower at python.org (Steve Dower)
Date: Thu, 17 Jan 2019 22:18:13 -0800
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <c070073d-cc15-3583-2458-14e25e95c5ca@python.org>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
 <c070073d-cc15-3583-2458-14e25e95c5ca@python.org>
Message-ID: <38dc18e0-b7e7-c0a4-c4a5-31b0797befb9@python.org>

I feel like I should clarify - not everyone who posted got it wrong, and
I understand there's a side discussion among those who are also
interested/participants in
https://discuss.python.org/t/demoting-the-is-operator-to-avoid-an-identity-crisis/86/
- but there was no of acknowledgement of Eryk Sun's correct and useful
answer which I find very disappointing and a great way to discourage
contributions.

We can, and should, do better, at least by thanking the person for their
response before running down a barely related side track.

On 17Jan.2019 2209, Steve Dower wrote:
> For everyone who managed to reply *hours* after Eryk Sun posted the
> correct answer and still get it wrong, here it is again in full.
> 
> As a bonus, here's a link to the place where this answer appears in the
> documentation:
> https://docs.python.org/3/library/ctypes.html#ctypes.py_object
> 
> Cheers,
> Steve
> 
> On 17Jan.2019 0550, eryk sun wrote:
>> On 1/17/19, Steven D'Aprano <steve at pearwood.info> wrote:
>>>
>>> I understand that the only way to pass the address of an object to
>>> ctypes is to use that id. Is that intentional?
>>
>> It's kind of dangerous to pass an object to C without an increment of
>> its reference count. The proper way is to use a simple pointer of type
>> "O" (object), which is already created for you as the "py_object"
>> type.
>>
>>     >>> ctypes.py_object._type_
>>     'O'
>>     >>> ctypes.py_object.__bases__
>>     (<class '_ctypes._SimpleCData'>,)
>>
>> It keeps a reference in the readonly _objects attribute. For example:
>>
>>     >>> b = bytearray(b'spam')
>>     >>> sys.getrefcount(b)
>>     2
>>     >>> cb = ctypes.py_object(b)
>>     >>> sys.getrefcount(b)
>>     3
>>     >>> cb._objects
>>     bytearray(b'spam')
>>     >>> del cb
>>     >>> sys.getrefcount(b)
>>     2
>>
>> If you need the address without relying on id(), cast to a void pointer:
>>
>>     >>> ctypes.POINTER(ctypes.c_void_p)(cb)[0] == id(b)
>>     True
>>
>> Or instantiate a c_void_p from the py_object as a buffer:
>>
>>     >>> ctypes.c_void_p.from_buffer(cb).value == id(b)
>>     True
>>
>> Note that ctypes.cast() doesn't work in this case. It's implemented as
>> an FFI function that takes the object address as a void pointer. The
>> from_param method of c_void_p doesn't support py_object:
>>
>>     >>> ctypes.c_void_p.from_param(cb)
>>     Traceback (most recent call last):
>>       File "<stdin>", line 1, in <module>
>>     TypeError: wrong type


From njs at pobox.com  Fri Jan 18 03:18:17 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 18 Jan 2019 00:18:17 -0800
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <c070073d-cc15-3583-2458-14e25e95c5ca@python.org>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
 <c070073d-cc15-3583-2458-14e25e95c5ca@python.org>
Message-ID: <CAPJVwBnqY=ea8v5BimBYfgcXpkmwQEFZnBvvJ3D8qTD5oh1bNw@mail.gmail.com>

On Thu, Jan 17, 2019, 22:11 Steve Dower <steve.dower at python.org wrote:

> For everyone who managed to reply *hours* after Eryk Sun posted the
> correct answer and still get it wrong, here it is again in full.
>
> As a bonus, here's a link to the place where this answer appears in the
> documentation:
> https://docs.python.org/3/library/ctypes.html#ctypes.py_object


Eryk's answer is actually much more useful than the documentation. I've
read that documentation many times, but always decided not to use py_object
because I couldn't figure out what it would actually do...

(I still probably won't use it because IME by the time I'm using ctypes and
PyObject* together I usually need manual control over refcounts, but it's
nice to know what it actually does.)

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190118/c1756945/attachment.html>

From steve at pearwood.info  Fri Jan 18 04:49:26 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 18 Jan 2019 20:49:26 +1100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <20190117113713.02a99b4d@fsol>
References: <20190117102605.GG13616@ando.pearwood.info>
 <20190117113713.02a99b4d@fsol>
Message-ID: <20190118094926.GL13616@ando.pearwood.info>

On Thu, Jan 17, 2019 at 11:37:13AM +0100, Antoine Pitrou wrote:

I said:

> > The id() function is documented as returning an abstract ID number. In 
> > CPython, that happens to have been implemented as the address of the 
> > object.
> > 
> > I understand that the only way to pass the address of an object to 
> > ctypes is to use that id. Is that intentional?


Antoine:
 
> Can you explain in detail what you're doing?

Code-wise, I'm not doing anything with ctypes.

Language-wise, I'm trying to get a definitive answer of whether or not 
id() returning the address of the object should be a guaranteed feature 
or not.

Across the entire Python ecosystem, no it isn't, as Jython and 
IronPython return consecutive integers. But should we consider it an 
intentional part of the CPython API?

There are developers who insist that when it comes to CPython, id() 
returning the object address is an intentional feature that they can and 
do rely on, because (so I was told by one of them) that using id() is 
the only way to get the address of an object from pure-Python.

According to this claim, using id() to get the address for use in ctypes 
is the correct and only way to do it, and this is a deliberate design 
choice by the core devs rather than an accident of the implementation. 
So long as you know you are using CPython, this is (so I was told) 
completely safe.

In the grand scheme of things this may be a pretty minor issue. But I 
suspect that it could be a pain point for implementations like PyPy that 
support both objects that move and a ctypes emulation.



-- 
Steve

From p.f.moore at gmail.com  Fri Jan 18 05:15:49 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 18 Jan 2019 10:15:49 +0000
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <20190118094926.GL13616@ando.pearwood.info>
References: <20190117102605.GG13616@ando.pearwood.info>
 <20190117113713.02a99b4d@fsol>
 <20190118094926.GL13616@ando.pearwood.info>
Message-ID: <CACac1F_Horz9rTXhV2271YN-hhrHEjsRukoC0GsuLjsLQp6vmA@mail.gmail.com>

On Fri, 18 Jan 2019 at 09:52, Steven D'Aprano <steve at pearwood.info> wrote:
> Code-wise, I'm not doing anything with ctypes.
>
> Language-wise, I'm trying to get a definitive answer of whether or not
> id() returning the address of the object should be a guaranteed feature
> or not.
>
> Across the entire Python ecosystem, no it isn't, as Jython and
> IronPython return consecutive integers. But should we consider it an
> intentional part of the CPython API?
>
> There are developers who insist that when it comes to CPython, id()
> returning the object address is an intentional feature that they can and
> do rely on, because (so I was told by one of them) that using id() is
> the only way to get the address of an object from pure-Python.
>
> According to this claim, using id() to get the address for use in ctypes
> is the correct and only way to do it, and this is a deliberate design
> choice by the core devs rather than an accident of the implementation.
> So long as you know you are using CPython, this is (so I was told)
> completely safe.
>
> In the grand scheme of things this may be a pretty minor issue. But I
> suspect that it could be a pain point for implementations like PyPy that
> support both objects that move and a ctypes emulation.

As per Eryk Sun's reply, the "correct" way to get an object address is
by using ctypes.py_object.

Supporting py_object may be a pain point for other implementations
that emulate ctypes, but then again, so is supporting the whole
CPython C API (which is where the py_object type is needed, so it's
basically the same problem).

So to answer your question, I'd say that no, id() returning the object
address is not, and should not be, a guaranteed aspect of CPython, and
the motivating issue of ctypes is solved within ctypes itself by using
py_object.

Paul

From steve at pearwood.info  Fri Jan 18 05:17:56 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 18 Jan 2019 21:17:56 +1100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
Message-ID: <20190118101756.GM13616@ando.pearwood.info>

On Thu, Jan 17, 2019 at 04:48:38PM -0800, Gregory P. Smith wrote:

> I've heard that libraries using ctypes, cffi, or cython code of various
> sorts in the real world wild today does abuse the unfortunate side effect
> of CPython's implementation of id(). I don't have specific instances of
> this in mind but trust what I've heard: that it is happening.

Indeed -- I've been told by one developer in no uncertain terms that 
using id() in this fashion is the only way to get the address of an 
object for use in ctypes. I don't know enough about ctypes to judge 
whether that is correct or not.

The sample code I've been shown is this:

    pointer_to_obj = id(obj)
    from_deref = ctypes.cast(pointer_to_obj, ctypes.py_object).value
    from_deref is obj  # True


> id() should never be considered to be the PyObject*.  In as much as code
> shouldn't assume it is running on top of a specific CPython implementation.
> If there is a _need_ to get a pointer to a C struct handle referencing a
> CPython C API PyObject, we should make an explicit API for that rather than
> the id() hack.  That way code can be explicit about its need, and code that
> is just doing a funky form of identity tracking without using is and is not
> can continue using id() without triggering regressive behavior on VMs that
> don't have a CPython compatible PyObject under the hood by default.

+1 to all of this.



-- 
Steve

From njs at pobox.com  Fri Jan 18 05:31:00 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 18 Jan 2019 02:31:00 -0800
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <20190118094926.GL13616@ando.pearwood.info>
References: <20190117102605.GG13616@ando.pearwood.info>
 <20190117113713.02a99b4d@fsol>
 <20190118094926.GL13616@ando.pearwood.info>
Message-ID: <CAPJVwB=3w1Aid59kwRxNy3AwYePeKOYGtYnD-HrOxC__HMM5Dw@mail.gmail.com>

On Fri, Jan 18, 2019 at 1:51 AM Steven D'Aprano <steve at pearwood.info> wrote:
> Across the entire Python ecosystem, no it isn't, as Jython and
> IronPython return consecutive integers. But should we consider it an
> intentional part of the CPython API?

It's always worked, there's substantial code in the wild that depends
on it, and AFAICT it doesn't cause any real harm, so to me it seems
like the only possible conclusion is that CPython will continue to
guarantee this.

For this argument I don't think it matters whether it was originally
intentional, or whether there's some other alternative that people
could use in theory.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From steve at pearwood.info  Fri Jan 18 05:51:09 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 18 Jan 2019 21:51:09 +1100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
Message-ID: <20190118105108.GN13616@ando.pearwood.info>

Thanks for the detailed answer. A further question below.


On Thu, Jan 17, 2019 at 07:50:51AM -0600, eryk sun wrote:
> On 1/17/19, Steven D'Aprano <steve at pearwood.info> wrote:
> >
> > I understand that the only way to pass the address of an object to
> > ctypes is to use that id. Is that intentional?
> 
> It's kind of dangerous to pass an object to C without an increment of
> its reference count.

"Kind of dangerous?" How dangerous?

If I am reading this correctly, I think you are saying that using id() 
in this way is never(?) correct.



-- 
Steve

From solipsis at pitrou.net  Fri Jan 18 05:52:31 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 18 Jan 2019 11:52:31 +0100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
References: <20190117102605.GG13616@ando.pearwood.info>
 <20190117113713.02a99b4d@fsol>
 <20190118094926.GL13616@ando.pearwood.info>
Message-ID: <20190118115231.597defab@fsol>

On Fri, 18 Jan 2019 20:49:26 +1100
Steven D'Aprano <steve at pearwood.info> wrote:
> 
> Language-wise, I'm trying to get a definitive answer of whether or not 
> id() returning the address of the object should be a guaranteed feature 
> or not.

For me, the definitive answer is "yes, it's a CPython feature".

However, it's obviously not a PyPy feature, and I'm not sure about other
implementations.  Anything with an object model that can eliminate
in-memory objects in favour of in-register values (for example using
tagged pointers or type specialization + lifetime analysis) is obviously
not able to hold the promise that id() returns the /address/ of the
"object".

That doesn't mean the CPython feature has to live forever.  We may want
to deprecate it at some point (though it's not obvious how to warn the
user: just because you're using id() doesn't mean you're interested in
the actual /address/, rather than some arbitrary unique id).

> According to this claim, using id() to get the address for use in ctypes 
> is the correct and only way to do it

I don't know why you keep repeating that.  You were already explained
that it's /not/ the correct and only way to get the address for use in
ctypes.

Regards

Antoine.



From solipsis at pitrou.net  Fri Jan 18 05:55:35 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 18 Jan 2019 11:55:35 +0100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
References: <20190117102605.GG13616@ando.pearwood.info>
 <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
 <c070073d-cc15-3583-2458-14e25e95c5ca@python.org>
 <CAPJVwBnqY=ea8v5BimBYfgcXpkmwQEFZnBvvJ3D8qTD5oh1bNw@mail.gmail.com>
Message-ID: <20190118115535.21434e43@fsol>

On Fri, 18 Jan 2019 00:18:17 -0800
Nathaniel Smith <njs at pobox.com> wrote:
> On Thu, Jan 17, 2019, 22:11 Steve Dower <steve.dower at python.org wrote:
> 
> > For everyone who managed to reply *hours* after Eryk Sun posted the
> > correct answer and still get it wrong, here it is again in full.
> >
> > As a bonus, here's a link to the place where this answer appears in the
> > documentation:
> > https://docs.python.org/3/library/ctypes.html#ctypes.py_object  
> 
> 
> Eryk's answer is actually much more useful than the documentation. I've
> read that documentation many times, but always decided not to use py_object
> because I couldn't figure out what it would actually do...

+1

Needless to say, this is an opportunity to improve the documentation ;-)

Regards

Antoine.



From solipsis at pitrou.net  Fri Jan 18 05:57:08 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 18 Jan 2019 11:57:08 +0100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>
Message-ID: <20190118115708.0607e34d@fsol>

On Fri, 18 Jan 2019 03:00:54 +0000
MRAB <python at mrabarnett.plus.com> wrote:
> On 2019-01-18 00:48, Gregory P. Smith wrote:
> > I've heard that libraries using ctypes, cffi, or cython code of various 
> > sorts in the real world wild today does abuse the unfortunate side 
> > effect of CPython's implementation of id(). I don't have specific 
> > instances of this in mind but trust what I've heard: that it is happening.
> > 
> > id() should never be considered to be the PyObject*.? In as much as code 
> > shouldn't assume it is running on top of a specific CPython implementation.
> > If there is a _need_ to get a pointer to a C struct handle referencing a 
> > CPython C API PyObject, we should make an explicit API for that rather 
> > than the id() hack.? That way code can be explicit about its need, and 
> > code that is just doing a funky form of identity tracking without using 
> > is and is not can continue using id() without triggering regressive 
> > behavior on VMs that don't have a CPython compatible PyObject under the 
> > hood by default.
> > 
> > [who uses id() anyways?]
> >   
> I use it in some of my code.
> 
> If I want to cache some objects, I put them in a dict, using the id as 
> the key. If I wanted to locate an object in a cache and didn't have 
> id(), I'd have to do a linear search for it.

Indeed.  I've used it for the same purpose in the past (identity-dict).

Regards

Antoine.



From solipsis at pitrou.net  Fri Jan 18 05:54:21 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 18 Jan 2019 11:54:21 +0100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
References: <20190117102605.GG13616@ando.pearwood.info>
 <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
 <c070073d-cc15-3583-2458-14e25e95c5ca@python.org>
 <38dc18e0-b7e7-c0a4-c4a5-31b0797befb9@python.org>
Message-ID: <20190118115421.07a38997@fsol>

On Thu, 17 Jan 2019 22:18:13 -0800
Steve Dower <steve.dower at python.org> wrote:
> I feel like I should clarify - not everyone who posted got it wrong, and
> I understand there's a side discussion among those who are also
> interested/participants in
> https://discuss.python.org/t/demoting-the-is-operator-to-avoid-an-identity-crisis/86/
> - but there was no of acknowledgement of Eryk Sun's correct and useful
> answer which I find very disappointing and a great way to discourage
> contributions.
> 
> We can, and should, do better, at least by thanking the person for their
> response before running down a barely related side track.

I can certainly thank Eryk for posting a much better answer than mine.

Regards

Antoine.



From steve at pearwood.info  Fri Jan 18 06:11:36 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 18 Jan 2019 22:11:36 +1100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <c070073d-cc15-3583-2458-14e25e95c5ca@python.org>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
 <c070073d-cc15-3583-2458-14e25e95c5ca@python.org>
Message-ID: <20190118111136.GO13616@ando.pearwood.info>

On Thu, Jan 17, 2019 at 10:09:36PM -0800, Steve Dower wrote:
> For everyone who managed to reply *hours* after Eryk Sun posted the
> correct answer and still get it wrong, here it is again in full.

Sorry, I'm confused by your response here. As far as I can see, nobody 
except Eryk Sun gave any technical details about how to correctly pass 
objects to ctypes, so I'm not sure what sense of "get it wrong" you 
mean.

A couple of people offered the opinion that we ought to offer an 
explicit ctypes API for getting the address of an object, decoupling 
that functionality from id(). Do you mean "wrong" in the sense that such 
an API would be unnecessary, given the existing solution Eryk Sun 
quoted?



> As a bonus, here's a link to the place where this answer appears in the
> documentation:
> https://docs.python.org/3/library/ctypes.html#ctypes.py_object

Thanks for the link, that's useful.



-- 
Steve

From walter at livinglogic.de  Fri Jan 18 06:46:45 2019
From: walter at livinglogic.de (Walter =?utf-8?q?D=C3=B6rwald?=)
Date: Fri, 18 Jan 2019 12:46:45 +0100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <20190118115708.0607e34d@fsol>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>
 <20190118115708.0607e34d@fsol>
Message-ID: <019B121A-5292-4E85-BDA3-814F029EA397@livinglogic.de>

On 18 Jan 2019, at 11:57, Antoine Pitrou wrote:

> On Fri, 18 Jan 2019 03:00:54 +0000
> MRAB <python at mrabarnett.plus.com> wrote:
>> On 2019-01-18 00:48, Gregory P. Smith wrote:
>>> I've heard that libraries using ctypes, cffi, or cython code of 
>>> various
>>> sorts in the real world wild today does abuse the unfortunate side
>>> effect of CPython's implementation of id(). I don't have specific
>>> instances of this in mind but trust what I've heard: that it is 
>>> happening.
>>>
>>> id() should never be considered to be the PyObject*.? In as much as 
>>> code
>>> shouldn't assume it is running on top of a specific CPython 
>>> implementation.
>>> If there is a _need_ to get a pointer to a C struct handle 
>>> referencing a
>>> CPython C API PyObject, we should make an explicit API for that 
>>> rather
>>> than the id() hack.? That way code can be explicit about its need, 
>>> and
>>> code that is just doing a funky form of identity tracking without 
>>> using
>>> is and is not can continue using id() without triggering regressive
>>> behavior on VMs that don't have a CPython compatible PyObject under 
>>> the
>>> hood by default.
>>>
>>> [who uses id() anyways?]
>>>
>> I use it in some of my code.
>>
>> If I want to cache some objects, I put them in a dict, using the id 
>> as
>> the key. If I wanted to locate an object in a cache and didn't have
>> id(), I'd have to do a linear search for it.
>
> Indeed.  I've used it for the same purpose in the past 
> (identity-dict).

Its useful in all situations where you do topology preserving 
transformations, for example pickling (i.e. object serialization) or a 
deep copy of some object structures.

In these cases you need a way to record and quickly detect whether 
you've handled a specific object before. In Python we can do that with a 
dictionary that has object ids as keys. Java provides IdentityHashMap 
for that. Javascript provides neither, so deep-copying objects in 
Javascript seems to be impossible.

> Regards
>
> Antoine.

Servus,
    Walter

From mertz at gnosis.cx  Fri Jan 18 10:29:44 2019
From: mertz at gnosis.cx (David Mertz)
Date: Fri, 18 Jan 2019 10:29:44 -0500
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <20190118115231.597defab@fsol>
References: <20190117102605.GG13616@ando.pearwood.info>
 <20190117113713.02a99b4d@fsol>
 <20190118094926.GL13616@ando.pearwood.info> <20190118115231.597defab@fsol>
Message-ID: <CAEbHw4Zb6xXfOeyBtn6y4FwMBFC9FEJxQg=gutWWBNJ0jJ2vCg@mail.gmail.com>

On Fri, Jan 18, 2019, 5:55 AM Antoine Pitrou <solipsis at pitrou.net wrote:

>
> > id() returning the address of the object should be a guaranteed feature
>
> For me, the definitive answer is "yes, it's a CPython feature".
> That doesn't mean the CPython feature has to live forever.  We may want
> to deprecate it at some point


Whenever I've taught Python (quite a bit between writing, in person, and
webinars), I have been very explicit in stating that id(obj) returns some
unique number for each object, and mentioned that for MANY Python objects
CPython users an implementation convenience of using the memory address.

Every time I've explained it I've said not to rely on that implementation
detail. It's not true for small integers, for example, even in CPython.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190118/12b2cdf8/attachment.html>

From mertz at gnosis.cx  Fri Jan 18 10:47:38 2019
From: mertz at gnosis.cx (David Mertz)
Date: Fri, 18 Jan 2019 10:47:38 -0500
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <CAEbHw4Zb6xXfOeyBtn6y4FwMBFC9FEJxQg=gutWWBNJ0jJ2vCg@mail.gmail.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <20190117113713.02a99b4d@fsol>
 <20190118094926.GL13616@ando.pearwood.info> <20190118115231.597defab@fsol>
 <CAEbHw4Zb6xXfOeyBtn6y4FwMBFC9FEJxQg=gutWWBNJ0jJ2vCg@mail.gmail.com>
Message-ID: <CAEbHw4Z-mufSpRCwS2MHo0Dd2_xbrzh2m9jNEHf+Dt2ihUsv8Q@mail.gmail.com>

Oh, bracket my brain glitch on small integers.  Yes, they still give id()
of memory address, they just get reused, which is different.  Nonetheless,
I never teach id(obj)
== ctypes.c_void_p.from_buffer(ctypes.py_object(b)).value ... and not only
because I only learned the latter spelling from eryk sun.

On Fri, Jan 18, 2019 at 10:29 AM David Mertz <mertz at gnosis.cx> wrote:

> On Fri, Jan 18, 2019, 5:55 AM Antoine Pitrou <solipsis at pitrou.net wrote:
>
>>
>> > id() returning the address of the object should be a guaranteed feature
>>
>> For me, the definitive answer is "yes, it's a CPython feature".
>> That doesn't mean the CPython feature has to live forever.  We may want
>> to deprecate it at some point
>
>
> Whenever I've taught Python (quite a bit between writing, in person, and
> webinars), I have been very explicit in stating that id(obj) returns some
> unique number for each object, and mentioned that for MANY Python objects
> CPython users an implementation convenience of using the memory address.
>
> Every time I've explained it I've said not to rely on that implementation
> detail. It's not true for small integers, for example, even in CPython.
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190118/ca1d0146/attachment.html>

From status at bugs.python.org  Fri Jan 18 13:08:12 2019
From: status at bugs.python.org (Python tracker)
Date: Fri, 18 Jan 2019 18:08:12 +0000
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190118180812.1.30E23CBDE928F183@roundup.psfhosted.org>


ACTIVITY SUMMARY (2019-01-11 - 2019-01-18)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6932 ( +0)
  closed 40595 (+58)
  total  47527 (+58)

Open issues with patches: 2766 


Issues opened (43)
==================

#35701: [uuid] 3.8 breaks weak references for UUIDs
https://bugs.python.org/issue35701  reopened by serhiy.storchaka

#35720: Memory leak in Modules/main.c:pymain_parse_cmdline_impl when u
https://bugs.python.org/issue35720  opened by Lucas Cimon

#35721: _UnixSubprocessTransport leaks socket pair if Popen fails
https://bugs.python.org/issue35721  opened by niklasf

#35722: disable_existing_loggers does not apply to the root logger
https://bugs.python.org/issue35722  opened by maggyero

#35723: Add "time zone index" cache to datetime objects
https://bugs.python.org/issue35723  opened by p-ganssle

#35724: Check for main interpreter when checking for "main" thread (fo
https://bugs.python.org/issue35724  opened by eric.snow

#35726: QueueHandler formating affects other handlers
https://bugs.python.org/issue35726  opened by David Ruggles

#35727: sys.exit() in a multiprocessing.Process does not align with Py
https://bugs.python.org/issue35727  opened by chrahunt

#35728: Tkinter font nametofont requires default root
https://bugs.python.org/issue35728  opened by terry.reedy

#35731: Modify to support multiple urls in webbrowser.open
https://bugs.python.org/issue35731  opened by arlenyu

#35733: isinstance(ast.Constant(value=True), ast.Num) should be False
https://bugs.python.org/issue35733  opened by Anthony Sottile

#35736: [xml.minidom] Missing component in table after getElementsByTa
https://bugs.python.org/issue35736  opened by MiKr41

#35737: crypt AuthenticationError introduced with new Linux kernel
https://bugs.python.org/issue35737  opened by icycle

#35739: Enable verbose of tests during PGO build on amd64 platforms
https://bugs.python.org/issue35739  opened by neyuru

#35740: openssl version 1.1.1 need to be there in cpython-source-deps 
https://bugs.python.org/issue35740  opened by ossdev07

#35741: unittest.skipUnless(time._STRUCT_TM_ITEMS == 11, "needs tm_zon
https://bugs.python.org/issue35741  opened by jianxu3

#35745: Add import statement in dataclass code snippet
https://bugs.python.org/issue35745  opened by Windson Yang

#35746: [ssl][CVE-2019-5010] TALOS-2018-0758 Denial of Service
https://bugs.python.org/issue35746  opened by Talos

#35748: urlparse library detecting wrong hostname leads to open redire
https://bugs.python.org/issue35748  opened by nsonaniya2010

#35749: Rewrite asyncio signal handler
https://bugs.python.org/issue35749  opened by asvetlov

#35751: traceback.clear_frames manages to deadlock a background task
https://bugs.python.org/issue35751  opened by tvoinarovskyi

#35752: test_buffer fails on ppc64le: memoryview pack_single() is misc
https://bugs.python.org/issue35752  opened by vstinner

#35754: When writing/closing a closed Popen.stdin, I get OSError vs. B
https://bugs.python.org/issue35754  opened by jimbo1qaz_

#35755: Remove current directory from posixpath.defpath to enhance sec
https://bugs.python.org/issue35755  opened by vstinner

#35756: Using `return value` in a generator function skips the returne
https://bugs.python.org/issue35756  opened by Bryan Koch

#35758: Disable x87 control word for MSVC ARM compiler
https://bugs.python.org/issue35758  opened by Minmin.Gong

#35759: inspect module does not implement introspection API for asynch
https://bugs.python.org/issue35759  opened by tkren

#35760: test_asyncio: test_async_gen_asyncio_gc_aclose_09() race condi
https://bugs.python.org/issue35760  opened by vstinner

#35761: Allow dataclasses to be updated in place
https://bugs.python.org/issue35761  opened by theophile

#35762: subprocess.Popen with universal_newlines and nonblocking strea
https://bugs.python.org/issue35762  opened by sambayer

#35763: IDLE calltips: make positional note less obtrusive
https://bugs.python.org/issue35763  opened by terry.reedy

#35764: IDLE: revise calltip doc
https://bugs.python.org/issue35764  opened by terry.reedy

#35765: Document references object x but doesn't show it in the exampl
https://bugs.python.org/issue35765  opened by Patrick Rice

#35766: Merge typed_ast back into CPython
https://bugs.python.org/issue35766  opened by gvanrossum

#35767: unittest loader doesn't work with partial test functions
https://bugs.python.org/issue35767  opened by fried

#35768: IDLE: Auto measure font fixed pitch characteristics
https://bugs.python.org/issue35768  opened by terry.reedy

#35769: IDLE: change new file name from  ''Untitled" to "untitled"
https://bugs.python.org/issue35769  opened by terry.reedy

#35770: IDLE: python -m idlelib fails on master on Mac OS 10.10.4
https://bugs.python.org/issue35770  opened by xtreak

#35771: IDLE: Fix tooltip Hovertiptest failure
https://bugs.python.org/issue35771  opened by terry.reedy

#35772: test_tarfile fails on ppc64le when using tmpfs filesystem
https://bugs.python.org/issue35772  opened by vstinner

#35773: test_bdb fails on AIX bot (regression)
https://bugs.python.org/issue35773  opened by Michael.Felt

#35774: ASAN, memory leak
https://bugs.python.org/issue35774  opened by Dhiraj_Mishra

#35775: Add a general selection function to statistics
https://bugs.python.org/issue35775  opened by remi.lapeyre



Most recent 15 issues with no replies (15)
==========================================

#35775: Add a general selection function to statistics
https://bugs.python.org/issue35775

#35774: ASAN, memory leak
https://bugs.python.org/issue35774

#35773: test_bdb fails on AIX bot (regression)
https://bugs.python.org/issue35773

#35771: IDLE: Fix tooltip Hovertiptest failure
https://bugs.python.org/issue35771

#35768: IDLE: Auto measure font fixed pitch characteristics
https://bugs.python.org/issue35768

#35766: Merge typed_ast back into CPython
https://bugs.python.org/issue35766

#35764: IDLE: revise calltip doc
https://bugs.python.org/issue35764

#35762: subprocess.Popen with universal_newlines and nonblocking strea
https://bugs.python.org/issue35762

#35760: test_asyncio: test_async_gen_asyncio_gc_aclose_09() race condi
https://bugs.python.org/issue35760

#35758: Disable x87 control word for MSVC ARM compiler
https://bugs.python.org/issue35758

#35741: unittest.skipUnless(time._STRUCT_TM_ITEMS == 11, "needs tm_zon
https://bugs.python.org/issue35741

#35737: crypt AuthenticationError introduced with new Linux kernel
https://bugs.python.org/issue35737

#35728: Tkinter font nametofont requires default root
https://bugs.python.org/issue35728

#35724: Check for main interpreter when checking for "main" thread (fo
https://bugs.python.org/issue35724

#35722: disable_existing_loggers does not apply to the root logger
https://bugs.python.org/issue35722



Most recent 15 issues waiting for review (15)
=============================================

#35775: Add a general selection function to statistics
https://bugs.python.org/issue35775

#35772: test_tarfile fails on ppc64le when using tmpfs filesystem
https://bugs.python.org/issue35772

#35769: IDLE: change new file name from  ''Untitled" to "untitled"
https://bugs.python.org/issue35769

#35767: unittest loader doesn't work with partial test functions
https://bugs.python.org/issue35767

#35759: inspect module does not implement introspection API for asynch
https://bugs.python.org/issue35759

#35755: Remove current directory from posixpath.defpath to enhance sec
https://bugs.python.org/issue35755

#35752: test_buffer fails on ppc64le: memoryview pack_single() is misc
https://bugs.python.org/issue35752

#35749: Rewrite asyncio signal handler
https://bugs.python.org/issue35749

#35746: [ssl][CVE-2019-5010] TALOS-2018-0758 Denial of Service
https://bugs.python.org/issue35746

#35739: Enable verbose of tests during PGO build on amd64 platforms
https://bugs.python.org/issue35739

#35733: isinstance(ast.Constant(value=True), ast.Num) should be False
https://bugs.python.org/issue35733

#35727: sys.exit() in a multiprocessing.Process does not align with Py
https://bugs.python.org/issue35727

#35726: QueueHandler formating affects other handlers
https://bugs.python.org/issue35726

#35724: Check for main interpreter when checking for "main" thread (fo
https://bugs.python.org/issue35724

#35723: Add "time zone index" cache to datetime objects
https://bugs.python.org/issue35723



Top 10 most discussed issues (10)
=================================

#35537: use os.posix_spawn in subprocess
https://bugs.python.org/issue35537  24 msgs

#33944: Deprecate and remove pth files
https://bugs.python.org/issue33944  21 msgs

#35752: test_buffer fails on ppc64le: memoryview pack_single() is misc
https://bugs.python.org/issue35752  18 msgs

#35701: [uuid] 3.8 breaks weak references for UUIDs
https://bugs.python.org/issue35701  15 msgs

#35746: [ssl][CVE-2019-5010] TALOS-2018-0758 Denial of Service
https://bugs.python.org/issue35746  13 msgs

#35755: Remove current directory from posixpath.defpath to enhance sec
https://bugs.python.org/issue35755  10 msgs

#35707: time.sleep() should support objects with __float__
https://bugs.python.org/issue35707   8 msgs

#35770: IDLE: python -m idlelib fails on master on Mac OS 10.10.4
https://bugs.python.org/issue35770   8 msgs

#34782: Pdb crashes when code is executed in a mapping that does not d
https://bugs.python.org/issue34782   7 msgs

#23428: Use the monotonic clock for thread conditions on POSIX platfor
https://bugs.python.org/issue23428   5 msgs



Issues closed (57)
==================

#8765: Tests unwillingly writing unicocde to raw streams
https://bugs.python.org/issue8765  closed by serhiy.storchaka

#22616: Allow connecting AST nodes with corresponding source ranges
https://bugs.python.org/issue22616  closed by levkivskyi

#23156: Remove tix install information in tkinter tix chapter of doc
https://bugs.python.org/issue23156  closed by terry.reedy

#23846: asyncio : ProactorEventLoop raised BlockingIOError when Thread
https://bugs.python.org/issue23846  closed by vstinner

#26226: Test failures with non-ascii character in hostname on Windows
https://bugs.python.org/issue26226  closed by vstinner

#26410: "incompatible pointer type" while compiling Python3.5.1
https://bugs.python.org/issue26410  closed by cheryl.sabella

#26414: os.defpath too permissive
https://bugs.python.org/issue26414  closed by eryksun

#27423: Failed assertions when running test.test_os on Windows
https://bugs.python.org/issue27423  closed by vstinner

#27426: Encoding mismatch causes some tests to fail on Windows
https://bugs.python.org/issue27426  closed by vstinner

#27500: ProactorEventLoop cannot open connection to ::1
https://bugs.python.org/issue27500  closed by vstinner

#31048: ResourceWarning in test_asyncio.test_events.ProactorEventLoopT
https://bugs.python.org/issue31048  closed by vstinner

#31267: threading.Timer object is affected by changes to system time: 
https://bugs.python.org/issue31267  closed by vstinner

#31777: IDLE: Let users add to font selection
https://bugs.python.org/issue31777  closed by terry.reedy

#32661: ProactorEventLoop locks up on close call
https://bugs.python.org/issue32661  closed by vstinner

#32866: zipimport loader.get_data() requires absolute zip file path
https://bugs.python.org/issue32866  closed by barry

#33301: Add __contains__ to pathlib
https://bugs.python.org/issue33301  closed by serhiy.storchaka

#33687: uu.py calls os.path.chmod which doesn't exist
https://bugs.python.org/issue33687  closed by berker.peksag

#33817: PyString_FromFormatV() fails to build empty strings
https://bugs.python.org/issue33817  closed by serhiy.storchaka

#33837: Closing asyncio.Server on asyncio.ProactorEventLoop causes all
https://bugs.python.org/issue33837  closed by vstinner

#33995: test_min_max_version in test_ssl.py fails when Python is built
https://bugs.python.org/issue33995  closed by vstinner

#34323: False timeout log message on proactor close
https://bugs.python.org/issue34323  closed by vstinner

#34512: Document platform-specific strftime() behavior for non-ASCII f
https://bugs.python.org/issue34512  closed by taleinat

#34569: test__xxsubinterpreters.ShareableTypeTests._assert_values fail
https://bugs.python.org/issue34569  closed by eric.snow

#34756: Few changes in sys.breakpointhook()
https://bugs.python.org/issue34756  closed by serhiy.storchaka

#34838: Improve arg clinic code generation for cases with type checkin
https://bugs.python.org/issue34838  closed by serhiy.storchaka

#34850: Emit a syntax warning for "is" with a literal
https://bugs.python.org/issue34850  closed by serhiy.storchaka

#35045: test_min_max_version (test.test_ssl.ContextTests) fails on Fed
https://bugs.python.org/issue35045  closed by vstinner

#35283: "threading._DummyThread" redefines "is_alive" but forgets "isA
https://bugs.python.org/issue35283  closed by asvetlov

#35423: Signal handling machinery still relies on "pending calls".
https://bugs.python.org/issue35423  closed by eric.snow

#35428: xml.etree.ElementTree.tostring violates W3 standards allowing 
https://bugs.python.org/issue35428  closed by serhiy.storchaka

#35494: Inaccurate error message for f-string
https://bugs.python.org/issue35494  closed by serhiy.storchaka

#35552: Do not read memory past the specified limit in PyUnicode_FromF
https://bugs.python.org/issue35552  closed by serhiy.storchaka

#35582: Argument Clinic: inline parsing code for functions with only p
https://bugs.python.org/issue35582  closed by serhiy.storchaka

#35599: asyncio windows_events.py IocpProactor bug
https://bugs.python.org/issue35599  closed by vstinner

#35619: Support custom data descriptors in pydoc
https://bugs.python.org/issue35619  closed by serhiy.storchaka

#35634: kwargs regression when there are multiple entries with the sam
https://bugs.python.org/issue35634  closed by serhiy.storchaka

#35662: Windows #define _PY_EMULATED_WIN_CV 0 bug
https://bugs.python.org/issue35662  closed by steve.dower

#35674: Expose os.posix_spawnp()
https://bugs.python.org/issue35674  closed by vstinner

#35698: [statistics] Division by 2 in statistics.median
https://bugs.python.org/issue35698  closed by steven.daprano

#35711: Print information about an unexpectedly pending error before c
https://bugs.python.org/issue35711  closed by vstinner

#35713: Fatal Python error: _PySys_BeginInit: can't initialize sys mod
https://bugs.python.org/issue35713  closed by Tasy

#35719: Optimize multi-argument math functions
https://bugs.python.org/issue35719  closed by serhiy.storchaka

#35725: Using for...in.. generator-iterator
https://bugs.python.org/issue35725  closed by steven.daprano

#35729: iterparse does not return the full subtree on "start" events
https://bugs.python.org/issue35729  closed by ned.deily

#35730: IDLE: Fix squeezer test_reload.
https://bugs.python.org/issue35730  closed by terry.reedy

#35732: Typo in library/warnings documentation
https://bugs.python.org/issue35732  closed by cheryl.sabella

#35734: Remove unused _BaseV4._is_valid_netmask in ipaddress
https://bugs.python.org/issue35734  closed by lin.lin

#35735: Current "make test" status for AIX
https://bugs.python.org/issue35735  closed by Michael.Felt

#35738: Update timeit documentation to reflect default repeat of five
https://bugs.python.org/issue35738  closed by vstinner

#35742: test_builtin fails after merging the fix for bpo-34756
https://bugs.python.org/issue35742  closed by serhiy.storchaka

#35743: Broken "Exception ignored in:" message on OSError's
https://bugs.python.org/issue35743  closed by salty-horse

#35744: Problem in the documentation of numpy.random.randint in python
https://bugs.python.org/issue35744  closed by vstinner

#35747: Python threading event wait influenced by date change
https://bugs.python.org/issue35747  closed by vstinner

#35750: process finished with exit code -1073740940 (0xc0000374)
https://bugs.python.org/issue35750  closed by eryksun

#35753: Importing call from unittest.mock directly causes ValueError
https://bugs.python.org/issue35753  closed by pablogsal

#35757: slow subprocess.Popen(..., close_fds=True)
https://bugs.python.org/issue35757  closed by benjamin.peterson

#35776: Virtualenv 16.2.0 Error Finding Pip
https://bugs.python.org/issue35776  closed by zach.ware

From greg.ewing at canterbury.ac.nz  Fri Jan 18 17:57:23 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 19 Jan 2019 11:57:23 +1300
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <CAPTjJmo5GM=HDxwyZDg7nK5wc82ROR425Wqgedow1eipwR5DQA@mail.gmail.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <CAPTjJmo5GM=HDxwyZDg7nK5wc82ROR425Wqgedow1eipwR5DQA@mail.gmail.com>
Message-ID: <5C4259D3.2030509@canterbury.ac.nz>

Chris Angelico wrote:
> I would be strongly in favour of ctypes gaining a "get address of
> object" function, which happens (in current CPythons) to return the
> same value as id() does, but is specifically tied to ctypes.

Isn't this what the ctypes.py_object type is for?

Also, any code that does anything with the address of an object
other than just pass it around is going to depend heavily on
the Python implementation being used, so the idea of an
implementation-independent way to deal with object addresses
seems problematic.

-- 
Greg

From rosuav at gmail.com  Fri Jan 18 18:02:21 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 19 Jan 2019 10:02:21 +1100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <5C4259D3.2030509@canterbury.ac.nz>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <CAPTjJmo5GM=HDxwyZDg7nK5wc82ROR425Wqgedow1eipwR5DQA@mail.gmail.com>
 <5C4259D3.2030509@canterbury.ac.nz>
Message-ID: <CAPTjJmo=Nz1c0Ejx0YtjeohkvO0q1W7udxZOHCD0Aad0pqnOvA@mail.gmail.com>

On Sat, Jan 19, 2019 at 9:58 AM Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Chris Angelico wrote:
> > I would be strongly in favour of ctypes gaining a "get address of
> > object" function, which happens (in current CPythons) to return the
> > same value as id() does, but is specifically tied to ctypes.
>
> Isn't this what the ctypes.py_object type is for?

I didn't know about it when I posted that (as, I suspect, others also
didn't), and as others have pointed out, this is a prime target for a
docs update. Scanning the docs as of today does not suggest a better
way to do things.

ChrisA

From greg.ewing at canterbury.ac.nz  Fri Jan 18 18:02:53 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 19 Jan 2019 12:02:53 +1300
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>
Message-ID: <5C425B1D.10703@canterbury.ac.nz>

MRAB wrote:
  If I want to cache some objects, I put them in a dict, using the id as
> the key. If I wanted to locate an object in a cache and didn't have 
> id(), I'd have to do a linear search for it.

That sounds dangerous. An id() is only valid as long as the object
it came from still exists, after which it can get re-used for a different
object. So when an object is flushed from your cache, you would have
to chase down all the places its id is being stored and eliminate them.

Are you sure you couldn't achieve the same thing more safely using
weak references?

-- 
Greg

From tim.peters at gmail.com  Fri Jan 18 18:25:48 2019
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 18 Jan 2019 17:25:48 -0600
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <5C425B1D.10703@canterbury.ac.nz>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>
 <5C425B1D.10703@canterbury.ac.nz>
Message-ID: <CAExdVNm8TUqQjVmAHNTyjB-fUmZK9QY3poArRSspGPeCJztBQw@mail.gmail.com>

[MRAB]
>>  If I want to cache some objects, I put them in a dict, using the id as
>> the key. If I wanted to locate an object in a cache and didn't have
>> id(), I'd have to do a linear search for it.

[Greg Ewing <greg.ewing at canterbury.ac.nz>]
> That sounds dangerous. An id() is only valid as long as the object
> it came from still exists, after which it can get re-used for a different
> object.

The objects are the values in such a dict.

    thedict[id(obj)] is obj

Therefore the objects can't become garbage before id(obj) is deleted
from the dict.

> So when an object is flushed from your cache, you would have
> to chase down all the places its id is being stored and eliminate them.

The dict itself keeps the objects alive.

> Are you sure you couldn't achieve the same thing more safely using
> weak references?

I can't say exactly what MRAB is doing.  I've done things "like that"
for decades, though, and have happily almost never used weakrefs.  I
wouldn't call my uses "caches", though - more like using dicts to
associate info with arbitrary objects (via using the object id as the
dict key), where the object implementations are out of my control and
don't properly support being used as dict keys.

This sometimes includes builtin mutable objects, like lists, or even
other dicts.

No such uses care about object addresses, though - just that id(obj)
returns a value usable as a dict key, unique among all reachable
objects at the time `id()` is called.

From python at mrabarnett.plus.com  Fri Jan 18 18:39:54 2019
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 18 Jan 2019 23:39:54 +0000
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <5C425B1D.10703@canterbury.ac.nz>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>
 <5C425B1D.10703@canterbury.ac.nz>
Message-ID: <b3d5c94c-7ca9-b82e-8df2-5136883a8e22@mrabarnett.plus.com>

On 2019-01-18 23:02, Greg Ewing wrote:
> MRAB wrote:
>    If I want to cache some objects, I put them in a dict, using the id as
>> the key. If I wanted to locate an object in a cache and didn't have 
>> id(), I'd have to do a linear search for it.
> 
> That sounds dangerous. An id() is only valid as long as the object
> it came from still exists, after which it can get re-used for a different
> object. So when an object is flushed from your cache, you would have
> to chase down all the places its id is being stored and eliminate them.
> 
> Are you sure you couldn't achieve the same thing more safely using
> weak references?
> 
I'm not storing the id anywhere else.

I could've used a list for the cache, but then when I wanted to remove 
an object I'd have to search for it, O(n). Using a dict makes it O(1).

From greg.ewing at canterbury.ac.nz  Fri Jan 18 19:28:06 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 19 Jan 2019 13:28:06 +1300
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <CAExdVNm8TUqQjVmAHNTyjB-fUmZK9QY3poArRSspGPeCJztBQw@mail.gmail.com>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>
 <5C425B1D.10703@canterbury.ac.nz>
 <CAExdVNm8TUqQjVmAHNTyjB-fUmZK9QY3poArRSspGPeCJztBQw@mail.gmail.com>
Message-ID: <5C426F16.6050409@canterbury.ac.nz>

Tim Peters wrote:

> The dict itself keeps the objects alive.

Yes, but the idea of a cache is that you're free to flush
things out of it to make room for something else without
breaking anything.

It sounds like MRAB is using ids as weak references,
without the assurance actual weak references give you
that they become invalidated when the refefenced object
goes away,

> No such uses care about object addresses, though - just that id(obj)
> returns a value usable as a dict key, unique among all reachable
> objects at the time `id()` is called.

Yep. In hindsight it was probably a mistake for the docs
to talk about addresses in relation to id() -- it seems to
have given some people unrealistic expectations.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Jan 18 19:35:14 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 19 Jan 2019 13:35:14 +1300
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <20190118101756.GM13616@ando.pearwood.info>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <20190118101756.GM13616@ando.pearwood.info>
Message-ID: <5C4270C2.4080106@canterbury.ac.nz>

Steven D'Aprano wrote:

> The sample code I've been shown is this:
> 
>     pointer_to_obj = id(obj)
>     from_deref = ctypes.cast(pointer_to_obj, ctypes.py_object).value
>     from_deref is obj  # True

There's no need to use id() or casting to create a ctypes.py_object
instance, you can just call it:

 >>> obj = (1,2,3)
 >>> obj
(1, 2, 3)
 >>> p = ctypes.py_object(obj)
 >>> p
py_object((1, 2, 3))
 >>> p.value
(1, 2, 3)
 >>> p.value is obj
True

-- 
Greg

From python at mrabarnett.plus.com  Fri Jan 18 20:59:20 2019
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 19 Jan 2019 01:59:20 +0000
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <5C426F16.6050409@canterbury.ac.nz>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>
 <5C425B1D.10703@canterbury.ac.nz>
 <CAExdVNm8TUqQjVmAHNTyjB-fUmZK9QY3poArRSspGPeCJztBQw@mail.gmail.com>
 <5C426F16.6050409@canterbury.ac.nz>
Message-ID: <f3801a83-83ca-bbe5-0501-52285c926bbe@mrabarnett.plus.com>

On 2019-01-19 00:28, Greg Ewing wrote:
> Tim Peters wrote:
> 
>> The dict itself keeps the objects alive.
> 
> Yes, but the idea of a cache is that you're free to flush
> things out of it to make room for something else without
> breaking anything.
> 
> It sounds like MRAB is using ids as weak references,
> without the assurance actual weak references give you
> that they become invalidated when the refefenced object
> goes away,
> 
"Cache" was the wrong word for what it does. I'm not using the id as a 
weak reference.

Sometimes I might want to store a collection of objects and their order 
isn't important. I can add an object to the collection, or remove an 
object from it.

If I used a list, adding would be quick, but removing would require 
searching the list.

By putting them in a dict, keyed by the id, I can remove an object in O(1).

Trust me, I'm not doing anything that's unreliable! (And I _have_ done 
programming in C with the Python API, so I know all about refcounts...) :-)

>> No such uses care about object addresses, though - just that id(obj)
>> returns a value usable as a dict key, unique among all reachable
>> objects at the time `id()` is called.
> 
> Yep. In hindsight it was probably a mistake for the docs
> to talk about addresses in relation to id() -- it seems to
> have given some people unrealistic expectations.
> 

From storchaka at gmail.com  Sat Jan 19 05:12:44 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 19 Jan 2019 12:12:44 +0200
Subject: [Python-Dev] Lost sight
Message-ID: <q1ut2o$pds$1@blaine.gmane.org>

I have virtually completely lost the sight of my right eye (and the loss 
is quickly progresses) and the sight of my left eye is weak. That is why 
my activity as a core developer was decreased significantly at recent 
time. My apologies to those who are waiting for my review. I will do it 
slowly.


From eryksun at gmail.com  Sat Jan 19 06:06:36 2019
From: eryksun at gmail.com (eryk sun)
Date: Sat, 19 Jan 2019 05:06:36 -0600
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
In-Reply-To: <20190118105108.GN13616@ando.pearwood.info>
References: <20190117102605.GG13616@ando.pearwood.info>
 <CACL+1av8m34gnS+7f9xd+8Wnp47yMydiqaxxOvOLbHhA3SPKdg@mail.gmail.com>
 <20190118105108.GN13616@ando.pearwood.info>
Message-ID: <CACL+1as5VWvHe7iM625ZOafvuk-HbCbuypV9+cxZBAaP6yeHwg@mail.gmail.com>

On 1/18/19, Steven D'Aprano <steve at pearwood.info> wrote:
> On Thu, Jan 17, 2019 at 07:50:51AM -0600, eryk sun wrote:
>>
>> It's kind of dangerous to pass an object to C without an increment of
>> its reference count.
>
> "Kind of dangerous?" How dangerous?

I take that back. Dangerous is too strong of a word. It can be managed
if we're careful to avoid expressions like c_function(id(f())). Using
py_object simply avoids that problem.

Bear with me while I make a few more comments about py_object, even
though it's straying off topic.

For a type "O" argument (i.e. py_object is in the function's
`argtypes`), we might be able to borrow the reference from the
argument tuple. As implemented, however, the argument actually keeps
its own reference. For example, we can observe this by calling the
from_param method:

    >>> b = bytearray(b'spam')
    >>> arg = ctypes.py_object.from_param(b)
    >>> print(arg)
    <cparam 'O' at 0x7f32a49699b0>
    >>> print(arg._obj)
    bytearray(b'spam')

This is due to the type "O" setfunc, which needs to keep a reference
to the object when setting the value of a py_object instance. The
reference is stored as the _objects attribute. (For non-simple pointer
and aggregate types, _objects is instead a dict keyed by the index as
a hexadecimal string.)

(The getfunc and setfunc of a simple ctypes object are called to get
and set the value, which also includes cases in which we don't have an
actual py_object instance, such as function call arguments; pointer
and array indexes; and struct and union fields. These functions are
defined in Modules/_ctypes/cfield.c.)

IMO, a downside of py_object is that it's a simple type, so the
getfunc gets called automatically when getting fields or indexes. This
is annoying for py_object since a NULL value raises ValueError.
Returning None in this case isn't possible, in contrast to other
simple pointer types. We can work around this by subclassing
py_object. For example:

    >>> a1 = (ctypes.py_object * 1)()
    >>> a1[0]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: PyObject is NULL

    py_object = type('py_object', (ctypes.py_object,), {})

    >>> a2 = (py_object * 1)()
    >>> a2[0]
    <py_object object at 0x7f10dc7d9158>

Then, like all ctypes pointers, a false boolean value means it's NULL:

    >>> bool(a2[0])
    False
    >>> a2[0] = b'spam'
    >>> bool(a2[0])
    True

py_object doesn't help if a library holds onto the pointer and tries
to use it later on. For example, with Python's C API there are
functions that 'steal' a reference (with the assumption that it's a
newly created object, in which case it's more like 'claiming'), such
as PyTuple_SetItem. In this case, we need to increment the reference
count via Py_IncRef.

py_object can be returned from a callback without leaking a reference,
assuming the library manages the new reference. In contrast, other
types that need memory support have to leak a reference (e.g.
c_wchar_p, i.e. type "Z", needs a capsule object for the wchar_t
buffer). In case of a leak, we get warned with RuntimeWarning('memory
leak in callback function.').

> If I am reading this correctly, I think you are saying that using id()
> in this way is never(?) correct.

Yes, it's incorrect, but I've been guilty of using id() like this,
too, because it's convenient. Perhaps we could provide a function
that's explicitly specified to return the address, if implemented.
Maybe call it sys.getaddress()?

In my first reply, I provided two alternatives that use ctypes to
return the address instead of id(). So there's that as well. The fine
print is that ctypes is optional in the standard library. Platforms
and implementations don't have to support it.

From christian at python.org  Sat Jan 19 06:22:29 2019
From: christian at python.org (Christian Heimes)
Date: Sat, 19 Jan 2019 12:22:29 +0100
Subject: [Python-Dev] Lost sight
In-Reply-To: <q1ut2o$pds$1@blaine.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <ea127340-91d8-6fdd-125f-9e10f3d5d349@python.org>

On 19/01/2019 11.12, Serhiy Storchaka wrote:
> I have virtually completely lost the sight of my right eye (and the loss
> is quickly progresses) and the sight of my left eye is weak. That is why
> my activity as a core developer was decreased significantly at recent
> time. My apologies to those who are waiting for my review. I will do it
> slowly.

Oh, I'm sorry to hear that and hope that you'll get better soon. Please
take care of yourself!

Christian

From solipsis at pitrou.net  Sat Jan 19 06:30:28 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 19 Jan 2019 12:30:28 +0100
Subject: [Python-Dev] ctypes: is it intentional that id() is the only
 way to get the address of an object?
References: <20190117102605.GG13616@ando.pearwood.info>
 <CAGE7PNL3oL4D=jvr=rK+gYgFkr89hhwrC6Tm6a8vOjD0o1+Phw@mail.gmail.com>
 <6f9aaf8a-5b4b-954b-ffd2-10ae9a6c55c9@mrabarnett.plus.com>
 <5C425B1D.10703@canterbury.ac.nz>
 <CAExdVNm8TUqQjVmAHNTyjB-fUmZK9QY3poArRSspGPeCJztBQw@mail.gmail.com>
 <5C426F16.6050409@canterbury.ac.nz>
Message-ID: <20190119123028.27627a9c@fsol>

On Sat, 19 Jan 2019 13:28:06 +1300
Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Tim Peters wrote:
> 
> > The dict itself keeps the objects alive.  
> 
> Yes, but the idea of a cache is that you're free to flush
> things out of it to make room for something else without
> breaking anything.
> 
> It sounds like MRAB is using ids as weak references,
> without the assurance actual weak references give you
> that they become invalidated when the refefenced object
> goes away,

Hmm...  That sounds nonsensical to me. By construction, if you're able
to get a reference to an object in pure Python, then the object is
alive.

(by pure Python I'm excluding ctypes hacks or the exploitation of bugs
in the CPython object implementation)

By the way, you can also have a WeakValueDictionary where keys are ids
and values are the corresponding objects, if you need both identity
lookup and weak references.

Regards

Antoine.



From andrew.svetlov at gmail.com  Sat Jan 19 06:31:42 2019
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sat, 19 Jan 2019 13:31:42 +0200
Subject: [Python-Dev] Lost sight
In-Reply-To: <ea127340-91d8-6fdd-125f-9e10f3d5d349@python.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
 <ea127340-91d8-6fdd-125f-9e10f3d5d349@python.org>
Message-ID: <CAL3CFcUUvOQUeryTFvR+Uqj2wnB0Yuw84QjE_MpOfSVC5NA05Q@mail.gmail.com>

That's sad to hear.
Get well soon!

On Sat, Jan 19, 2019 at 1:22 PM Christian Heimes <christian at python.org>
wrote:

> On 19/01/2019 11.12, Serhiy Storchaka wrote:
> > I have virtually completely lost the sight of my right eye (and the loss
> > is quickly progresses) and the sight of my left eye is weak. That is why
> > my activity as a core developer was decreased significantly at recent
> > time. My apologies to those who are waiting for my review. I will do it
> > slowly.
>
> Oh, I'm sorry to hear that and hope that you'll get better soon. Please
> take care of yourself!
>
> Christian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>


-- 
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190119/9235b0ce/attachment.html>

From p.f.moore at gmail.com  Sat Jan 19 06:35:59 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 19 Jan 2019 11:35:59 +0000
Subject: [Python-Dev] Lost sight
In-Reply-To: <q1ut2o$pds$1@blaine.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <CACac1F-u-rSD8t2t0Stg45G0vsxmedYX+z-FJOpYsSyB-A6X0g@mail.gmail.com>

On Sat, 19 Jan 2019 at 10:15, Serhiy Storchaka <storchaka at gmail.com> wrote:
>
> I have virtually completely lost the sight of my right eye (and the loss
> is quickly progresses) and the sight of my left eye is weak. That is why
> my activity as a core developer was decreased significantly at recent
> time. My apologies to those who are waiting for my review. I will do it
> slowly.

Sorry to hear about that! Take care of yourself, your health is the
most important thing.

Best wishes,
Paul

From yselivanov.ml at gmail.com  Sat Jan 19 14:13:48 2019
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Sat, 19 Jan 2019 14:13:48 -0500
Subject: [Python-Dev] Lost sight
In-Reply-To: <q1ut2o$pds$1@blaine.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <CA+St6D2XrCtjBmbmJrb33yOz+8eFmJ=XbEPnvEYKH36dckCS=w@mail.gmail.com>

Sorry to hear this, Serhiy.  Hope you'll get better soon.

Yury

On Sat, Jan 19, 2019 at 5:15 AM Serhiy Storchaka <storchaka at gmail.com> wrote:
>
> I have virtually completely lost the sight of my right eye (and the loss
> is quickly progresses) and the sight of my left eye is weak. That is why
> my activity as a core developer was decreased significantly at recent
> time. My apologies to those who are waiting for my review. I will do it
> slowly.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com



-- 
         Yury

From mariatta at python.org  Sat Jan 19 17:25:03 2019
From: mariatta at python.org (Mariatta Wijaya)
Date: Sat, 19 Jan 2019 16:25:03 -0600
Subject: [Python-Dev] Lost sight
In-Reply-To: <q1ut2o$pds$1@blaine.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <CAGbohnbbqgzdz6w+hk38GQqbxJyH9iXfsi1OEr1oRanVmeJ=Vg@mail.gmail.com>

Sorry to hear that. Please take care.

On Sat, Jan 19, 2019, 4:15 AM Serhiy Storchaka <storchaka at gmail.com wrote:

> I have virtually completely lost the sight of my right eye (and the loss
> is quickly progresses) and the sight of my left eye is weak. That is why
> my activity as a core developer was decreased significantly at recent
> time. My apologies to those who are waiting for my review. I will do it
> slowly.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mariatta%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190119/b67d65c3/attachment.html>

From chris.jerdonek at gmail.com  Sat Jan 19 18:16:16 2019
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sat, 19 Jan 2019 15:16:16 -0800
Subject: [Python-Dev] Lost sight
In-Reply-To: <q1ut2o$pds$1@blaine.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <CAOTb1wcRq0A_T+oQECG-NLNwsb4KwmstqqtGaSDO+6P4FV5SqA@mail.gmail.com>

Hi Serhiy,

That's terrible and sounds frightening. Were you able to get medical
care to get a diagnosis and treatment if needed?

We all hope your condition improves.

--Chris

On Sat, Jan 19, 2019 at 2:14 AM Serhiy Storchaka <storchaka at gmail.com> wrote:
>
> I have virtually completely lost the sight of my right eye (and the loss
> is quickly progresses) and the sight of my left eye is weak. That is why
> my activity as a core developer was decreased significantly at recent
> time. My apologies to those who are waiting for my review. I will do it
> slowly.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com

From stephane at wirtel.be  Sat Jan 19 23:54:21 2019
From: stephane at wirtel.be (Stephane Wirtel)
Date: Sun, 20 Jan 2019 05:54:21 +0100
Subject: [Python-Dev] Lost sight
In-Reply-To: <CAGbohnbbqgzdz6w+hk38GQqbxJyH9iXfsi1OEr1oRanVmeJ=Vg@mail.gmail.com>
References: <q1ut2o$pds$1@blaine.gmane.org>
 <CAGbohnbbqgzdz6w+hk38GQqbxJyH9iXfsi1OEr1oRanVmeJ=Vg@mail.gmail.com>
Message-ID: <7591B4B8-64B4-4AF8-B6CF-9020CBEE1918@wirtel.be>

Really sorry Serhiy, hope the best for you, take care. Firstly the most important thing, is your health. 

St?phane 

> Le 19 janv. 2019 ? 23:25, Mariatta Wijaya <mariatta at python.org> a ?crit :
> 
> Sorry to hear that. Please take care.
> 
>> On Sat, Jan 19, 2019, 4:15 AM Serhiy Storchaka <storchaka at gmail.com wrote:
>> I have virtually completely lost the sight of my right eye (and the loss 
>> is quickly progresses) and the sight of my left eye is weak. That is why 
>> my activity as a core developer was decreased significantly at recent 
>> time. My apologies to those who are waiting for my review. I will do it 
>> slowly.
>> 
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mariatta%40python.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/stephane%40wirtel.be
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190120/5e63cd65/attachment.html>

From songofacandy at gmail.com  Sun Jan 20 22:25:51 2019
From: songofacandy at gmail.com (INADA Naoki)
Date: Mon, 21 Jan 2019 12:25:51 +0900
Subject: [Python-Dev] Lost sight
In-Reply-To: <q1ut2o$pds$1@blaine.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <CAEfz+TzMwNJ6mK_GfqppyZg4wu2CsMRQgPTqc83y4Txfex6Zcg@mail.gmail.com>

Hi, Serhiy.

I'm so sorry about it.
I hope you will get better.
Please take care of yourself.

From raymond.hettinger at gmail.com  Mon Jan 21 02:04:12 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 20 Jan 2019 23:04:12 -0800
Subject: [Python-Dev] Lost sight
In-Reply-To: <q1ut2o$pds$1@blaine.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <EE83E3FA-DAC6-4E5F-978D-21D8708F33EE@gmail.com>


> On Jan 19, 2019, at 2:12 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 
> I have virtually completely lost the sight of my right eye (and the loss is quickly progresses) and the sight of my left eye is weak. 

I hope this only temporary.  Best wishes.


Raymond

From solipsis at pitrou.net  Mon Jan 21 03:39:40 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 21 Jan 2019 09:39:40 +0100
Subject: [Python-Dev] Lost sight
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <20190121093940.65d3e258@fsol>


Hi Serhiy,

On Sat, 19 Jan 2019 12:12:44 +0200
Serhiy Storchaka <storchaka at gmail.com> wrote:
> I have virtually completely lost the sight of my right eye (and the loss 
> is quickly progresses) and the sight of my left eye is weak. That is why 
> my activity as a core developer was decreased significantly at recent 
> time. My apologies to those who are waiting for my review. I will do it 
> slowly.

I really hope you're going to get better.  If there is something the
community can do for you, please say so ;-)

Regards

Antoine.



From storchaka at gmail.com  Mon Jan 21 10:26:17 2019
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 21 Jan 2019 17:26:17 +0200
Subject: [Python-Dev] Lost sight
In-Reply-To: <q1ut2o$pds$1@blaine.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <q24upf$h0v$1@ciao.gmane.org>

Thank you very match, all who have expressed compassion here and 
privately. I am very touched. It at least helped me feel a little better 
psychologically.


From ncoghlan at gmail.com  Tue Jan 22 06:09:55 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 22 Jan 2019 21:09:55 +1000
Subject: [Python-Dev] Lost sight
In-Reply-To: <20190121093940.65d3e258@fsol>
References: <q1ut2o$pds$1@blaine.gmane.org> <20190121093940.65d3e258@fsol>
Message-ID: <CADiSq7dSaqJXXGOxPgiM5OYg6TbySHU-Bn66uyYCwPBLwhagtQ@mail.gmail.com>

On Mon, 21 Jan 2019 at 18:43, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 19 Jan 2019 12:12:44 +0200
> Serhiy Storchaka <storchaka at gmail.com> wrote:
> > I have virtually completely lost the sight of my right eye (and the loss
> > is quickly progresses) and the sight of my left eye is weak. That is why
> > my activity as a core developer was decreased significantly at recent
> > time. My apologies to those who are waiting for my review. I will do it
> > slowly.
>
> I really hope you're going to get better.  If there is something the
> community can do for you, please say so ;-)

Hear, hear - if we can help in any way, please let us know.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stephan.reiter at gmail.com  Tue Jan 22 09:32:22 2019
From: stephan.reiter at gmail.com (Stephan Reiter)
Date: Tue, 22 Jan 2019 15:32:22 +0100
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
Message-ID: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>

Hi all!

I am new to the list and arriving with a concrete problem that I'd
like to fix myself.

I am embedding Python (3.6) into my C++ application and I would like
to run Python scripts isolated from each other using sub-interpreters.
I am not using threads; everything is supposed to run in the
application's main thread.

I noticed that if I create an interpreter, switch to it and execute
code that imports numpy (1.13), my application will hang.

  ntdll.dll!NtWaitForSingleObject() Unknown
  KernelBase.dll!WaitForSingleObjectEx() Unknown
> python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0, _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245 C
  [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
  python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C
  python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line 370 C
  python36.dll!PyGILState_Ensure() Line 855 C
  umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown
  umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown
  umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown
  umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown
  umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown
  [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
  [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C
  python36.dll!call_function(_object * * *
pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line
4822 C

Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
PyEval_RestoreThread on the (auto) threadstate of the main
interpreter. And that's wrong.
We are already holding the GIL with the threadstate of our current
sub-interpreter, so there's no need to switch.

I know that the GIL API is not fully compatible with sub-interpreters,
as issues #10915 and #15751 illustrate.

But since I need to support calls to PyGILState_Ensure - numpy is the
best example -, I am trying to improve the situation here:
https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597

That change may be naive, but it does the trick for my use case. If
totally wrong, I don't mind pursuing another alley.

Essentially, I'd like to ask for some guidance in how to tackle this
problem while keeping the current GIL API unchanged (to avoid breaking
modules).

I am also wondering how I can test any changes I am proposing. Is
there a test suite for interpreters, for example?

Thank you very much,
Stephan

From solipsis at pitrou.net  Tue Jan 22 12:03:47 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 22 Jan 2019 18:03:47 +0100
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
Message-ID: <20190122180347.4f8f2b6e@fsol>

On Tue, 22 Jan 2019 15:32:22 +0100
Stephan Reiter <stephan.reiter at gmail.com> wrote:
> 
> Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
> PyEval_RestoreThread on the (auto) threadstate of the main
> interpreter. And that's wrong.
> We are already holding the GIL with the threadstate of our current
> sub-interpreter, so there's no need to switch.
> 
> I know that the GIL API is not fully compatible with sub-interpreters,
> as issues #10915 and #15751 illustrate.

That's a pity.

Note that there is a patch on https://bugs.python.org/issue10915 that
could probably solve the issue if it had been applied some years ago ;-)

(yes, it needs C extension authors to use the new API, but Numpy is a
well-maintained library and would probably have accepted a patch for
that; so would Cython probably)

> Essentially, I'd like to ask for some guidance in how to tackle this
> problem while keeping the current GIL API unchanged (to avoid breaking
> modules).

I'm not aware of any solution which does not require designing a new
API, unfortunately.

> I am also wondering how I can test any changes I am proposing. Is
> there a test suite for interpreters, for example?

You'll find a couple of them in test_embed.py, test_capi.py and
test_threading.py.

Regards

Antoine.



From njs at pobox.com  Tue Jan 22 16:39:21 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 22 Jan 2019 13:39:21 -0800
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
Message-ID: <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>

There are currently numerous incompatibilities between numpy and
subinterpreters, and no concrete plan for fixing them. The numpy team does
not consider subinterpreters to be a supported configuration, and can't
help you with any issues you run into. I know the concept of
subinterpreters is really appealing, but unfortunately the CPython
implementation is not really mature or widely supported... are you
absolutely certain you need to use subinterpreters for your application?

On Tue, Jan 22, 2019, 08:27 Stephan Reiter <stephan.reiter at gmail.com wrote:

> Hi all!
>
> I am new to the list and arriving with a concrete problem that I'd
> like to fix myself.
>
> I am embedding Python (3.6) into my C++ application and I would like
> to run Python scripts isolated from each other using sub-interpreters.
> I am not using threads; everything is supposed to run in the
> application's main thread.
>
> I noticed that if I create an interpreter, switch to it and execute
> code that imports numpy (1.13), my application will hang.
>
>   ntdll.dll!NtWaitForSingleObject() Unknown
>   KernelBase.dll!WaitForSingleObjectEx() Unknown
> > python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0,
> _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245
> C
>   [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
>   python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C
>   python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line
> 370 C
>   python36.dll!PyGILState_Ensure() Line 855 C
>   umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown
>   umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown
>   umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown
>   umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown
>   umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown
>   [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
>   [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line
> 2480 C
>   python36.dll!call_function(_object * * *
> pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line
> 4822 C
>
> Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
> PyEval_RestoreThread on the (auto) threadstate of the main
> interpreter. And that's wrong.
> We are already holding the GIL with the threadstate of our current
> sub-interpreter, so there's no need to switch.
>
> I know that the GIL API is not fully compatible with sub-interpreters,
> as issues #10915 and #15751 illustrate.
>
> But since I need to support calls to PyGILState_Ensure - numpy is the
> best example -, I am trying to improve the situation here:
>
> https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597
>
> That change may be naive, but it does the trick for my use case. If
> totally wrong, I don't mind pursuing another alley.
>
> Essentially, I'd like to ask for some guidance in how to tackle this
> problem while keeping the current GIL API unchanged (to avoid breaking
> modules).
>
> I am also wondering how I can test any changes I am proposing. Is
> there a test suite for interpreters, for example?
>
> Thank you very much,
> Stephan
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190122/b3460709/attachment.html>

From levkivskyi at gmail.com  Tue Jan 22 17:45:49 2019
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Tue, 22 Jan 2019 22:45:49 +0000
Subject: [Python-Dev] Source of truth for C-API
Message-ID: <CAOMjWkmVTdNyhkotcoBn08z_hS7d-tk393xEvaOrMKPFFreLKw@mail.gmail.com>

Hi,

I recently modified signatures of two functions PyNode_AddChild() and
PyParser_AddToken().
These two functions are not listed in C-API docs on docs.python.org, and
are not included in Python.h. However, their names look like they may be
part of C-API. So there appeared a question, what is the source of truth
for C-API, is there an official list? Or is it just the content of Python.h?

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190122/081a07da/attachment.html>

From vstinner at redhat.com  Tue Jan 22 18:17:48 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 23 Jan 2019 00:17:48 +0100
Subject: [Python-Dev] Source of truth for C-API
In-Reply-To: <CAOMjWkmVTdNyhkotcoBn08z_hS7d-tk393xEvaOrMKPFFreLKw@mail.gmail.com>
References: <CAOMjWkmVTdNyhkotcoBn08z_hS7d-tk393xEvaOrMKPFFreLKw@mail.gmail.com>
Message-ID: <CA+3bQGEWC5m5X_-DD6vJh=YZBZqL+uwzezc=Q2WVSU6Mz9yJcw@mail.gmail.com>

I consider that frameobject.h is also part of the C API, even if it's
not included by Python.h. For example, PyFrame_GetLineNumber() is part
of the C API, it's just that you have to explicitly #include
"frameobject.h".

I'm not aware of any tool to automatically list the content of the C API.

Counter example: PyFrame_New() is not documented. But I still consider
it to be part of the C API.

... Yeah, the definition of the "C API" is unclear to most peole :-)

See https://pythoncapi.readthedocs.io/ : "Design a new better C API for Python".

Victor

Le mar. 22 janv. 2019 ? 23:49, Ivan Levkivskyi <levkivskyi at gmail.com> a ?crit :
>
> Hi,
>
> I recently modified signatures of two functions PyNode_AddChild() and PyParser_AddToken().
> These two functions are not listed in C-API docs on docs.python.org, and are not included in Python.h. However, their names look like they may be part of C-API. So there appeared a question, what is the source of truth for C-API, is there an official list? Or is it just the content of Python.h?
>
> --
> Ivan
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From stephan.reiter at gmail.com  Tue Jan 22 21:33:16 2019
From: stephan.reiter at gmail.com (Stephan Reiter)
Date: Wed, 23 Jan 2019 03:33:16 +0100
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
Message-ID: <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>

Thanks for the answers so far. I appreciate them!

Nathaniel, I'd like to allow Python plugins in my application. A
plugin should be allowed to bring its own modules along (i.e.
plugin-specific subdir is in sys.path when the plugin is active) and
hence some isolation of them will be needed, so that they can use
different versions of a given module. That's my main motivation for
using subinterpreters.
I thought about running plugins out-of-processes - a separate process
for every plugin - and allow them to communicate with my application
via RPC. But that makes it more complex to implement the API my
application will offer and will slow down things due to the need to
copy data.
Maybe you have another idea for me? :)

Henry, Antoine, thanks for your input; I'll check out the tests and
see what I can learn from issue 10915.

Stephan

Am Di., 22. Jan. 2019 um 22:39 Uhr schrieb Nathaniel Smith <njs at pobox.com>:
>
> There are currently numerous incompatibilities between numpy and subinterpreters, and no concrete plan for fixing them. The numpy team does not consider subinterpreters to be a supported configuration, and can't help you with any issues you run into. I know the concept of subinterpreters is really appealing, but unfortunately the CPython implementation is not really mature or widely supported... are you absolutely certain you need to use subinterpreters for your application?
>
> On Tue, Jan 22, 2019, 08:27 Stephan Reiter <stephan.reiter at gmail.com wrote:
>>
>> Hi all!
>>
>> I am new to the list and arriving with a concrete problem that I'd
>> like to fix myself.
>>
>> I am embedding Python (3.6) into my C++ application and I would like
>> to run Python scripts isolated from each other using sub-interpreters.
>> I am not using threads; everything is supposed to run in the
>> application's main thread.
>>
>> I noticed that if I create an interpreter, switch to it and execute
>> code that imports numpy (1.13), my application will hang.
>>
>>   ntdll.dll!NtWaitForSingleObject() Unknown
>>   KernelBase.dll!WaitForSingleObjectEx() Unknown
>> > python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0, _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245 C
>>   [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
>>   python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C
>>   python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line 370 C
>>   python36.dll!PyGILState_Ensure() Line 855 C
>>   umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown
>>   umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown
>>   umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown
>>   umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown
>>   umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown
>>   [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
>>   [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C
>>   python36.dll!call_function(_object * * *
>> pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line
>> 4822 C
>>
>> Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
>> PyEval_RestoreThread on the (auto) threadstate of the main
>> interpreter. And that's wrong.
>> We are already holding the GIL with the threadstate of our current
>> sub-interpreter, so there's no need to switch.
>>
>> I know that the GIL API is not fully compatible with sub-interpreters,
>> as issues #10915 and #15751 illustrate.
>>
>> But since I need to support calls to PyGILState_Ensure - numpy is the
>> best example -, I am trying to improve the situation here:
>> https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597
>>
>> That change may be naive, but it does the trick for my use case. If
>> totally wrong, I don't mind pursuing another alley.
>>
>> Essentially, I'd like to ask for some guidance in how to tackle this
>> problem while keeping the current GIL API unchanged (to avoid breaking
>> modules).
>>
>> I am also wondering how I can test any changes I am proposing. Is
>> there a test suite for interpreters, for example?
>>
>> Thank you very much,
>> Stephan
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com

From njs at pobox.com  Tue Jan 22 22:51:24 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 22 Jan 2019 19:51:24 -0800
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
Message-ID: <CAPJVwBkTRetP6NPiju7MFed8M3az7z3Nnu0A2BS96eNagBjhyg@mail.gmail.com>

On Tue, Jan 22, 2019 at 6:33 PM Stephan Reiter <stephan.reiter at gmail.com> wrote:
>
> Thanks for the answers so far. I appreciate them!
>
> Nathaniel, I'd like to allow Python plugins in my application. A
> plugin should be allowed to bring its own modules along (i.e.
> plugin-specific subdir is in sys.path when the plugin is active) and
> hence some isolation of them will be needed, so that they can use
> different versions of a given module. That's my main motivation for
> using subinterpreters.
> I thought about running plugins out-of-processes - a separate process
> for every plugin - and allow them to communicate with my application
> via RPC. But that makes it more complex to implement the API my
> application will offer and will slow down things due to the need to
> copy data.
> Maybe you have another idea for me? :)

Not really, sorry! I believe that most applications that support
Python plugins (like blender, gimp, libreoffice, etc.), do it by using
a single shared environment for all plugins. This is also how every
application written in Python works, so at the ecosystem level there's
a lot of pressure on module authors to make it possible to assemble
them into a single coherent environment.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From steve.dower at python.org  Wed Jan 23 01:58:46 2019
From: steve.dower at python.org (Steve Dower)
Date: Tue, 22 Jan 2019 22:58:46 -0800
Subject: [Python-Dev] Source of truth for C-API
In-Reply-To: <CA+3bQGEWC5m5X_-DD6vJh=YZBZqL+uwzezc=Q2WVSU6Mz9yJcw@mail.gmail.com>
References: <CAOMjWkmVTdNyhkotcoBn08z_hS7d-tk393xEvaOrMKPFFreLKw@mail.gmail.com>
 <CA+3bQGEWC5m5X_-DD6vJh=YZBZqL+uwzezc=Q2WVSU6Mz9yJcw@mail.gmail.com>
Message-ID: <3ba8a094-dcb3-13f3-cdb4-2162f4c5e9ac@python.org>

On 22Jan.2019 1517, Victor Stinner wrote:
> I'm not aware of any tool to automatically list the content of the C API.

The shell script attached to https://bugs.python.org/issue23903 should
be able to do it with different preprocessor values (we originally
intended to detect inconsistencies in the stable API, but when we found
lots of existing inconsistencies we couldn't agree on how to deal with
them).

Cheers,
Steve

From encukou at gmail.com  Wed Jan 23 05:30:11 2019
From: encukou at gmail.com (Petr Viktorin)
Date: Wed, 23 Jan 2019 11:30:11 +0100
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
Message-ID: <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>

On 1/23/19 3:33 AM, Stephan Reiter wrote:
> Thanks for the answers so far. I appreciate them!
> 
> Nathaniel, I'd like to allow Python plugins in my application. A
> plugin should be allowed to bring its own modules along (i.e.
> plugin-specific subdir is in sys.path when the plugin is active) and
> hence some isolation of them will be needed, so that they can use
> different versions of a given module. That's my main motivation for
> using subinterpreters.
> I thought about running plugins out-of-processes - a separate process
> for every plugin - and allow them to communicate with my application
> via RPC. But that makes it more complex to implement the API my
> application will offer and will slow down things due to the need to
> copy data.
> Maybe you have another idea for me? :)

Try to make the plugins work together. Look into using pip/PyPI for your 
plugins. Try to make it so each package ("plugin") would have only one 
module/package, and dependencies would be other packages that can be 
installed individually and shared. And keep in mind you can set up your 
own package index, or distribute/install individual package files.

If that's not possible, and you want things to work now, go with subprocess.

If you want to help make subinterpreters work better, there are several 
people scratching at the problem from different angles. Most/all would 
welcome help, but don't expect any short-term benefits.
(FWIW, my own effort is currently blocked on PEP 580, and I hope to move 
forward after a Council is elected.)


> Henry, Antoine, thanks for your input; I'll check out the tests and
> see what I can learn from issue 10915.
> 
> Stephan
> 
> Am Di., 22. Jan. 2019 um 22:39 Uhr schrieb Nathaniel Smith <njs at pobox.com>:
>>
>> There are currently numerous incompatibilities between numpy and subinterpreters, and no concrete plan for fixing them. The numpy team does not consider subinterpreters to be a supported configuration, and can't help you with any issues you run into. I know the concept of subinterpreters is really appealing, but unfortunately the CPython implementation is not really mature or widely supported... are you absolutely certain you need to use subinterpreters for your application?
>>
>> On Tue, Jan 22, 2019, 08:27 Stephan Reiter <stephan.reiter at gmail.com wrote:
>>>
>>> Hi all!
>>>
>>> I am new to the list and arriving with a concrete problem that I'd
>>> like to fix myself.
>>>
>>> I am embedding Python (3.6) into my C++ application and I would like
>>> to run Python scripts isolated from each other using sub-interpreters.
>>> I am not using threads; everything is supposed to run in the
>>> application's main thread.
>>>
>>> I noticed that if I create an interpreter, switch to it and execute
>>> code that imports numpy (1.13), my application will hang.
>>>
>>>    ntdll.dll!NtWaitForSingleObject() Unknown
>>>    KernelBase.dll!WaitForSingleObjectEx() Unknown
>>>> python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0, _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245 C
>>>    [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
>>>    python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C
>>>    python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line 370 C
>>>    python36.dll!PyGILState_Ensure() Line 855 C
>>>    umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown
>>>    umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown
>>>    umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown
>>>    umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown
>>>    umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown
>>>    [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
>>>    [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C
>>>    python36.dll!call_function(_object * * *
>>> pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line
>>> 4822 C
>>>
>>> Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
>>> PyEval_RestoreThread on the (auto) threadstate of the main
>>> interpreter. And that's wrong.
>>> We are already holding the GIL with the threadstate of our current
>>> sub-interpreter, so there's no need to switch.
>>>
>>> I know that the GIL API is not fully compatible with sub-interpreters,
>>> as issues #10915 and #15751 illustrate.
>>>
>>> But since I need to support calls to PyGILState_Ensure - numpy is the
>>> best example -, I am trying to improve the situation here:
>>> https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597
>>>
>>> That change may be naive, but it does the trick for my use case. If
>>> totally wrong, I don't mind pursuing another alley.
>>>
>>> Essentially, I'd like to ask for some guidance in how to tackle this
>>> problem while keeping the current GIL API unchanged (to avoid breaking
>>> modules).
>>>
>>> I am also wondering how I can test any changes I am proposing. Is
>>> there a test suite for interpreters, for example?
>>>
>>> Thank you very much,
>>> Stephan
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com
> 

From levkivskyi at gmail.com  Wed Jan 23 10:24:16 2019
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 23 Jan 2019 15:24:16 +0000
Subject: [Python-Dev] Source of truth for C-API
In-Reply-To: <3ba8a094-dcb3-13f3-cdb4-2162f4c5e9ac@python.org>
References: <CAOMjWkmVTdNyhkotcoBn08z_hS7d-tk393xEvaOrMKPFFreLKw@mail.gmail.com>
 <CA+3bQGEWC5m5X_-DD6vJh=YZBZqL+uwzezc=Q2WVSU6Mz9yJcw@mail.gmail.com>
 <3ba8a094-dcb3-13f3-cdb4-2162f4c5e9ac@python.org>
Message-ID: <CAOMjWkniKXCeWqrx5oE5+0WVrqgtPkpRdvoWyKpS+xBpCY1tgQ@mail.gmail.com>

Thanks for advice Victor and Steve!

I looked at the list, and the two functions I mentioned are not in the
list. So I assume the best strategy for now is to wait until first
alpha-beta releases are out, and see if anyone complains.

--
Ivan



On Wed, 23 Jan 2019 at 06:58, Steve Dower <steve.dower at python.org> wrote:

> On 22Jan.2019 1517, Victor Stinner wrote:
> > I'm not aware of any tool to automatically list the content of the C API.
>
> The shell script attached to https://bugs.python.org/issue23903 should
> be able to do it with different preprocessor values (we originally
> intended to detect inconsistencies in the stable API, but when we found
> lots of existing inconsistencies we couldn't agree on how to deal with
> them).
>
> Cheers,
> Steve
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190123/9662d75d/attachment.html>

From stephan.reiter at gmail.com  Wed Jan 23 05:50:11 2019
From: stephan.reiter at gmail.com (Stephan Reiter)
Date: Wed, 23 Jan 2019 11:50:11 +0100
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
Message-ID: <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>

Hi!

Well, the plugins would be created by third-parties and I'd like them
to enable bunding of modules with their plugins.
I am afraid of modules with the same name, but being different, or
different versions of modules being used by different plugins. If
plugins share an interpreter, the module with a given name that is
imported first sticks around forever and for all plugins.

I am thinking about this design:
- Plugins don't maintain state in their Python world. They expose
functions, my application calls them.
- Everytime I call into them, they are presented with a clean global
namespace. After the call, the namespace (dict) is thrown away. That
releases any objects the plugin code has created.
- So, then I could also actively unload modules they loaded. But I do
know that this is problematic in particular for modules that use
native code.

I am interested in both a short-term and a long-term solution.
Actually, making subinterpreters work better is pretty sexy ...
because it's hard. :-)

Stephan

Am Mi., 23. Jan. 2019 um 11:30 Uhr schrieb Petr Viktorin <encukou at gmail.com>:
>
> On 1/23/19 3:33 AM, Stephan Reiter wrote:
> > Thanks for the answers so far. I appreciate them!
> >
> > Nathaniel, I'd like to allow Python plugins in my application. A
> > plugin should be allowed to bring its own modules along (i.e.
> > plugin-specific subdir is in sys.path when the plugin is active) and
> > hence some isolation of them will be needed, so that they can use
> > different versions of a given module. That's my main motivation for
> > using subinterpreters.
> > I thought about running plugins out-of-processes - a separate process
> > for every plugin - and allow them to communicate with my application
> > via RPC. But that makes it more complex to implement the API my
> > application will offer and will slow down things due to the need to
> > copy data.
> > Maybe you have another idea for me? :)
>
> Try to make the plugins work together. Look into using pip/PyPI for your
> plugins. Try to make it so each package ("plugin") would have only one
> module/package, and dependencies would be other packages that can be
> installed individually and shared. And keep in mind you can set up your
> own package index, or distribute/install individual package files.
>
> If that's not possible, and you want things to work now, go with subprocess.
>
> If you want to help make subinterpreters work better, there are several
> people scratching at the problem from different angles. Most/all would
> welcome help, but don't expect any short-term benefits.
> (FWIW, my own effort is currently blocked on PEP 580, and I hope to move
> forward after a Council is elected.)
>
>
> > Henry, Antoine, thanks for your input; I'll check out the tests and
> > see what I can learn from issue 10915.
> >
> > Stephan
> >
> > Am Di., 22. Jan. 2019 um 22:39 Uhr schrieb Nathaniel Smith <njs at pobox.com>:
> >>
> >> There are currently numerous incompatibilities between numpy and subinterpreters, and no concrete plan for fixing them. The numpy team does not consider subinterpreters to be a supported configuration, and can't help you with any issues you run into. I know the concept of subinterpreters is really appealing, but unfortunately the CPython implementation is not really mature or widely supported... are you absolutely certain you need to use subinterpreters for your application?
> >>
> >> On Tue, Jan 22, 2019, 08:27 Stephan Reiter <stephan.reiter at gmail.com wrote:
> >>>
> >>> Hi all!
> >>>
> >>> I am new to the list and arriving with a concrete problem that I'd
> >>> like to fix myself.
> >>>
> >>> I am embedding Python (3.6) into my C++ application and I would like
> >>> to run Python scripts isolated from each other using sub-interpreters.
> >>> I am not using threads; everything is supposed to run in the
> >>> application's main thread.
> >>>
> >>> I noticed that if I create an interpreter, switch to it and execute
> >>> code that imports numpy (1.13), my application will hang.
> >>>
> >>>    ntdll.dll!NtWaitForSingleObject() Unknown
> >>>    KernelBase.dll!WaitForSingleObjectEx() Unknown
> >>>> python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0, _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245 C
> >>>    [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
> >>>    python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C
> >>>    python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line 370 C
> >>>    python36.dll!PyGILState_Ensure() Line 855 C
> >>>    umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown
> >>>    umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown
> >>>    umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown
> >>>    umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown
> >>>    umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown
> >>>    [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
> >>>    [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C
> >>>    python36.dll!call_function(_object * * *
> >>> pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line
> >>> 4822 C
> >>>
> >>> Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
> >>> PyEval_RestoreThread on the (auto) threadstate of the main
> >>> interpreter. And that's wrong.
> >>> We are already holding the GIL with the threadstate of our current
> >>> sub-interpreter, so there's no need to switch.
> >>>
> >>> I know that the GIL API is not fully compatible with sub-interpreters,
> >>> as issues #10915 and #15751 illustrate.
> >>>
> >>> But since I need to support calls to PyGILState_Ensure - numpy is the
> >>> best example -, I am trying to improve the situation here:
> >>> https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597
> >>>
> >>> That change may be naive, but it does the trick for my use case. If
> >>> totally wrong, I don't mind pursuing another alley.
> >>>
> >>> Essentially, I'd like to ask for some guidance in how to tackle this
> >>> problem while keeping the current GIL API unchanged (to avoid breaking
> >>> modules).
> >>>
> >>> I am also wondering how I can test any changes I am proposing. Is
> >>> there a test suite for interpreters, for example?
> >>>
> >>> Thank you very much,
> >>> Stephan
> >>> _______________________________________________
> >>> Python-Dev mailing list
> >>> Python-Dev at python.org
> >>> https://mail.python.org/mailman/listinfo/python-dev
> >>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com
> >

From vstinner at redhat.com  Wed Jan 23 11:41:29 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 23 Jan 2019 17:41:29 +0100
Subject: [Python-Dev] Source of truth for C-API
In-Reply-To: <CAOMjWkniKXCeWqrx5oE5+0WVrqgtPkpRdvoWyKpS+xBpCY1tgQ@mail.gmail.com>
References: <CAOMjWkmVTdNyhkotcoBn08z_hS7d-tk393xEvaOrMKPFFreLKw@mail.gmail.com>
 <CA+3bQGEWC5m5X_-DD6vJh=YZBZqL+uwzezc=Q2WVSU6Mz9yJcw@mail.gmail.com>
 <3ba8a094-dcb3-13f3-cdb4-2162f4c5e9ac@python.org>
 <CAOMjWkniKXCeWqrx5oE5+0WVrqgtPkpRdvoWyKpS+xBpCY1tgQ@mail.gmail.com>
Message-ID: <CA+3bQGGhpLxz-QcOvm-HH5dbq8M2VurHpENyD_-G=P6VnOQVkA@mail.gmail.com>

What is your change? Did you remove these functions? Change their parameters?

Victor

Le mer. 23 janv. 2019 ? 16:24, Ivan Levkivskyi <levkivskyi at gmail.com> a ?crit :
>
> Thanks for advice Victor and Steve!
>
> I looked at the list, and the two functions I mentioned are not in the list. So I assume the best strategy for now is to wait until first alpha-beta releases are out, and see if anyone complains.
>
> --
> Ivan
>
>
>
> On Wed, 23 Jan 2019 at 06:58, Steve Dower <steve.dower at python.org> wrote:
>>
>> On 22Jan.2019 1517, Victor Stinner wrote:
>> > I'm not aware of any tool to automatically list the content of the C API.
>>
>> The shell script attached to https://bugs.python.org/issue23903 should
>> be able to do it with different preprocessor values (we originally
>> intended to detect inconsistencies in the stable API, but when we found
>> lots of existing inconsistencies we couldn't agree on how to deal with
>> them).
>>
>> Cheers,
>> Steve



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From levkivskyi at gmail.com  Wed Jan 23 11:43:41 2019
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 23 Jan 2019 16:43:41 +0000
Subject: [Python-Dev] Source of truth for C-API
In-Reply-To: <CA+3bQGGhpLxz-QcOvm-HH5dbq8M2VurHpENyD_-G=P6VnOQVkA@mail.gmail.com>
References: <CAOMjWkmVTdNyhkotcoBn08z_hS7d-tk393xEvaOrMKPFFreLKw@mail.gmail.com>
 <CA+3bQGEWC5m5X_-DD6vJh=YZBZqL+uwzezc=Q2WVSU6Mz9yJcw@mail.gmail.com>
 <3ba8a094-dcb3-13f3-cdb4-2162f4c5e9ac@python.org>
 <CAOMjWkniKXCeWqrx5oE5+0WVrqgtPkpRdvoWyKpS+xBpCY1tgQ@mail.gmail.com>
 <CA+3bQGGhpLxz-QcOvm-HH5dbq8M2VurHpENyD_-G=P6VnOQVkA@mail.gmail.com>
Message-ID: <CAOMjWk==qJJuaxBmj8GaNgCY-z_S-Qq3WDitoajUxhgBcg0GhA@mail.gmail.com>

I added to extra parameters to each, see
https://github.com/python/cpython/pull/11605/files#diff-d350c56a842065575842defb8aaa9f27

--
Ivan

On Wed, 23 Jan 2019 at 16:41, Victor Stinner <vstinner at redhat.com> wrote:

> What is your change? Did you remove these functions? Change their
> parameters?
>
> Victor
>
> Le mer. 23 janv. 2019 ? 16:24, Ivan Levkivskyi <levkivskyi at gmail.com> a
> ?crit :
> >
> > Thanks for advice Victor and Steve!
> >
> > I looked at the list, and the two functions I mentioned are not in the
> list. So I assume the best strategy for now is to wait until first
> alpha-beta releases are out, and see if anyone complains.
> >
> > --
> > Ivan
> >
> >
> >
> > On Wed, 23 Jan 2019 at 06:58, Steve Dower <steve.dower at python.org>
> wrote:
> >>
> >> On 22Jan.2019 1517, Victor Stinner wrote:
> >> > I'm not aware of any tool to automatically list the content of the C
> API.
> >>
> >> The shell script attached to https://bugs.python.org/issue23903 should
> >> be able to do it with different preprocessor values (we originally
> >> intended to detect inconsistencies in the stable API, but when we found
> >> lots of existing inconsistencies we couldn't agree on how to deal with
> >> them).
> >>
> >> Cheers,
> >> Steve
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190123/86090a88/attachment.html>

From vstinner at redhat.com  Wed Jan 23 11:52:15 2019
From: vstinner at redhat.com (Victor Stinner)
Date: Wed, 23 Jan 2019 17:52:15 +0100
Subject: [Python-Dev] Source of truth for C-API
In-Reply-To: <CAOMjWk==qJJuaxBmj8GaNgCY-z_S-Qq3WDitoajUxhgBcg0GhA@mail.gmail.com>
References: <CAOMjWkmVTdNyhkotcoBn08z_hS7d-tk393xEvaOrMKPFFreLKw@mail.gmail.com>
 <CA+3bQGEWC5m5X_-DD6vJh=YZBZqL+uwzezc=Q2WVSU6Mz9yJcw@mail.gmail.com>
 <3ba8a094-dcb3-13f3-cdb4-2162f4c5e9ac@python.org>
 <CAOMjWkniKXCeWqrx5oE5+0WVrqgtPkpRdvoWyKpS+xBpCY1tgQ@mail.gmail.com>
 <CA+3bQGGhpLxz-QcOvm-HH5dbq8M2VurHpENyD_-G=P6VnOQVkA@mail.gmail.com>
 <CAOMjWk==qJJuaxBmj8GaNgCY-z_S-Qq3WDitoajUxhgBcg0GhA@mail.gmail.com>
Message-ID: <CA+3bQGH-bUA3+9M305NvfqfPDDydkX0TE6KZWZUcUwTqmFP-Jw@mail.gmail.com>

I suggest you to add a new function and leaves the existing function
unchanged. "Just in case".

You may deprecate the old functions at the same time using Py_DEPRECATED().

Victor

Le mer. 23 janv. 2019 ? 17:44, Ivan Levkivskyi <levkivskyi at gmail.com> a ?crit :
>
> I added to extra parameters to each, see https://github.com/python/cpython/pull/11605/files#diff-d350c56a842065575842defb8aaa9f27
>
> --
> Ivan
>
> On Wed, 23 Jan 2019 at 16:41, Victor Stinner <vstinner at redhat.com> wrote:
>>
>> What is your change? Did you remove these functions? Change their parameters?
>>
>> Victor
>>
>> Le mer. 23 janv. 2019 ? 16:24, Ivan Levkivskyi <levkivskyi at gmail.com> a ?crit :
>> >
>> > Thanks for advice Victor and Steve!
>> >
>> > I looked at the list, and the two functions I mentioned are not in the list. So I assume the best strategy for now is to wait until first alpha-beta releases are out, and see if anyone complains.
>> >
>> > --
>> > Ivan
>> >
>> >
>> >
>> > On Wed, 23 Jan 2019 at 06:58, Steve Dower <steve.dower at python.org> wrote:
>> >>
>> >> On 22Jan.2019 1517, Victor Stinner wrote:
>> >> > I'm not aware of any tool to automatically list the content of the C API.
>> >>
>> >> The shell script attached to https://bugs.python.org/issue23903 should
>> >> be able to do it with different preprocessor values (we originally
>> >> intended to detect inconsistencies in the stable API, but when we found
>> >> lots of existing inconsistencies we couldn't agree on how to deal with
>> >> them).
>> >>
>> >> Cheers,
>> >> Steve
>>
>>
>>
>> --
>> Night gathers, and now my watch begins. It shall not end until my death.



-- 
Night gathers, and now my watch begins. It shall not end until my death.

From ericsnowcurrently at gmail.com  Wed Jan 23 12:11:45 2019
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 23 Jan 2019 10:11:45 -0700
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
Message-ID: <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>

Hi Stephan,

On Tue, Jan 22, 2019 at 9:25 AM Stephan Reiter <stephan.reiter at gmail.com> wrote:
> I am new to the list and arriving with a concrete problem that I'd
> like to fix myself.

That is great!  Statements like that are a good way to get folks
interested in your success. :)

> I am embedding Python (3.6) into my C++ application and I would like
> to run Python scripts isolated from each other using sub-interpreters.
> I am not using threads; everything is supposed to run in the
> application's main thread.

FYI, running multiple interpreters in the same (e.g. main) thread
isn't as well thought out as running them in separate threads.  There
may be assumptions in the runtime that would cause crashes or
inconsistency in the runtime, so be vigilant.  Is there a reason not
to run the subinterpreters in separate threads?

Regarding isolation, keep in mind that there are some limitations.  At
an intrinsic level subinterpreters are never truly isolated since they
run in the same process.  This matters if you have concerns about
security (which you should always consider) and stability (if a
subinterpreter crashes then your whole process crashes).  You can find
that complete isolation via subprocess & multiprocessing.

On top of intrinsic isolation, currently subinterpreters have gaps in
isolation that need fixing.  For instance, they share a lot of
module-global state, as well as builtin types and singletons.  So data
can leak between subinterpreters unexpectedly.

Finally, at the Python level subinterpreters don't have a good way to
pass data around.  (I'm working on that. [1])  Naturally at the C
level you can keep pointers to objects and share data that way.  Just
keep in mind that doing so relies on the GIL (in an
interpreter-per-thread scenario, which you're avoiding).  In a world
where subinterpreters don't share the GIL [2] (and you're running one
interpreter per thread) you'll end up with refcounting races, leading
to crashes.  Just keep that mind if you decide to switch to
one-subinterpreter-per-thread.

On Tue, Jan 22, 2019 at 8:09 PM Stephan Reiter <stephan.reiter at gmail.com> wrote:
> Nathaniel, I'd like to allow Python plugins in my application. A
> plugin should be allowed to bring its own modules along (i.e.
> plugin-specific subdir is in sys.path when the plugin is active) and
> hence some isolation of them will be needed, so that they can use
> different versions of a given module. That's my main motivation for
> using subinterpreters.

That's an interesting approach.  Using subinterpreters would indeed
give you isolation between the sets of imported modules.

As you noticed, you'll run into some problems when extension modules
are involved.  There aren't any great workarounds yet .
Subinterpreters are tied pretty tightly to the core runtime so it's
hard to attack the problem from the outside.  Furthermore,
subinterpreters aren't widely used yet so folks haven't been very
motivated to fix the runtime.  (FWIW, that is changing.)

> I thought about running plugins out-of-processes - a separate process
> for every plugin - and allow them to communicate with my application
> via RPC. But that makes it more complex to implement the API my
> application will offer and will slow down things due to the need to
> copy data.

Yep.  It might be worth it though.  Note that running
plugins/extensions in separate processes is a fairly common approach
for a variety of solid technical reasons (e.g. security, stability).
FWIW, there are some tools available (or soon to be) for sharing data
more efficiently (e.g. shared memory in multiprocessing, PEP 574)

> Maybe you have another idea for me? :)

* single proc -- keep using subinterpreters
  + dlmopen or the Windows equivalent (I hesitate to suggest this
hack, but it might help somewhat with extension modules)
  + help fix the problems with subinterpreters :)
* single proc -- no subinterpreters
  + import hook to put plugins in their own namespace (tricky with
extension modules)
  + extend importlib to do the same
  + swap sys.modules in and out around plugin use
* multi-proc -- one process per plugin
  + subprocess
  + multiprocessing

On Wed, Jan 23, 2019 at 8:48 AM Stephan Reiter <stephan.reiter at gmail.com> wrote:
> Well, the plugins would be created by third-parties and I'd like them
> to enable bunding of modules with their plugins.
> I am afraid of modules with the same name, but being different, or
> different versions of modules being used by different plugins. If
> plugins share an interpreter, the module with a given name that is
> imported first sticks around forever and for all plugins.
>
> I am thinking about this design:
> - Plugins don't maintain state in their Python world. They expose
> functions, my application calls them.
> - Everytime I call into them, they are presented with a clean global
> namespace. After the call, the namespace (dict) is thrown away. That
> releases any objects the plugin code has created.
> - So, then I could also actively unload modules they loaded. But I do
> know that this is problematic in particular for modules that use
> native code.
>
> I am interested in both a short-term and a long-term solution.
> Actually, making subinterpreters work better is pretty sexy ...
> because it's hard. :-)

Petr noted that a number of people are working on getting
subinterpreters to a good place.  That includes me. [1][2] :)  We'd
welcome any help!

-eric


[1] https://www.python.org/dev/peps/pep-0554/
[2] https://github.com/ericsnowcurrently/multi-core-python

From stephan.reiter at gmail.com  Wed Jan 23 14:41:24 2019
From: stephan.reiter at gmail.com (Stephan Reiter)
Date: Wed, 23 Jan 2019 20:41:24 +0100
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
 <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
Message-ID: <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>

You all do make me feel very welcome in this community! Thank you very much! :-)

And thank you for all the thought and time you put into your message,
Eric. I do appreciate in particular all the alternatives you
presented; you provide a good picture of my options.
Not ruling out any of them, I'll stick with (single process + multiple
subinterpreters + plugins can't keep state in Python + all my Python
calls are performed on the main thread) for the time being. That's
quite a limited environment, which I hope I can make work in the long
run. And I think the concept of subinterpreters is nice and I'd like
to spend some time on the challenge of improving the situation.

So, I updated my changes and have the following on top of 3.6.1 at the moment:
https://github.com/stephanreiter/cpython/commit/c1afa0c8cdfab862f409f1c7ff02b189f5191cbe

I did what Henry suggested and ran the Python test suite. On Windows,
with my changes I get as output:

357 tests OK.

2 tests failed:
    test_re test_subprocess

46 tests skipped:
    test_bz2 test_crypt test_curses test_dbm_gnu test_dbm_ndbm
    test_devpoll test_epoll test_fcntl test_fork1 test_gdb test_grp
    test_idle test_ioctl test_kqueue test_lzma test_nis test_openpty
    test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd
    test_readline test_resource test_smtpnet test_socketserver
    test_spwd test_sqlite test_ssl test_syslog test_tcl
    test_threadsignals test_timeout test_tix test_tk test_ttk_guionly
    test_ttk_textonly test_turtle test_urllib2net test_urllibnet
    test_wait3 test_wait4 test_winsound test_xmlrpc_net test_zipfile64

Total duration: 6 min 20 sec
Tests result: FAILURE

I dropped my changes and ran the test suite again using vanilla Python
and got the same result.
So, it seems that the change doesn't break anything that is tested,
but that probably doesn't mean a lot.

Tomorrow, I'll investigate the following situation if I find time:

If we create a fresh OS thread and make it call PyGILState_Ensure, it
won't have a PyThreadState saved under autoTLSkey. That means it will
create one using the main interpreter. I, as the developer embedding
Python into my application and using multiple interpreters, have no
control here. Maybe I know that under current conditions a certain
other interpreter should be used.

I'll try to provoke this situation and then introduce a callback from
Python into my application that will allow me to specify which
interpreter should be used, e.g. code as follows:

PyInterpreter *pickAnInterpreter() {
  return activePlugin ? activePlugin->interpreter : nullptr; //
nullptr maps to main interpreter
}

PyGILState_SetNewThreadInterpreterSelectionCallback(&pickAnInterpreter);

Maybe rubbish. But I think a valuable experiment that will give me a
better understanding.

Stephan

Am Mi., 23. Jan. 2019 um 18:11 Uhr schrieb Eric Snow
<ericsnowcurrently at gmail.com>:
>
> Hi Stephan,
>
> On Tue, Jan 22, 2019 at 9:25 AM Stephan Reiter <stephan.reiter at gmail.com> wrote:
> > I am new to the list and arriving with a concrete problem that I'd
> > like to fix myself.
>
> That is great!  Statements like that are a good way to get folks
> interested in your success. :)
>
> > I am embedding Python (3.6) into my C++ application and I would like
> > to run Python scripts isolated from each other using sub-interpreters.
> > I am not using threads; everything is supposed to run in the
> > application's main thread.
>
> FYI, running multiple interpreters in the same (e.g. main) thread
> isn't as well thought out as running them in separate threads.  There
> may be assumptions in the runtime that would cause crashes or
> inconsistency in the runtime, so be vigilant.  Is there a reason not
> to run the subinterpreters in separate threads?
>
> Regarding isolation, keep in mind that there are some limitations.  At
> an intrinsic level subinterpreters are never truly isolated since they
> run in the same process.  This matters if you have concerns about
> security (which you should always consider) and stability (if a
> subinterpreter crashes then your whole process crashes).  You can find
> that complete isolation via subprocess & multiprocessing.
>
> On top of intrinsic isolation, currently subinterpreters have gaps in
> isolation that need fixing.  For instance, they share a lot of
> module-global state, as well as builtin types and singletons.  So data
> can leak between subinterpreters unexpectedly.
>
> Finally, at the Python level subinterpreters don't have a good way to
> pass data around.  (I'm working on that. [1])  Naturally at the C
> level you can keep pointers to objects and share data that way.  Just
> keep in mind that doing so relies on the GIL (in an
> interpreter-per-thread scenario, which you're avoiding).  In a world
> where subinterpreters don't share the GIL [2] (and you're running one
> interpreter per thread) you'll end up with refcounting races, leading
> to crashes.  Just keep that mind if you decide to switch to
> one-subinterpreter-per-thread.
>
> On Tue, Jan 22, 2019 at 8:09 PM Stephan Reiter <stephan.reiter at gmail.com> wrote:
> > Nathaniel, I'd like to allow Python plugins in my application. A
> > plugin should be allowed to bring its own modules along (i.e.
> > plugin-specific subdir is in sys.path when the plugin is active) and
> > hence some isolation of them will be needed, so that they can use
> > different versions of a given module. That's my main motivation for
> > using subinterpreters.
>
> That's an interesting approach.  Using subinterpreters would indeed
> give you isolation between the sets of imported modules.
>
> As you noticed, you'll run into some problems when extension modules
> are involved.  There aren't any great workarounds yet .
> Subinterpreters are tied pretty tightly to the core runtime so it's
> hard to attack the problem from the outside.  Furthermore,
> subinterpreters aren't widely used yet so folks haven't been very
> motivated to fix the runtime.  (FWIW, that is changing.)
>
> > I thought about running plugins out-of-processes - a separate process
> > for every plugin - and allow them to communicate with my application
> > via RPC. But that makes it more complex to implement the API my
> > application will offer and will slow down things due to the need to
> > copy data.
>
> Yep.  It might be worth it though.  Note that running
> plugins/extensions in separate processes is a fairly common approach
> for a variety of solid technical reasons (e.g. security, stability).
> FWIW, there are some tools available (or soon to be) for sharing data
> more efficiently (e.g. shared memory in multiprocessing, PEP 574)
>
> > Maybe you have another idea for me? :)
>
> * single proc -- keep using subinterpreters
>   + dlmopen or the Windows equivalent (I hesitate to suggest this
> hack, but it might help somewhat with extension modules)
>   + help fix the problems with subinterpreters :)
> * single proc -- no subinterpreters
>   + import hook to put plugins in their own namespace (tricky with
> extension modules)
>   + extend importlib to do the same
>   + swap sys.modules in and out around plugin use
> * multi-proc -- one process per plugin
>   + subprocess
>   + multiprocessing
>
> On Wed, Jan 23, 2019 at 8:48 AM Stephan Reiter <stephan.reiter at gmail.com> wrote:
> > Well, the plugins would be created by third-parties and I'd like them
> > to enable bunding of modules with their plugins.
> > I am afraid of modules with the same name, but being different, or
> > different versions of modules being used by different plugins. If
> > plugins share an interpreter, the module with a given name that is
> > imported first sticks around forever and for all plugins.
> >
> > I am thinking about this design:
> > - Plugins don't maintain state in their Python world. They expose
> > functions, my application calls them.
> > - Everytime I call into them, they are presented with a clean global
> > namespace. After the call, the namespace (dict) is thrown away. That
> > releases any objects the plugin code has created.
> > - So, then I could also actively unload modules they loaded. But I do
> > know that this is problematic in particular for modules that use
> > native code.
> >
> > I am interested in both a short-term and a long-term solution.
> > Actually, making subinterpreters work better is pretty sexy ...
> > because it's hard. :-)
>
> Petr noted that a number of people are working on getting
> subinterpreters to a good place.  That includes me. [1][2] :)  We'd
> welcome any help!
>
> -eric
>
>
> [1] https://www.python.org/dev/peps/pep-0554/
> [2] https://github.com/ericsnowcurrently/multi-core-python

From larry at hastings.org  Wed Jan 23 20:15:53 2019
From: larry at hastings.org (Larry Hastings)
Date: Wed, 23 Jan 2019 17:15:53 -0800
Subject: [Python-Dev] Lost sight
In-Reply-To: <q24upf$h0v$1@ciao.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org> <q24upf$h0v$1@ciao.gmane.org>
Message-ID: <df7c860b-f30e-f4e5-e1a1-58d54e701787@hastings.org>


You have been a marvel, and an enormous boon to the Python community.? 
You should not feel bad about anything.? Best wishes to you for your 
future endeavors!


//arry/

On 1/21/19 7:26 AM, Serhiy Storchaka wrote:
> Thank you very match, all who have expressed compassion here and 
> privately. I am very touched. It at least helped me feel a little 
> better psychologically.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/larry%40hastings.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190123/34f72302/attachment.html>

From steve at holdenweb.com  Thu Jan 24 04:36:35 2019
From: steve at holdenweb.com (Steve Holden)
Date: Thu, 24 Jan 2019 09:36:35 +0000
Subject: [Python-Dev] Lost sight
In-Reply-To: <q1ut2o$pds$1@blaine.gmane.org>
References: <q1ut2o$pds$1@blaine.gmane.org>
Message-ID: <CAMofdRB2iNm-ONckR9zH06vRxeppo3BdjH0X4O0eq5P5AdwhBA@mail.gmail.com>

Like everyone else, I am sorry to hear this news, and extend my sympathies.
It's marvellous that you are still able to work on Python at all, still.
Since you have to be careful not to do too much, you will just have to
choose your battles carefully. Thank you for your many contributions to
Python to date.

Kind regards
 Steve


On Sat, Jan 19, 2019 at 10:14 AM Serhiy Storchaka <storchaka at gmail.com>
wrote:

> I have virtually completely lost the sight of my right eye (and the loss
> is quickly progresses) and the sight of my left eye is weak. That is why
> my activity as a core developer was decreased significantly at recent
> time. My apologies to those who are waiting for my review. I will do it
> slowly.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190124/d6b61f17/attachment.html>

From tjreedy at udel.edu  Thu Jan 24 15:18:05 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 24 Jan 2019 15:18:05 -0500
Subject: [Python-Dev] Add more SyntaxWarnings?
Message-ID: <q2d6i0$63gm$1@blaine.gmane.org>

https://bugs.python.org/issue15248 is about situations like the following:

 >>> [(1,2,3)
      (4,5,6)]
Traceback (most recent call last):
   File "<pyshell#4>", line 2, in <module>
     (4,5,6)]
TypeError: 'tuple' object is not callable

The original poster requested that the error message be augmented with 
something like "(missing preceding comma?)"

Ezio Melotti suggested a FAQ entry like
https://docs.python.org/3/faq/programming.html#why-am-i-getting-an-unboundlocalerror-when-the-variable-has-a-value

(I think such entries below in a separate doc and will try to post on 
python-ideas when I have a prototype.)

Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a 
proof-of-concept diff that handled the above and many similar cases. The 
diff is based on the idea that while we can only positively identify 
'callables' at runtime, we *can* negatively identify many non-callables 
when compiling.  Ditto for subscriptables and indexables.

Serhiy concluded with
"This patch was inspired by usability improvements in GCC 8.
https://developers.redhat.com/blog/2018/03/15/gcc-8-usability-improvements/
I haven't created a pull request because I have doubts about that this 
should be in the compiler rather of a third-party linter. But if several 
other core developers will support this idea I'll continue working in 
this direction."

I was impressed with how clear and readable the patch is and consider it 
a plausible enhancement.  I would like other core developers to comment.

-- 
Terry Jan Reedy


From chris.barker at noaa.gov  Thu Jan 24 16:25:38 2019
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Thu, 24 Jan 2019 21:25:38 +0000
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
 <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
 <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
Message-ID: <CALGmxEJEhsUjhszSL+fk0Vrz4HfJ9QDjGZUgdi_aAXYCO+LYiw@mail.gmail.com>

If your primary concern is module clashes between plugins, maybe you
can hack around that:

1) if the plugins are providing copies of any other modules, then you
can simply require them to put them in their own namespace ? that is,
a plug-in is a single package, with however many sub modules as it may
need.

2) if plugins might require third party packages that need to be
isolated, then maybe you could use an import hook that
re-names/isolates the modules each plugin loads, so they are kept
separate.

I haven?t thought through how to do any of this, but in principle, you
can have the same module loaded twice if it has a different name.


Not that sub interpreters aren?t cool and useful, but you can probably
handle module clashes in a simpler way.

-CHB



Sent from my iPhone

> On Jan 23, 2019, at 11:41 AM, Stephan Reiter <stephan.reiter at gmail.com> wrote:
>
> You all do make me feel very welcome in this community! Thank you very much! :-)
>
> And thank you for all the thought and time you put into your message,
> Eric. I do appreciate in particular all the alternatives you
> presented; you provide a good picture of my options.
> Not ruling out any of them, I'll stick with (single process + multiple
> subinterpreters + plugins can't keep state in Python + all my Python
> calls are performed on the main thread) for the time being. That's
> quite a limited environment, which I hope I can make work in the long
> run. And I think the concept of subinterpreters is nice and I'd like
> to spend some time on the challenge of improving the situation.
>
> So, I updated my changes and have the following on top of 3.6.1 at the moment:
> https://github.com/stephanreiter/cpython/commit/c1afa0c8cdfab862f409f1c7ff02b189f5191cbe
>
> I did what Henry suggested and ran the Python test suite. On Windows,
> with my changes I get as output:
>
> 357 tests OK.
>
> 2 tests failed:
>   test_re test_subprocess
>
> 46 tests skipped:
>   test_bz2 test_crypt test_curses test_dbm_gnu test_dbm_ndbm
>   test_devpoll test_epoll test_fcntl test_fork1 test_gdb test_grp
>   test_idle test_ioctl test_kqueue test_lzma test_nis test_openpty
>   test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd
>   test_readline test_resource test_smtpnet test_socketserver
>   test_spwd test_sqlite test_ssl test_syslog test_tcl
>   test_threadsignals test_timeout test_tix test_tk test_ttk_guionly
>   test_ttk_textonly test_turtle test_urllib2net test_urllibnet
>   test_wait3 test_wait4 test_winsound test_xmlrpc_net test_zipfile64
>
> Total duration: 6 min 20 sec
> Tests result: FAILURE
>
> I dropped my changes and ran the test suite again using vanilla Python
> and got the same result.
> So, it seems that the change doesn't break anything that is tested,
> but that probably doesn't mean a lot.
>
> Tomorrow, I'll investigate the following situation if I find time:
>
> If we create a fresh OS thread and make it call PyGILState_Ensure, it
> won't have a PyThreadState saved under autoTLSkey. That means it will
> create one using the main interpreter. I, as the developer embedding
> Python into my application and using multiple interpreters, have no
> control here. Maybe I know that under current conditions a certain
> other interpreter should be used.
>
> I'll try to provoke this situation and then introduce a callback from
> Python into my application that will allow me to specify which
> interpreter should be used, e.g. code as follows:
>
> PyInterpreter *pickAnInterpreter() {
> return activePlugin ? activePlugin->interpreter : nullptr; //
> nullptr maps to main interpreter
> }
>
> PyGILState_SetNewThreadInterpreterSelectionCallback(&pickAnInterpreter);
>
> Maybe rubbish. But I think a valuable experiment that will give me a
> better understanding.
>
> Stephan
>
> Am Mi., 23. Jan. 2019 um 18:11 Uhr schrieb Eric Snow
> <ericsnowcurrently at gmail.com>:
>>
>> Hi Stephan,
>>
>>> On Tue, Jan 22, 2019 at 9:25 AM Stephan Reiter <stephan.reiter at gmail.com> wrote:
>>> I am new to the list and arriving with a concrete problem that I'd
>>> like to fix myself.
>>
>> That is great!  Statements like that are a good way to get folks
>> interested in your success. :)
>>
>>> I am embedding Python (3.6) into my C++ application and I would like
>>> to run Python scripts isolated from each other using sub-interpreters.
>>> I am not using threads; everything is supposed to run in the
>>> application's main thread.
>>
>> FYI, running multiple interpreters in the same (e.g. main) thread
>> isn't as well thought out as running them in separate threads.  There
>> may be assumptions in the runtime that would cause crashes or
>> inconsistency in the runtime, so be vigilant.  Is there a reason not
>> to run the subinterpreters in separate threads?
>>
>> Regarding isolation, keep in mind that there are some limitations.  At
>> an intrinsic level subinterpreters are never truly isolated since they
>> run in the same process.  This matters if you have concerns about
>> security (which you should always consider) and stability (if a
>> subinterpreter crashes then your whole process crashes).  You can find
>> that complete isolation via subprocess & multiprocessing.
>>
>> On top of intrinsic isolation, currently subinterpreters have gaps in
>> isolation that need fixing.  For instance, they share a lot of
>> module-global state, as well as builtin types and singletons.  So data
>> can leak between subinterpreters unexpectedly.
>>
>> Finally, at the Python level subinterpreters don't have a good way to
>> pass data around.  (I'm working on that. [1])  Naturally at the C
>> level you can keep pointers to objects and share data that way.  Just
>> keep in mind that doing so relies on the GIL (in an
>> interpreter-per-thread scenario, which you're avoiding).  In a world
>> where subinterpreters don't share the GIL [2] (and you're running one
>> interpreter per thread) you'll end up with refcounting races, leading
>> to crashes.  Just keep that mind if you decide to switch to
>> one-subinterpreter-per-thread.
>>
>>> On Tue, Jan 22, 2019 at 8:09 PM Stephan Reiter <stephan.reiter at gmail.com> wrote:
>>> Nathaniel, I'd like to allow Python plugins in my application. A
>>> plugin should be allowed to bring its own modules along (i.e.
>>> plugin-specific subdir is in sys.path when the plugin is active) and
>>> hence some isolation of them will be needed, so that they can use
>>> different versions of a given module. That's my main motivation for
>>> using subinterpreters.
>>
>> That's an interesting approach.  Using subinterpreters would indeed
>> give you isolation between the sets of imported modules.
>>
>> As you noticed, you'll run into some problems when extension modules
>> are involved.  There aren't any great workarounds yet .
>> Subinterpreters are tied pretty tightly to the core runtime so it's
>> hard to attack the problem from the outside.  Furthermore,
>> subinterpreters aren't widely used yet so folks haven't been very
>> motivated to fix the runtime.  (FWIW, that is changing.)
>>
>>> I thought about running plugins out-of-processes - a separate process
>>> for every plugin - and allow them to communicate with my application
>>> via RPC. But that makes it more complex to implement the API my
>>> application will offer and will slow down things due to the need to
>>> copy data.
>>
>> Yep.  It might be worth it though.  Note that running
>> plugins/extensions in separate processes is a fairly common approach
>> for a variety of solid technical reasons (e.g. security, stability).
>> FWIW, there are some tools available (or soon to be) for sharing data
>> more efficiently (e.g. shared memory in multiprocessing, PEP 574)
>>
>>> Maybe you have another idea for me? :)
>>
>> * single proc -- keep using subinterpreters
>> + dlmopen or the Windows equivalent (I hesitate to suggest this
>> hack, but it might help somewhat with extension modules)
>> + help fix the problems with subinterpreters :)
>> * single proc -- no subinterpreters
>> + import hook to put plugins in their own namespace (tricky with
>> extension modules)
>> + extend importlib to do the same
>> + swap sys.modules in and out around plugin use
>> * multi-proc -- one process per plugin
>> + subprocess
>> + multiprocessing
>>
>>> On Wed, Jan 23, 2019 at 8:48 AM Stephan Reiter <stephan.reiter at gmail.com> wrote:
>>> Well, the plugins would be created by third-parties and I'd like them
>>> to enable bunding of modules with their plugins.
>>> I am afraid of modules with the same name, but being different, or
>>> different versions of modules being used by different plugins. If
>>> plugins share an interpreter, the module with a given name that is
>>> imported first sticks around forever and for all plugins.
>>>
>>> I am thinking about this design:
>>> - Plugins don't maintain state in their Python world. They expose
>>> functions, my application calls them.
>>> - Everytime I call into them, they are presented with a clean global
>>> namespace. After the call, the namespace (dict) is thrown away. That
>>> releases any objects the plugin code has created.
>>> - So, then I could also actively unload modules they loaded. But I do
>>> know that this is problematic in particular for modules that use
>>> native code.
>>>
>>> I am interested in both a short-term and a long-term solution.
>>> Actually, making subinterpreters work better is pretty sexy ...
>>> because it's hard. :-)
>>
>> Petr noted that a number of people are working on getting
>> subinterpreters to a good place.  That includes me. [1][2] :)  We'd
>> welcome any help!
>>
>> -eric
>>
>>
>> [1] https://www.python.org/dev/peps/pep-0554/
>> [2] https://github.com/ericsnowcurrently/multi-core-python
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

From nas-python at arctrix.com  Thu Jan 24 17:01:26 2019
From: nas-python at arctrix.com (Neil Schemenauer)
Date: Thu, 24 Jan 2019 16:01:26 -0600
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <q2d6i0$63gm$1@blaine.gmane.org>
References: <q2d6i0$63gm$1@blaine.gmane.org>
Message-ID: <20190124220126.luvclamlqxl2paxh@python.ca>

On 2019-01-24, Terry Reedy wrote:
> Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a
> proof-of-concept diff that handled the above and many similar cases.

I believe that in general we should give better errors or warnings
if we can do it without huge difficulty.  Serhiy's patch is quite
simple.  The same check *could* be done by a linting tool.  Putting
it in CPython will make it more widely available.  These checks
could be helpful to beginners who probably won't have linting tools
setup.

I think we should not make it an error, otherwise we have changed
Python "the language".  We don't want to force other Python
implementations to do the same check.  It might be hard for them to
implement.  So, SyntaxWarning seems like a reasonable compromise.

Regards,

  Neil

From guido at python.org  Thu Jan 24 17:25:32 2019
From: guido at python.org (Guido van Rossum)
Date: Thu, 24 Jan 2019 14:25:32 -0800
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <20190124220126.luvclamlqxl2paxh@python.ca>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
Message-ID: <CAP7+vJKLOXP-2cHDv+af2M0jt_Rfjue+FSkRxX64nDifO_rxvw@mail.gmail.com>

In this case I agree that a SyntaxWarning is a good idea, per Serhiy's
patch. I would be even more conservative, and only warn if the first object
is a tuple -- the case of the missing comma in the original example appears
likely enough, but I don't expect people to write e.g. `[[1, 2], (3, 4)]`
very often, so leaving the comma out there would be very unlikely.

Regarding the issue of when it's appropriate to issue a SyntaxWarning vs.
when to leave it up to linters, again I would recommend great caution and
only warn about code that is *definitely* going to fail when executed. (Or
at least is *definitely* not going to please the programmer -- one of the
first cases where we added a SyntaxWarning was actually `assert(condition,
message)`, which "fails" by never failing. :-) But this is one of those
cases.

On Thu, Jan 24, 2019 at 2:10 PM Neil Schemenauer <nas-python at arctrix.com>
wrote:

> On 2019-01-24, Terry Reedy wrote:
> > Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a
> > proof-of-concept diff that handled the above and many similar cases.
>
> I believe that in general we should give better errors or warnings
> if we can do it without huge difficulty.  Serhiy's patch is quite
> simple.  The same check *could* be done by a linting tool.  Putting
> it in CPython will make it more widely available.  These checks
> could be helpful to beginners who probably won't have linting tools
> setup.
>
> I think we should not make it an error, otherwise we have changed
> Python "the language".  We don't want to force other Python
> implementations to do the same check.  It might be hard for them to
> implement.  So, SyntaxWarning seems like a reasonable compromise.
>
> Regards,
>
>   Neil
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190124/5e8598a5/attachment.html>

From steve at pearwood.info  Thu Jan 24 17:40:35 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 25 Jan 2019 09:40:35 +1100
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <20190124220126.luvclamlqxl2paxh@python.ca>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
Message-ID: <20190124224035.GB26605@ando.pearwood.info>

On Thu, Jan 24, 2019 at 04:01:26PM -0600, Neil Schemenauer wrote:
> On 2019-01-24, Terry Reedy wrote:
> > Serhiy Storchaka suggested a compiler SyntaxWarning and uploaded a
> > proof-of-concept diff that handled the above and many similar cases.
> 
> I believe that in general we should give better errors or warnings
> if we can do it without huge difficulty.  Serhiy's patch is quite
> simple.  The same check *could* be done by a linting tool.  Putting
> it in CPython will make it more widely available.  These checks
> could be helpful to beginners who probably won't have linting tools
> setup.

+1 to what Neil says here.

> I think we should not make it an error, otherwise we have changed
> Python "the language".

We're allowed to, we're the ones who say what the language is :-)


The remainder of my comments are more speculative.

> We don't want to force other Python
> implementations to do the same check.  It might be hard for them to
> implement.  So, SyntaxWarning seems like a reasonable compromise.

We could say that implementations are allowed to raise errors at compile 
time instead of run time, but aren't required to. Then it becomes a 
matter of "quality of implementation".

For literal ints, strings, None, etc we can tell at compile time that an 
error will occur. All of these cannot fail to raise (short of changing 
the interpreter, in which case you're not using Python anymore):

    1 + "1"  # raises TypeError
    None[x]  # TypeError
    1.234(x)  # TypeError
    "spam".idnex("a")  # AttributeError

In these specific cases, there is nothing wrong with the *syntax*, but a 
compiler should be permitted to immediately raise the same exception 
that would otherwise occur at run time. This is a form of peephole 
optimization, I guess.

If people truly wanted to delay the exception, they could either turn 
off the peephole optimizer, or work around it:

    a = 1; a + "1"

Thoughts?


-- 
Steve

From rosuav at gmail.com  Thu Jan 24 17:52:53 2019
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 25 Jan 2019 09:52:53 +1100
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <20190124224035.GB26605@ando.pearwood.info>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
Message-ID: <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>

On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano <steve at pearwood.info> wrote:
> We could say that implementations are allowed to raise errors at compile
> time instead of run time, but aren't required to. Then it becomes a
> matter of "quality of implementation".
>
> For literal ints, strings, None, etc we can tell at compile time that an
> error will occur. All of these cannot fail to raise (short of changing
> the interpreter, in which case you're not using Python anymore):
>
>     1 + "1"  # raises TypeError
>     None[x]  # TypeError
>     1.234(x)  # TypeError
>     "spam".idnex("a")  # AttributeError
>
> In these specific cases, there is nothing wrong with the *syntax*, but a
> compiler should be permitted to immediately raise the same exception
> that would otherwise occur at run time. This is a form of peephole
> optimization, I guess.

+1. If it's something that the peephole optimizer is already allowed
to change (eg "1"+"1" is constant-folded) and there is absolutely no
way that it can ever be changed at run time, then raising at compile
time can't hurt [1]. It'd be as implementation-dependent and
version-dependent as the peephole optimizer itself.

Does there need to be a new subclass of SyntaxError for "Technically
Valid But Not Meaningful" problems? Is there value in distinguishing
"InevitableTypeError" from "InevitableAttributeError"?

ChrisA

[1] Yes, I know about XKCD 1172, but if someone's saying "if
shouldnt_happen: None[None]" then that's their problem.

From guido at python.org  Thu Jan 24 18:24:41 2019
From: guido at python.org (Guido van Rossum)
Date: Thu, 24 Jan 2019 15:24:41 -0800
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
Message-ID: <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>

On Thu, Jan 24, 2019 at 2:55 PM Chris Angelico <rosuav at gmail.com> wrote:

> On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano <steve at pearwood.info>
> wrote:
> > We could say that implementations are allowed to raise errors at compile
> > time instead of run time, but aren't required to. Then it becomes a
> > matter of "quality of implementation".
> >
> > For literal ints, strings, None, etc we can tell at compile time that an
> > error will occur. All of these cannot fail to raise (short of changing
> > the interpreter, in which case you're not using Python anymore):
> >
> >     1 + "1"  # raises TypeError
> >     None[x]  # TypeError
> >     1.234(x)  # TypeError
> >     "spam".idnex("a")  # AttributeError
> >
> > In these specific cases, there is nothing wrong with the *syntax*, but a
> > compiler should be permitted to immediately raise the same exception
> > that would otherwise occur at run time. This is a form of peephole
> > optimization, I guess.
>
> +1. If it's something that the peephole optimizer is already allowed
> to change (eg "1"+"1" is constant-folded) and there is absolutely no
> way that it can ever be changed at run time, then raising at compile
> time can't hurt [1]. It'd be as implementation-dependent and
> version-dependent as the peephole optimizer itself.
>

I'm -1 on all of these cases. There's nothing mysterious about e.g.
`TypeError: unsupported operand type(s) for +: 'int' and 'str'`, unlike the
case of the two concatenated tuples. (Surely people get errors about
int+str all the time, and they've never complained -- unlike the tuple
tuple case.)


> Does there need to be a new subclass of SyntaxError for "Technically
> Valid But Not Meaningful" problems? Is there value in distinguishing
> "InevitableTypeError" from "InevitableAttributeError"?
>

I don't think there's a *general* problem to be solved here.


> ChrisA
>
> [1] Yes, I know about XKCD 1172, but if someone's saying "if
> shouldnt_happen: None[None]" then that's their problem.
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190124/818c1fcd/attachment.html>

From chris.barker at noaa.gov  Thu Jan 24 18:45:49 2019
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Thu, 24 Jan 2019 23:45:49 +0000
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
Message-ID: <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>

>. There's nothing mysterious about e.g. `TypeError: unsupported operand
type(s) for +: 'int' and 'str'`, unlike the case of the two concatenated
tuples. (Surely people get errors about int+str all the time, and they've
never complained -- unlike the tuple tuple case.)

Well, yes, that particular example is pretty clear. But as a rule, there
are a LOT of errors that can be pretty mysterious to newbies.

I would love to see Python become generally more informative with errors.

In this case, you?d probably get a similar error, but it?s still nice to
get it sooner, and if the hooks are in place, We could have others that are
really helpful.

-CHB


Is there value in distinguishing
> "InevitableTypeError" from "InevitableAttributeError"?
>

I don?t think so ? what we need are helpful error messages. If it will be
raised at compile time, then it won?t generally be catchable in s
try-except? so the actual exception type isn?t very important.

-CHB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190124/f7b9f0b7/attachment-0001.html>

From guido at python.org  Thu Jan 24 18:58:50 2019
From: guido at python.org (Guido van Rossum)
Date: Thu, 24 Jan 2019 15:58:50 -0800
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
Message-ID: <CAP7+vJLwu4d-iOPJ0+X7fcbrVDYZ2+gV+6sxnDneZO73UK2=dQ@mail.gmail.com>

On Thu, Jan 24, 2019 at 3:45 PM Chris Barker - NOAA Federal <
chris.barker at noaa.gov> wrote:

> >. There's nothing mysterious about e.g. `TypeError: unsupported operand
> type(s) for +: 'int' and 'str'`, unlike the case of the two concatenated
> tuples. (Surely people get errors about int+str all the time, and they've
> never complained -- unlike the tuple tuple case.)
>
> Well, yes, that particular example is pretty clear. But as a rule, there
> are a LOT of errors that can be pretty mysterious to newbies.
>
> I would love to see Python become generally more informative with errors.
>
> In this case, you?d probably get a similar error, but it?s still nice to
> get it sooner, and if the hooks are in place, We could have others that are
> really helpful.
>

I doubt people are writing `42 + "abc"` by accident. They'll write `x + y`
and by accident the types won't match. So better error messages at runtime
would help. But I doubt we'll see much mileage out of the syntax checks.
And making things work different based on whether it's a literal or a
variable isn't very Pythonic.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190124/5145fa93/attachment.html>

From eric at trueblade.com  Thu Jan 24 18:32:08 2019
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 24 Jan 2019 18:32:08 -0500
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <b450a99a-d4b7-4c49-4e79-3df3707967dc@trueblade.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <b450a99a-d4b7-4c49-4e79-3df3707967dc@trueblade.com>
Message-ID: <d905f8f0-d9a4-0423-5762-206cabbb6f2b@trueblade.com>

On 1/24/2019 6:16 PM, Eric V. Smith wrote:
> On 1/24/2019 5:52 PM, Chris Angelico wrote:
>> +1. If it's something that the peephole optimizer is already allowed
>> to change (eg "1"+"1" is constant-folded) and there is absolutely no
>> way that it can ever be changed at run time, then raising at compile
>> time can't hurt [1]. It'd be as implementation-dependent and
>> version-dependent as the peephole optimizer itself.
>
> It would be a change if the code is never called. I'm not sure we care 
> about code that's never called, but it is a change.
>
Which Chris already pointed out, in some text that I inadvertently 
deleted. I didn't catch his meaning, which is my fault.

Eric


From eric at trueblade.com  Thu Jan 24 18:16:32 2019
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 24 Jan 2019 18:16:32 -0500
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
Message-ID: <b450a99a-d4b7-4c49-4e79-3df3707967dc@trueblade.com>

On 1/24/2019 5:52 PM, Chris Angelico wrote:
> On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano <steve at pearwood.info> wrote:
>> We could say that implementations are allowed to raise errors at compile
>> time instead of run time, but aren't required to. Then it becomes a
>> matter of "quality of implementation".
>>
>> For literal ints, strings, None, etc we can tell at compile time that an
>> error will occur. All of these cannot fail to raise (short of changing
>> the interpreter, in which case you're not using Python anymore):
>>
>>      1 + "1"  # raises TypeError
>>      None[x]  # TypeError
>>      1.234(x)  # TypeError
>>      "spam".idnex("a")  # AttributeError
>>
>> In these specific cases, there is nothing wrong with the *syntax*, but a
>> compiler should be permitted to immediately raise the same exception
>> that would otherwise occur at run time. This is a form of peephole
>> optimization, I guess.
> 
> +1. If it's something that the peephole optimizer is already allowed
> to change (eg "1"+"1" is constant-folded) and there is absolutely no
> way that it can ever be changed at run time, then raising at compile
> time can't hurt [1]. It'd be as implementation-dependent and
> version-dependent as the peephole optimizer itself.

It would be a change if the code is never called. I'm not sure we care 
about code that's never called, but it is a change.

Eric


From steve at pearwood.info  Thu Jan 24 21:05:10 2019
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 25 Jan 2019 13:05:10 +1100
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
Message-ID: <20190125020510.GC26605@ando.pearwood.info>

At the cost of breaking threading a bit, I'm going to reply to a few 
different people in one post.

On Fri, Jan 25, 2019 at 09:52:53AM +1100, Chris Angelico wrote:
> On Fri, Jan 25, 2019 at 9:42 AM Steven D'Aprano <steve at pearwood.info> wrote:
> > We could say that implementations are allowed to raise errors at compile
> > time instead of run time, but aren't required to. Then it becomes a
> > matter of "quality of implementation".
[...]

> Does there need to be a new subclass of SyntaxError for "Technically
> Valid But Not Meaningful" problems? Is there value in distinguishing
> "InevitableTypeError" from "InevitableAttributeError"?

If we did this (which is looking less and less likely...) I would argue 
for plain old TypeError and AttributeError. There should be no 
difference in the exception raised, only in the timing (compile time 
versus run time).



On Thu, Jan 24, 2019 at 11:45:49PM +0000, Chris Barker wrote:
> I would love to see Python become generally more informative with 
> errors.
>
> In this case, you?d probably get a similar error, but it?s still nice 
> to get it sooner

That is precisely my thinking.



On Thu, Jan 24, 2019 at 03:24:41PM -0800, Guido van Rossum wrote:
> I'm -1 on all of these cases. There's nothing mysterious about e.g.
> `TypeError: unsupported operand type(s) for +: 'int' and 'str'`

Its not about the error being mysterious, its about raising the 
exception as early as possible. It shouldn't take 20 years to discover 
that this snippet buried deep in a large code base:

    if datetime.date.today().year == 2029:
        spam = 1 + '1'

will fail :-)



On Thu, Jan 24, 2019 at 03:58:50PM -0800, Guido van Rossum wrote:
> But I doubt we'll see much mileage out of the syntax checks.

Frankly, I agree, so I'm not going to argue hard for this. It was more 
speculation than a solid proposal.

A project manager I used to work with would have dismissed this as "turd 
polishing", but I prefer to think of it as "craftsmanship" -- a bit like 
a carpenter ensuring that his screws don't protrude above the surface of 
the wood. Sure, its not going to hold the wood together any better, but 
its just a nicer end-user experience :-)

I expect that the cost of implementing these checks would probably 
exceed the benefit of detecting the errors early, but I could be wrong. 
Maybe it would be easy to implement.

But if there is strong opposition to it, I'm not going to champion the 
idea any further.



-- 
Steve

From turnbull.stephen.fw at u.tsukuba.ac.jp  Thu Jan 24 23:06:26 2019
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Fri, 25 Jan 2019 13:06:26 +0900
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
Message-ID: <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>

Chris Barker - NOAA Federal via Python-Dev writes:

 > Well, yes, that particular example is pretty clear. But as a rule,
 > there are a LOT of errors that can be pretty mysterious to newbies.

Isn't that the very definition of "newbie"?  That's half a joke, but I
really don't think that programmers new to Python should be the
standard.  The problematic cases are those where even a relatively
experienced Python programmer needs to be told why an error is raised,
because it's too hard to figure out from background knowledge of the
language, you need to know about implementation internals.

 > I would love to see Python become generally more informative with
 > errors.

I would love to see all automated systems become more informative with
errors!  One thing I like about Python the development community is
that people are doing something about it.  Slowly but surely ....

 > I don?t think so ? what we need are helpful error messages. If it
 > will be raised at compile time, then it won?t generally be
 > catchable in s try-except? so the actual exception type isn?t very
 > important.

+1

In general depending much on exception type is a hit-or-miss affair.

From status at bugs.python.org  Fri Jan 25 13:07:54 2019
From: status at bugs.python.org (Python tracker)
Date: Fri, 25 Jan 2019 18:07:54 +0000
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20190125180754.1.11345FE25A15E0CB@roundup.psfhosted.org>


ACTIVITY SUMMARY (2019-01-18 - 2019-01-25)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    6953 (+21)
  closed 40626 (+31)
  total  47579 (+52)

Open issues with patches: 2771 


Issues opened (39)
==================

#33416: Add endline and endcolumn to every AST node
https://bugs.python.org/issue33416  reopened by levkivskyi

#34656: [CVE-2018-20406] memory exhaustion in Modules/_pickle.c:1393
https://bugs.python.org/issue34656  reopened by vstinner

#35711: Print information about an unexpectedly pending error before c
https://bugs.python.org/issue35711  reopened by gregory.p.smith

#35780: Recheck logic in the C version of the lru_cache()
https://bugs.python.org/issue35780  opened by rhettinger

#35782: Missing whitespace after comma in randrange raise error
https://bugs.python.org/issue35782  opened by louielu

#35783: incorrect example of fetching messages in imaplib documentatio
https://bugs.python.org/issue35783  opened by Max Varnar

#35784: document that hashlib.new takes kwargs
https://bugs.python.org/issue35784  opened by joernheissler

#35786: get_lock() method is not present for Values created using mult
https://bugs.python.org/issue35786  opened by Lorenzo Persichetti

#35790: Correct a statement about sys.exc_info() values restoration
https://bugs.python.org/issue35790  opened by maggyero

#35792: Specifying AbstractEventLoop.run_in_executor as a coroutine co
https://bugs.python.org/issue35792  opened by chrahunt

#35795: test_pkgutil test_zipapp  fail in AMD64 Windows7 SP1 3.x and A
https://bugs.python.org/issue35795  opened by pablogsal

#35796: time.localtime returns error for negative values
https://bugs.python.org/issue35796  opened by mba

#35797: concurrent.futures.ProcessPoolExecutor does not work in venv o
https://bugs.python.org/issue35797  opened by chrullrich

#35798: duplicate SyntaxWarning: "is" with a literal
https://bugs.python.org/issue35798  opened by jwilk

#35799: fix or remove smtpd.PureProxy
https://bugs.python.org/issue35799  opened by samuelcolvin

#35800: remove smtpd.MailmanProxy
https://bugs.python.org/issue35800  opened by samuelcolvin

#35801: venv in 3.7 references python3 executable
https://bugs.python.org/issue35801  opened by Paul Watson

#35802: os.stat / os.lstat always present, but code checks hastattr(os
https://bugs.python.org/issue35802  opened by Anthony Sottile

#35803: Test and document that `dir=...` in tempfile may be PathLike
https://bugs.python.org/issue35803  opened by Anthony Sottile

#35805: email package folds msg-id identifiers using RFC2047 encoded w
https://bugs.python.org/issue35805  opened by mjpieters

#35807: Update bundled pip to 19.0
https://bugs.python.org/issue35807  opened by pradyunsg

#35808: Let's retire pgen
https://bugs.python.org/issue35808  opened by gvanrossum

#35809: test_concurrent_futures.ProcessPoolForkExecutorDeadlockTest fa
https://bugs.python.org/issue35809  opened by xtreak

#35810: Object Initialization Bug with Heap-allocated Types
https://bugs.python.org/issue35810  opened by eelizondo

#35811: py.exe should unset the __PYVENV_LAUNCHER__ environment variab
https://bugs.python.org/issue35811  opened by eryksun

#35812: Don't log an exception from the main coroutine in asyncio.run(
https://bugs.python.org/issue35812  opened by asvetlov

#35813: shared memory construct to avoid need for serialization betwee
https://bugs.python.org/issue35813  opened by davin

#35815: Able to instantiate a subclass with abstract methods from __in
https://bugs.python.org/issue35815  opened by jbasko

#35816: csv.DictReader, skipinitialspace does not ignore tabs
https://bugs.python.org/issue35816  opened by andre.lehmann

#35818: test_email: test_localtime_daylight_false_dst_true() fails if 
https://bugs.python.org/issue35818  opened by schwab

#35820: Inconsistent behavior when parsing IP address
https://bugs.python.org/issue35820  opened by tinutomson

#35821: Clarify when logging events are propagated when propagate is t
https://bugs.python.org/issue35821  opened by chris.jerdonek

#35822: _queue _queuemodule.c is missing inside the Setup file
https://bugs.python.org/issue35822  opened by Igor Z

#35823: Use vfork() in subprocess on Linux
https://bugs.python.org/issue35823  opened by izbyshev

#35824: http.cookies._CookiePattern modifying regular expressions
https://bugs.python.org/issue35824  opened by MeiK

#35825: Py_UNICODE_SIZE=4 fails to link on Windows
https://bugs.python.org/issue35825  opened by kristof

#35826: Typo in example for async with statement with condition
https://bugs.python.org/issue35826  opened by mhchia

#35827: C API dictionary views type checkers are not documented
https://bugs.python.org/issue35827  opened by salty-horse

#35828: test_multiprocessing_* tests - success versus fail varies over
https://bugs.python.org/issue35828  opened by Michael.Felt



Most recent 15 issues with no replies (15)
==========================================

#35828: test_multiprocessing_* tests - success versus fail varies over
https://bugs.python.org/issue35828

#35827: C API dictionary views type checkers are not documented
https://bugs.python.org/issue35827

#35825: Py_UNICODE_SIZE=4 fails to link on Windows
https://bugs.python.org/issue35825

#35822: _queue _queuemodule.c is missing inside the Setup file
https://bugs.python.org/issue35822

#35815: Able to instantiate a subclass with abstract methods from __in
https://bugs.python.org/issue35815

#35813: shared memory construct to avoid need for serialization betwee
https://bugs.python.org/issue35813

#35812: Don't log an exception from the main coroutine in asyncio.run(
https://bugs.python.org/issue35812

#35808: Let's retire pgen
https://bugs.python.org/issue35808

#35807: Update bundled pip to 19.0
https://bugs.python.org/issue35807

#35803: Test and document that `dir=...` in tempfile may be PathLike
https://bugs.python.org/issue35803

#35801: venv in 3.7 references python3 executable
https://bugs.python.org/issue35801

#35799: fix or remove smtpd.PureProxy
https://bugs.python.org/issue35799

#35792: Specifying AbstractEventLoop.run_in_executor as a coroutine co
https://bugs.python.org/issue35792

#35790: Correct a statement about sys.exc_info() values restoration
https://bugs.python.org/issue35790

#35784: document that hashlib.new takes kwargs
https://bugs.python.org/issue35784



Most recent 15 issues waiting for review (15)
=============================================

#35823: Use vfork() in subprocess on Linux
https://bugs.python.org/issue35823

#35813: shared memory construct to avoid need for serialization betwee
https://bugs.python.org/issue35813

#35810: Object Initialization Bug with Heap-allocated Types
https://bugs.python.org/issue35810

#35803: Test and document that `dir=...` in tempfile may be PathLike
https://bugs.python.org/issue35803

#35802: os.stat / os.lstat always present, but code checks hastattr(os
https://bugs.python.org/issue35802

#35800: remove smtpd.MailmanProxy
https://bugs.python.org/issue35800

#35798: duplicate SyntaxWarning: "is" with a literal
https://bugs.python.org/issue35798

#35790: Correct a statement about sys.exc_info() values restoration
https://bugs.python.org/issue35790

#35782: Missing whitespace after comma in randrange raise error
https://bugs.python.org/issue35782

#35780: Recheck logic in the C version of the lru_cache()
https://bugs.python.org/issue35780

#35775: Add a general selection function to statistics
https://bugs.python.org/issue35775

#35769: IDLE: change new file name from  ''Untitled" to "untitled"
https://bugs.python.org/issue35769

#35767: unittest loader doesn't work with partial test functions
https://bugs.python.org/issue35767

#35766: Merge typed_ast back into CPython
https://bugs.python.org/issue35766

#35759: inspect module does not implement introspection API for asynch
https://bugs.python.org/issue35759



Top 10 most discussed issues (10)
=================================

#35797: concurrent.futures.ProcessPoolExecutor does not work in venv o
https://bugs.python.org/issue35797  13 msgs

#35780: Recheck logic in the C version of the lru_cache()
https://bugs.python.org/issue35780  10 msgs

#17005: Add a topological sort algorithm
https://bugs.python.org/issue17005   9 msgs

#35811: py.exe should unset the __PYVENV_LAUNCHER__ environment variab
https://bugs.python.org/issue35811   9 msgs

#27035: Cannot set exit code in atexit callback
https://bugs.python.org/issue27035   8 msgs

#35224: PEP 572: Assignment Expressions
https://bugs.python.org/issue35224   8 msgs

#35707: time.sleep() should support objects with __float__
https://bugs.python.org/issue35707   8 msgs

#34656: [CVE-2018-20406] memory exhaustion in Modules/_pickle.c:1393
https://bugs.python.org/issue34656   6 msgs

#35537: use os.posix_spawn in subprocess
https://bugs.python.org/issue35537   6 msgs

#20479: Efficiently support weight/frequency mappings in the statistic
https://bugs.python.org/issue20479   5 msgs



Issues closed (33)
==================

#14802: Python fails to compile with VC11 ARM configuration
https://bugs.python.org/issue14802  closed by pitrou

#15613: argparse ArgumentDefaultsHelpFormatter interacts badly with --
https://bugs.python.org/issue15613  closed by cheryl.sabella

#20239: Allow repeated deletion of unittest.mock.Mock attributes
https://bugs.python.org/issue20239  closed by pablogsal

#20911: urllib 'headers' is not a well defined data type
https://bugs.python.org/issue20911  closed by demian.brecht

#32603: Deprecation warning on strings used in re module
https://bugs.python.org/issue32603  closed by pablogsal

#34134: multiprocessing memory huge usage
https://bugs.python.org/issue34134  closed by pitrou

#35486: subprocess module import hooks breaks back compatibility
https://bugs.python.org/issue35486  closed by ncoghlan

#35507: multiprocessing: seg fault when creating RawArray from numpy c
https://bugs.python.org/issue35507  closed by xtreak

#35627: multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7
https://bugs.python.org/issue35627  closed by pitrou

#35680: [2.7] Coverity scan: Passing freed pointer "name" as an argume
https://bugs.python.org/issue35680  closed by cstratak

#35699: distutils cannot find Build Tools 2017 since 3.7.2
https://bugs.python.org/issue35699  closed by ned.deily

#35720: Memory leak in Modules/main.c:pymain_parse_cmdline_impl when u
https://bugs.python.org/issue35720  closed by vstinner

#35722: disable_existing_loggers does not apply to the root logger
https://bugs.python.org/issue35722  closed by vinay.sajip

#35733: isinstance(ast.Constant(value=True), ast.Num) should be False
https://bugs.python.org/issue35733  closed by serhiy.storchaka

#35756: Using `return value` in a generator function skips the returne
https://bugs.python.org/issue35756  closed by terry.reedy

#35770: IDLE: python -m idlelib fails on master on Mac OS 10.10.4
https://bugs.python.org/issue35770  closed by terry.reedy

#35772: test_tarfile fails on ppc64le when using tmpfs filesystem
https://bugs.python.org/issue35772  closed by vstinner

#35777: mismatched eval() and ast.literal_eval() behavior with unicode
https://bugs.python.org/issue35777  closed by steven.daprano

#35778: RF: ``pathlib.Path.checksum()`` member
https://bugs.python.org/issue35778  closed by pitrou

#35779: Print friendly version message in REPL
https://bugs.python.org/issue35779  closed by eric.snow

#35781: `logger.warn` method is used in "Logging HOWTO" documentation 
https://bugs.python.org/issue35781  closed by vinay.sajip

#35785: argparse crashes in gettext when processing missing arguments
https://bugs.python.org/issue35785  closed by eric.fahlgren

#35787: shlex.split inserts extra item on backslash space space
https://bugs.python.org/issue35787  closed by eric.smith

#35788: smtpd.PureProxy and smtpd.MailmanProxy broken by extra kwargs,
https://bugs.python.org/issue35788  closed by r.david.murray

#35789: Typo in unittest.mock docs
https://bugs.python.org/issue35789  closed by jamercee

#35791: Unexpected exception with importlib
https://bugs.python.org/issue35791  closed by ncoghlan

#35793: round() doesn't return the right value when I put 0.5 in it.
https://bugs.python.org/issue35793  closed by ned.deily

#35794: test_posix.py test failure
https://bugs.python.org/issue35794  closed by pablogsal

#35804: v3.6.8 _ctypes win32 compiled with pgo crash
https://bugs.python.org/issue35804  closed by steve.dower

#35806: typing module adds objects to sys.modules that don't look like
https://bugs.python.org/issue35806  closed by gvanrossum

#35814: Syntax quirk with variable annotations
https://bugs.python.org/issue35814  closed by levkivskyi

#35817: IDLE 2.713 on debian 9.6 over WSL W10 IdentationError
https://bugs.python.org/issue35817  closed by terry.reedy

#35819: Fatal Python error
https://bugs.python.org/issue35819  closed by zach.ware

From ncoghlan at gmail.com  Sun Jan 27 09:01:36 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 28 Jan 2019 00:01:36 +1000
Subject: [Python-Dev] Anyone else seeing a lack of caching in local docs
 builds?
Message-ID: <CADiSq7fMghm44aPg7W4XW=xhVgdWO=FwLWynEcS77fLeh7d7XA@mail.gmail.com>

Hi folks,

I'm currently seeing a behaviour where every time I run "make html",
all 474 source files get rebuilt.

For now, I'm assuming I've messed something up with my local docs
build setup, but figured I'd ask if anyone else was seeing this, in
case it was actually broken at the build level (CI wouldn't pick this
up, since it always builds from scratch anyway).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jan 27 09:04:47 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 28 Jan 2019 00:04:47 +1000
Subject: [Python-Dev] Source of truth for C-API
In-Reply-To: <CA+3bQGH-bUA3+9M305NvfqfPDDydkX0TE6KZWZUcUwTqmFP-Jw@mail.gmail.com>
References: <CAOMjWkmVTdNyhkotcoBn08z_hS7d-tk393xEvaOrMKPFFreLKw@mail.gmail.com>
 <CA+3bQGEWC5m5X_-DD6vJh=YZBZqL+uwzezc=Q2WVSU6Mz9yJcw@mail.gmail.com>
 <3ba8a094-dcb3-13f3-cdb4-2162f4c5e9ac@python.org>
 <CAOMjWkniKXCeWqrx5oE5+0WVrqgtPkpRdvoWyKpS+xBpCY1tgQ@mail.gmail.com>
 <CA+3bQGGhpLxz-QcOvm-HH5dbq8M2VurHpENyD_-G=P6VnOQVkA@mail.gmail.com>
 <CAOMjWk==qJJuaxBmj8GaNgCY-z_S-Qq3WDitoajUxhgBcg0GhA@mail.gmail.com>
 <CA+3bQGH-bUA3+9M305NvfqfPDDydkX0TE6KZWZUcUwTqmFP-Jw@mail.gmail.com>
Message-ID: <CADiSq7eosNG+72VCeNc45s88MOjjy94c0VUWx+MMQt7wiVSGBg@mail.gmail.com>

On Thu, 24 Jan 2019 at 02:56, Victor Stinner <vstinner at redhat.com> wrote:
>
> I suggest you to add a new function and leaves the existing function
> unchanged. "Just in case".
>
> You may deprecate the old functions at the same time using Py_DEPRECATED().

And potentially put an underscore in front of the new ones (or even
hide them behind Py_BUILD_CORE).

Some of these older APIs are only public because the modern
conventions we use for hiding implementation details weren't
established yet.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jan 27 09:16:57 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 28 Jan 2019 00:16:57 +1000
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
 <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
 <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
Message-ID: <CADiSq7fcfg-M7mePOJFSfhb3kBZbS=4rFDXA5OLxA76dW8fFEQ@mail.gmail.com>

On Thu, 24 Jan 2019 at 05:45, Stephan Reiter <stephan.reiter at gmail.com> wrote:
> If we create a fresh OS thread and make it call PyGILState_Ensure, it
> won't have a PyThreadState saved under autoTLSkey. That means it will
> create one using the main interpreter. I, as the developer embedding
> Python into my application and using multiple interpreters, have no
> control here. Maybe I know that under current conditions a certain
> other interpreter should be used.
>
> I'll try to provoke this situation and then introduce a callback from
> Python into my application that will allow me to specify which
> interpreter should be used, e.g. code as follows:
>
> PyInterpreter *pickAnInterpreter() {
>   return activePlugin ? activePlugin->interpreter : nullptr; //
> nullptr maps to main interpreter
> }
>
> PyGILState_SetNewThreadInterpreterSelectionCallback(&pickAnInterpreter);
>
> Maybe rubbish. But I think a valuable experiment that will give me a
> better understanding.

That actually sounds like a pretty plausible approach to me, at least
for cases where the embedding application maintains some other state
that lets it know which interpreter a new thread should be associated
with. The best aspect of it is that it would let the embedding
application decide how to handle registration of previously unknown
threads with the Python runtime *without* requiring that all existing
extension modules switch to a new thread registration API first.

I'll pass the concept along to Graham Dumpleton (author of the
mod_wsgi module for Apache httpd) to see if an interface like this
might be enough to resolve some of the major compatibility issues
mod_wsgi currently encounters with subinterpreters.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jan 27 09:28:25 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 28 Jan 2019 00:28:25 +1000
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <b450a99a-d4b7-4c49-4e79-3df3707967dc@trueblade.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <b450a99a-d4b7-4c49-4e79-3df3707967dc@trueblade.com>
Message-ID: <CADiSq7d9ad-mvrOmuT2AiaGBRr3VRdnFPaEKvTdZepq6qpeq2A@mail.gmail.com>

On Fri, 25 Jan 2019 at 10:15, Eric V. Smith <eric at trueblade.com> wrote:
> It would be a change if the code is never called. I'm not sure we care
> about code that's never called, but it is a change.

The biggest problem with converting runtime errors to compile time
errors is that it means affected dead code goes from being a
readability & maintainability problem to "Python X.Y breaks my
project".

SyntaxWarning splits the difference nicely, since the compiler
complains about the same dead code that would confuse a human reader,
but also ignores it and moves on.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stephan.reiter at gmail.com  Sun Jan 27 09:32:04 2019
From: stephan.reiter at gmail.com (Stephan Reiter)
Date: Sun, 27 Jan 2019 15:32:04 +0100
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CADiSq7fcfg-M7mePOJFSfhb3kBZbS=4rFDXA5OLxA76dW8fFEQ@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
 <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
 <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
 <CADiSq7fcfg-M7mePOJFSfhb3kBZbS=4rFDXA5OLxA76dW8fFEQ@mail.gmail.com>
Message-ID: <CAFctyt+K1TavZqL4pBrBpaspDnUd2G0udeGzBzQ4SkNc9sG2Dw@mail.gmail.com>

Cool. Thanks, Nick!

I did experiments based on this idea (
https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622)
and haven't rejected it yet. :-)

Together with the other fix (
https://github.com/stephanreiter/cpython/commit/c1afa0c8cdfab862f409f1c7ff02b189f5191cbe),
numpy at least is happy in my Python-hosting app.

I will pursue the idea of swapping sys.modules in a single Interpreter now
because that wouldn't require patching Python and I might get the mileage
out of this approach I need.

Still interested in improving sub-interpreters, though. I just need to
balance short and long term solution. :-)

Stephan


Den s?n. 27. jan. 2019, 15.17 skrev Nick Coghlan <ncoghlan at gmail.com:

> On Thu, 24 Jan 2019 at 05:45, Stephan Reiter <stephan.reiter at gmail.com>
> wrote:
> > If we create a fresh OS thread and make it call PyGILState_Ensure, it
> > won't have a PyThreadState saved under autoTLSkey. That means it will
> > create one using the main interpreter. I, as the developer embedding
> > Python into my application and using multiple interpreters, have no
> > control here. Maybe I know that under current conditions a certain
> > other interpreter should be used.
> >
> > I'll try to provoke this situation and then introduce a callback from
> > Python into my application that will allow me to specify which
> > interpreter should be used, e.g. code as follows:
> >
> > PyInterpreter *pickAnInterpreter() {
> >   return activePlugin ? activePlugin->interpreter : nullptr; //
> > nullptr maps to main interpreter
> > }
> >
> > PyGILState_SetNewThreadInterpreterSelectionCallback(&pickAnInterpreter);
> >
> > Maybe rubbish. But I think a valuable experiment that will give me a
> > better understanding.
>
> That actually sounds like a pretty plausible approach to me, at least
> for cases where the embedding application maintains some other state
> that lets it know which interpreter a new thread should be associated
> with. The best aspect of it is that it would let the embedding
> application decide how to handle registration of previously unknown
> threads with the Python runtime *without* requiring that all existing
> extension modules switch to a new thread registration API first.
>
> I'll pass the concept along to Graham Dumpleton (author of the
> mod_wsgi module for Apache httpd) to see if an interface like this
> might be enough to resolve some of the major compatibility issues
> mod_wsgi currently encounters with subinterpreters.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190127/e4e0bb59/attachment.html>

From tjreedy at udel.edu  Sun Jan 27 14:40:58 2019
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 27 Jan 2019 14:40:58 -0500
Subject: [Python-Dev] Anyone else seeing a lack of caching in local docs
 builds?
In-Reply-To: <CADiSq7fMghm44aPg7W4XW=xhVgdWO=FwLWynEcS77fLeh7d7XA@mail.gmail.com>
References: <CADiSq7fMghm44aPg7W4XW=xhVgdWO=FwLWynEcS77fLeh7d7XA@mail.gmail.com>
Message-ID: <q2l1gb$46ga$1@blaine.gmane.org>

On 1/27/2019 9:01 AM, Nick Coghlan wrote:
> Hi folks,
> 
> I'm currently seeing a behaviour where every time I run "make html",
> all 474 source files get rebuilt.

I just rebuilt, ditto, all 474.  Caching only works when I rebuild 
'soon' (at least within same day) after a complete rebuild.  I just 
updated and only function.rst was changed and only function.html and the 
indexes were rebuilt.

> For now, I'm assuming I've messed something up with my local docs
> build setup, but figured I'd ask if anyone else was seeing this, in
> case it was actually broken at the build level (CI wouldn't pick this
> up, since it always builds from scratch anyway).

I think something is broken.  Caching used to work better.


-- 
Terry Jan Reedy


From njs at pobox.com  Sun Jan 27 15:06:23 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 27 Jan 2019 12:06:23 -0800
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAFctyt+K1TavZqL4pBrBpaspDnUd2G0udeGzBzQ4SkNc9sG2Dw@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
 <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
 <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
 <CADiSq7fcfg-M7mePOJFSfhb3kBZbS=4rFDXA5OLxA76dW8fFEQ@mail.gmail.com>
 <CAFctyt+K1TavZqL4pBrBpaspDnUd2G0udeGzBzQ4SkNc9sG2Dw@mail.gmail.com>
Message-ID: <CAPJVwBmYysxBXWt_d=GZx6popztqYhzPFU5Okm5H9PJzkf8Q7Q@mail.gmail.com>

On Sun, Jan 27, 2019, 06:34 Stephan Reiter <stephan.reiter at gmail.com wrote:

> Cool. Thanks, Nick!
>
> I did experiments based on this idea (
> https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622)
> and haven't rejected it yet. :-)
>
> Together with the other fix (
> https://github.com/stephanreiter/cpython/commit/c1afa0c8cdfab862f409f1c7ff02b189f5191cbe),
> numpy at least is happy in my Python-hosting app.
>

So again, just to make sure you're aware, even if it looks like it's
working right now, there are definitely many subtle ways that numpy will
break when used in a subinterpreter and this configuration is not supported
by the numpy devs. If you discover later that there's some strange crash,
or even that you've been getting incorrect results for months without
noticing, then the numpy devs will be sympathetic but will probably close
your bugs without further investigation.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190127/01487ced/attachment.html>

From greg at krypto.org  Sun Jan 27 15:27:28 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 27 Jan 2019 12:27:28 -0800
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CALGmxEJEhsUjhszSL+fk0Vrz4HfJ9QDjGZUgdi_aAXYCO+LYiw@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
 <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
 <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
 <CALGmxEJEhsUjhszSL+fk0Vrz4HfJ9QDjGZUgdi_aAXYCO+LYiw@mail.gmail.com>
Message-ID: <CAGE7PNLRHpv_ur21LBZgeXFEAi7+FLCdXA0OpayH-uydAUjAsg@mail.gmail.com>

On Thu, Jan 24, 2019 at 1:25 PM Chris Barker - NOAA Federal via Python-Dev <
python-dev at python.org> wrote:

> If your primary concern is module clashes between plugins, maybe you
> can hack around that:
>
> 1) if the plugins are providing copies of any other modules, then you
> can simply require them to put them in their own namespace ? that is,
> a plug-in is a single package, with however many sub modules as it may
> need.
>
> 2) if plugins might require third party packages that need to be
> isolated, then maybe you could use an import hook that
> re-names/isolates the modules each plugin loads, so they are kept
> separate.
>
> I haven?t thought through how to do any of this, but in principle, you
> can have the same module loaded twice if it has a different name.
>

This is dangerous for extension modules.  C is a single global space
unrelated to Python module names that cannot be isolated without
intentionally building and linking each desired extension module statically
and configured not to export its own symbols (no-export-dynamic).  Non
trivial.

Suggesting importing the same extension module multiple times under
different Python sys.modules names is a recipe for disaster.  Most
extension module code is not written with that in mind.  So while *some*
things happen to "work", many others blow up in unexpected hard to debug
ways.

Not that sub interpreters aren?t cool and useful, but you can probably
> handle module clashes in a simpler way.
>

They're a cool and useful theory... but I really do not recommend their use
for code importing other libraries expecting to be isolated.  CPython
doesn't offer multiple isolated runtimes in a process today.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190127/f60b3335/attachment.html>

From ncoghlan at gmail.com  Mon Jan 28 03:02:16 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 28 Jan 2019 18:02:16 +1000
Subject: [Python-Dev] Anyone else seeing a lack of caching in local docs
 builds?
In-Reply-To: <q2l1gb$46ga$1@blaine.gmane.org>
References: <CADiSq7fMghm44aPg7W4XW=xhVgdWO=FwLWynEcS77fLeh7d7XA@mail.gmail.com>
 <q2l1gb$46ga$1@blaine.gmane.org>
Message-ID: <CADiSq7d5iH+mb12YC2stkuYap9zWdMz1SQLA3fGx9sk=76twUg@mail.gmail.com>

On Mon, 28 Jan 2019 at 05:44, Terry Reedy <tjreedy at udel.edu> wrote:
> On 1/27/2019 9:01 AM, Nick Coghlan wrote:
> > For now, I'm assuming I've messed something up with my local docs
> > build setup, but figured I'd ask if anyone else was seeing this, in
> > case it was actually broken at the build level (CI wouldn't pick this
> > up, since it always builds from scratch anyway).
>
> I think something is broken.  Caching used to work better.

In my case, I had an older version of blurb installed, which was
keeping NEWS from rendering properly, and I suspect that was
preventing Sphinx from marking the cache as valid.

I'm not sure about that theory though, as I would have expected an
invalid NEWS file to only keep the NEWS file from being cached.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Jan 28 03:27:45 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 28 Jan 2019 18:27:45 +1000
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAFctyt+K1TavZqL4pBrBpaspDnUd2G0udeGzBzQ4SkNc9sG2Dw@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
 <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
 <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
 <CADiSq7fcfg-M7mePOJFSfhb3kBZbS=4rFDXA5OLxA76dW8fFEQ@mail.gmail.com>
 <CAFctyt+K1TavZqL4pBrBpaspDnUd2G0udeGzBzQ4SkNc9sG2Dw@mail.gmail.com>
Message-ID: <CADiSq7dUKgWWO0XjimjYZ+f+vDNnGueCmp0T=D+Eh4HheLC_eg@mail.gmail.com>

On Mon, 28 Jan 2019 at 00:32, Stephan Reiter <stephan.reiter at gmail.com> wrote:
>
> Cool. Thanks, Nick!
>
> I did experiments based on this idea (https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622) and haven't rejected it yet. :-)

After talking to Graham about this, I unfortunately realised that the
reason the callback approach is appearing to work for you is because
your application is single-threaded, so you can readily map any
invocation of the callback to the desired interpreter. Multi-threaded
applications won't have that luxury - they need to be able to set the
callback target on a per-thread basis.

Graham actually described a plausible approach for doing that several
years back: https://bugs.python.org/issue10915#msg126387

We have much better subinterpreter testing support now, so if this is
any area that you're interested in, one potential place to start would
be to get Antoine's patch back to a point where it applies and
compiles again.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stephan.reiter at gmail.com  Mon Jan 28 04:36:36 2019
From: stephan.reiter at gmail.com (Stephan Reiter)
Date: Mon, 28 Jan 2019 10:36:36 +0100
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CADiSq7dUKgWWO0XjimjYZ+f+vDNnGueCmp0T=D+Eh4HheLC_eg@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
 <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
 <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
 <CADiSq7fcfg-M7mePOJFSfhb3kBZbS=4rFDXA5OLxA76dW8fFEQ@mail.gmail.com>
 <CAFctyt+K1TavZqL4pBrBpaspDnUd2G0udeGzBzQ4SkNc9sG2Dw@mail.gmail.com>
 <CADiSq7dUKgWWO0XjimjYZ+f+vDNnGueCmp0T=D+Eh4HheLC_eg@mail.gmail.com>
Message-ID: <CAFctytL2ufUubm_8mcwhf9xKGN63h9me+4UDkXfkFJRPwcaGeA@mail.gmail.com>

Reading through that post, I think I have everything covered but this here:
- The third and final scenario, and the one where the extended GIL
state functions for Ensure is still required, is where code doesn't
have the GIL as yet and wants to make a call into sub interpreter
rather than the main interpreter, where it already has a pointer to
the sub interpreter and nothing more. In this case the new
PyGILState_EnsureEx() function is used, with the sub interpreter being
passed as argument.

If I understand it correctly, it means the following in practice:
Whenever I or a third-party library start a new thread, we need to
query what interpreter we are running at the moment (in the thread
that is starting the new thread) and pass that information on to the
new thread so that it can initialize the GIL for itself.

Pseudo code ahead:
void do_in_thread(func_t *what) {
  PyThreadState* state = PyThreadState_Get(); /// or new
PyInterpreterState_Current();
  PyInterpreterState *interpreter = state->interp;
  std::thread t([what, interpreter] {
    auto s = PyGILState_EnsureEx(interpreter);
    what();
    PyGILState_Release(s); // could also release before what() because
TLS was updated and next PyGILState_Ensure() will work
  });
}

Did I get that right?

Stephan

Am Mo., 28. Jan. 2019 um 09:27 Uhr schrieb Nick Coghlan <ncoghlan at gmail.com>:
>
> On Mon, 28 Jan 2019 at 00:32, Stephan Reiter <stephan.reiter at gmail.com> wrote:
> >
> > Cool. Thanks, Nick!
> >
> > I did experiments based on this idea (https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622) and haven't rejected it yet. :-)
>
> After talking to Graham about this, I unfortunately realised that the
> reason the callback approach is appearing to work for you is because
> your application is single-threaded, so you can readily map any
> invocation of the callback to the desired interpreter. Multi-threaded
> applications won't have that luxury - they need to be able to set the
> callback target on a per-thread basis.
>
> Graham actually described a plausible approach for doing that several
> years back: https://bugs.python.org/issue10915#msg126387
>
> We have much better subinterpreter testing support now, so if this is
> any area that you're interested in, one potential place to start would
> be to get Antoine's patch back to a point where it applies and
> compiles again.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Mon Jan 28 05:23:27 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 28 Jan 2019 11:23:27 +0100
Subject: [Python-Dev] Anyone else seeing a lack of caching in local docs
 builds?
References: <CADiSq7fMghm44aPg7W4XW=xhVgdWO=FwLWynEcS77fLeh7d7XA@mail.gmail.com>
 <q2l1gb$46ga$1@blaine.gmane.org>
 <CADiSq7d5iH+mb12YC2stkuYap9zWdMz1SQLA3fGx9sk=76twUg@mail.gmail.com>
Message-ID: <20190128112327.16a18a79@fsol>

On Mon, 28 Jan 2019 18:02:16 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Mon, 28 Jan 2019 at 05:44, Terry Reedy <tjreedy at udel.edu> wrote:
> > On 1/27/2019 9:01 AM, Nick Coghlan wrote:  
> > > For now, I'm assuming I've messed something up with my local docs
> > > build setup, but figured I'd ask if anyone else was seeing this, in
> > > case it was actually broken at the build level (CI wouldn't pick this
> > > up, since it always builds from scratch anyway).  
> >
> > I think something is broken.  Caching used to work better.  
> 
> In my case, I had an older version of blurb installed, which was
> keeping NEWS from rendering properly, and I suspect that was
> preventing Sphinx from marking the cache as valid.
> 
> I'm not sure about that theory though, as I would have expected an
> invalid NEWS file to only keep the NEWS file from being cached.

I've often noticed oddities in Sphinx caching on other projects.
I'm not sure the algorithm is extremely precise.

Regards

Antoine.



From ncoghlan at gmail.com  Mon Jan 28 06:42:37 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 28 Jan 2019 21:42:37 +1000
Subject: [Python-Dev] Sub-interpreters: importing numpy causes hang
In-Reply-To: <CAFctytL2ufUubm_8mcwhf9xKGN63h9me+4UDkXfkFJRPwcaGeA@mail.gmail.com>
References: <CAFctytKDsnVyOY9_23DspQRFkFjQCH9VambJ=-y2f+S0B=AwEg@mail.gmail.com>
 <CAPJVwBnag_wzUwZ38RiRFrkOsEOy1Uy21SCi=TuayM-Ty7usPg@mail.gmail.com>
 <CAFctytJSZsyOogbVDE4OataHSm5r5OndSZS8+Fu0t=O28_3THg@mail.gmail.com>
 <db9bf9d4-6b67-e1b5-17f5-db218eb9a9ec@gmail.com>
 <CAFctyt+fLRjSaBNUyjfi2wbB3FYrLwi49RGebwE_z1Jw+hFhgQ@mail.gmail.com>
 <CALFfu7D=4yQA_fT+BYZKZZXPCoYHQ69-pV_kuqLYTx378PbfEg@mail.gmail.com>
 <CAFctytJNOFTSw1Stevu78vHYLNGO2qc5_t8huymuZzKuXd3m2A@mail.gmail.com>
 <CADiSq7fcfg-M7mePOJFSfhb3kBZbS=4rFDXA5OLxA76dW8fFEQ@mail.gmail.com>
 <CAFctyt+K1TavZqL4pBrBpaspDnUd2G0udeGzBzQ4SkNc9sG2Dw@mail.gmail.com>
 <CADiSq7dUKgWWO0XjimjYZ+f+vDNnGueCmp0T=D+Eh4HheLC_eg@mail.gmail.com>
 <CAFctytL2ufUubm_8mcwhf9xKGN63h9me+4UDkXfkFJRPwcaGeA@mail.gmail.com>
Message-ID: <CADiSq7fRYNVLhsCvuVMopaTyY11CL5oFqh1j7PTTJyF3tYw2-A@mail.gmail.com>

On Mon, 28 Jan 2019 at 19:36, Stephan Reiter <stephan.reiter at gmail.com> wrote:
>
> Reading through that post, I think I have everything covered but this here:
> - The third and final scenario, and the one where the extended GIL
> state functions for Ensure is still required, is where code doesn't
> have the GIL as yet and wants to make a call into sub interpreter
> rather than the main interpreter, where it already has a pointer to
> the sub interpreter and nothing more. In this case the new
> PyGILState_EnsureEx() function is used, with the sub interpreter being
> passed as argument.
>
> If I understand it correctly, it means the following in practice:
> Whenever I or a third-party library start a new thread, we need to
> query what interpreter we are running at the moment (in the thread
> that is starting the new thread) and pass that information on to the
> new thread so that it can initialize the GIL for itself.
>
> Pseudo code ahead:
> void do_in_thread(func_t *what) {
>   PyThreadState* state = PyThreadState_Get(); /// or new
> PyInterpreterState_Current();
>   PyInterpreterState *interpreter = state->interp;
>   std::thread t([what, interpreter] {
>     auto s = PyGILState_EnsureEx(interpreter);
>     what();
>     PyGILState_Release(s); // could also release before what() because
> TLS was updated and next PyGILState_Ensure() will work
>   });
> }
>
> Did I get that right?

Yeah, I think that's the essence of it, although the other case that
can come up is when the parent thread just created a new
subinterpreter (that only changes how it acquires the pointer though -
the challenge of getting a child thread to make proper use of that
pointer remains the same).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris.barker at noaa.gov  Mon Jan 28 11:26:26 2019
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Mon, 28 Jan 2019 08:26:26 -0800
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>

> But as a rule,
>> there are a LOT of errors that can be pretty mysterious to newbies.
>
> Isn't that the very definition of "newbie"?  That's half a joke, but I
> really don't think that programmers new to Python should be the
> standard.

Python is broadly advocated (and used) as a first language to learn.
Because it is.

So I think anything we can do to help newbies, *that doesn?t make the
language less powerful, or even more annoying for experienced
developers* is a good thing.

That is: prioritizing newbie-friendliness is good. Prioritizing it
over other important things is not.

-CHB



> The problematic cases are those where even a relatively
> experienced Python programmer needs to be told why an error is raised,
> because it's too hard to figure out from background knowledge of the
> language, you need to know about implementation internals.
>
>> I would love to see Python become generally more informative with
>> errors.
>
> I would love to see all automated systems become more informative with
> errors!  One thing I like about Python the development community is
> that people are doing something about it.  Slowly but surely ....
>
>> I don?t think so ? what we need are helpful error messages. If it
>> will be raised at compile time, then it won?t generally be
>> catchable in s try-except? so the actual exception type isn?t very
>> important.
>
> +1
>
> In general depending much on exception type is a hit-or-miss affair.

From turnbull.stephen.fw at u.tsukuba.ac.jp  Mon Jan 28 23:44:17 2019
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Tue, 29 Jan 2019 13:44:17 +0900
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
 <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
Message-ID: <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>

Chris Barker - NOAA Federal writes:
 > > > But as a rule,
 > > > there are a LOT of errors that can be pretty mysterious to newbies.
 > >
 > > Isn't that the very definition of "newbie"?  That's half a joke, but I
 > > really don't think that programmers new to Python should be the
 > > standard.
 > 
 > Python is broadly advocated (and used) as a first language to learn.
 > Because it is.
 > 
 > So I think anything we can do to help newbies, *that doesn?t make the
 > language less powerful, or even more annoying for experienced
 > developers* is a good thing.
 > 
 > That is: prioritizing newbie-friendliness is good. Prioritizing it
 > over other important things is not.

I don't disagree.  I disagree with the conclusion that it's worth the
effort to try to improve all error messages that confuse new users,
because new users (by definition) don't know enough to respond
usefully in many cases.  In those cases, they need to be told what's
going on and why, where more experienced users can figure it out from
their background knowledge of Python semantics.  Embedding a "theory
of operations" note in every error message would be possible, but I
don't think it's a good idea -- it would certainly make the language
more annoying for experienced developers.

That's why I proposed the criterion

 > > The problematic cases are those where even a relatively
 > > experienced Python programmer needs to be told why an error is
 > > raised, because it's too hard to figure out from background
 > > knowledge of the language, you need to know about implementation
 > > internals.

Maybe that's too strict.  The pleasant hacks of giving ``quit`` and
``help`` "values" as variables are certainly useful.  On the contrary,
getting rid of the print statement was perhaps a better idea. :-)

Steve

From ncoghlan at gmail.com  Tue Jan 29 08:44:57 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 29 Jan 2019 23:44:57 +1000
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
 <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
 <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7eD-nmZPh1-WrSSO8v=imhzxr9kX4-AY7iSSVVmoYUxXQ@mail.gmail.com>

On Tue, 29 Jan 2019 at 14:47, Stephen J. Turnbull
<turnbull.stephen.fw at u.tsukuba.ac.jp> wrote:
> I don't disagree.  I disagree with the conclusion that it's worth the
> effort to try to improve all error messages that confuse new users,
> because new users (by definition) don't know enough to respond
> usefully in many cases.  In those cases, they need to be told what's
> going on and why, where more experienced users can figure it out from
> their background knowledge of Python semantics.  Embedding a "theory
> of operations" note in every error message would be possible, but I
> don't think it's a good idea -- it would certainly make the language
> more annoying for experienced developers.

FWIW, we have pretty decent evidence that error messages don't have to
provide a wonderful explanation on their own in order to be helpful:
they just need to be distinctive enough that a web search will
reliably get you to a page that gives you relevant information.

Pre-seeded answers on Stack Overflow are excellent for handling the
second half of that approach (see [1] for a specific example).

Cheers,
Nick.

[1] https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-parentheses-in-call-to-print-mean-in-python

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From python at mrabarnett.plus.com  Tue Jan 29 13:55:37 2019
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 29 Jan 2019 18:55:37 +0000
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <CADiSq7eD-nmZPh1-WrSSO8v=imhzxr9kX4-AY7iSSVVmoYUxXQ@mail.gmail.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
 <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
 <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7eD-nmZPh1-WrSSO8v=imhzxr9kX4-AY7iSSVVmoYUxXQ@mail.gmail.com>
Message-ID: <aa2e4d34-6192-6af2-d62a-997dbcc5e645@mrabarnett.plus.com>

On 2019-01-29 13:44, Nick Coghlan wrote:
> On Tue, 29 Jan 2019 at 14:47, Stephen J. Turnbull
> <turnbull.stephen.fw at u.tsukuba.ac.jp> wrote:
>> I don't disagree.  I disagree with the conclusion that it's worth the
>> effort to try to improve all error messages that confuse new users,
>> because new users (by definition) don't know enough to respond
>> usefully in many cases.  In those cases, they need to be told what's
>> going on and why, where more experienced users can figure it out from
>> their background knowledge of Python semantics.  Embedding a "theory
>> of operations" note in every error message would be possible, but I
>> don't think it's a good idea -- it would certainly make the language
>> more annoying for experienced developers.
> 
> FWIW, we have pretty decent evidence that error messages don't have to
> provide a wonderful explanation on their own in order to be helpful:
> they just need to be distinctive enough that a web search will
> reliably get you to a page that gives you relevant information.
> 
> Pre-seeded answers on Stack Overflow are excellent for handling the
> second half of that approach (see [1] for a specific example).
> 
> Cheers,
> Nick.
> 
> [1] https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-parentheses-in-call-to-print-mean-in-python
> 
I have a vague recollection that a certain computer system (Amiga?) had 
a 'why' command. If it reported an error, you could type "why" and it 
would give you more details.

I suspect that all that was happening was that when the error occurred 
it would store the additional details somewhere that the 'why' command 
would simply retrieve.

From pmiscml at gmail.com  Tue Jan 29 14:51:42 2019
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Tue, 29 Jan 2019 22:51:42 +0300
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <aa2e4d34-6192-6af2-d62a-997dbcc5e645@mrabarnett.plus.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
 <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
 <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7eD-nmZPh1-WrSSO8v=imhzxr9kX4-AY7iSSVVmoYUxXQ@mail.gmail.com>
 <aa2e4d34-6192-6af2-d62a-997dbcc5e645@mrabarnett.plus.com>
Message-ID: <20190129225142.62636b7b@x230>

Hello,

On Tue, 29 Jan 2019 18:55:37 +0000
MRAB <python at mrabarnett.plus.com> wrote:

[]

> > https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-parentheses-in-call-to-print-mean-in-python 
> I have a vague recollection that a certain computer system (Amiga?)
> had a 'why' command. If it reported an error, you could type "why"
> and it would give you more details.
> 
> I suspect that all that was happening was that when the error
> occurred it would store the additional details somewhere that the
> 'why' command would simply retrieve.

Well, a typical Linux install includes "errno" util:

$ errno 13
EACCES 13 Permission denied

So, hail to error numbers. And indeed, even if some may think they're
artifact of 1980'ies, they're alive, well, and useful:

$ pylint --help-msg W1651
:deprecated-itertools-function (W1651): *Accessing a deprecated function on the itertools module*
  Used when accessing a function on itertools that has been removed in Python 3.
  This message belongs to the python3 checker. It can't be emitted when using
  Python >= 3.0.


-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From pmiscml at gmail.com  Wed Jan 30 12:54:06 2019
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Wed, 30 Jan 2019 20:54:06 +0300
Subject: [Python-Dev] [ANN] "compiler" package resurrected
Message-ID: <20190130205406.10253578@x230>

Hello,

I'm sorry for posting here and not to python-announce, somehow I think
(perhaps naively) that it may be of interest to people who are
interested in Python development. At the very least, creation of the
original package is (very likely, I didn't trace up to that) was
discussed on python-dev, its removal was discussed on python-dev, so
why revival of it can't be noted here?

Because, turns out, in old good times, there was a bytecode compiler
written in Python, and even as a part of stdlib. Well, actually it's
still there in the latest 2.7 release, with a proud notice: "Remove
in Python3". The point is that I'm with Python since 1.5 times, and
never knew about this package. I'd generally consider that to be
something to be ashamed and hush of, but unfortunately I found that to
be recurring pattern: people interested in playing with a Python
compiler discover "by a chance" and "suddenly" that they should look no
beyond the stdlib for their (usually pretty simple for starters) needs
- oftentimes, after they already started on the more effortful path
(I faithfully spent 2 days on trying to extract a bytecode compiler
from PyPy first). 

With that intro, here's it - the port of Python2 compiler package
(https://docs.python.org/2/library/compiler.html) to Python3:

https://github.com/pfalcon/python-compiler

Currently, it generates bytecode compatible with CPython3.5, and is
able to compile its entire Lib/ (which includes not just stdlib modules
per se, but also tests, and the real "teeth-cracking" stuff is there of
course), except for test_super.py, for which current implementation's
teeth are indeed too weak yet.  

Now that it passes the compile-stdlib test, the idea is to refactor
it into something which can be easily played with and extended. We'll
see how it goes.

As one of the example of what's intended to be easily done with it, see
thread
https://mail.python.org/pipermail/python-dev/2019-January/155991.html .
I started it when updating the codegen for Python3, but also shows the
intended purpose - it would easy to analyze if an except handler body
contains "del exc" and if not, skip generating superfluous bytecode.  

-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From stefan_ml at behnel.de  Wed Jan 30 13:12:21 2019
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 30 Jan 2019 19:12:21 +0100
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <aa2e4d34-6192-6af2-d62a-997dbcc5e645@mrabarnett.plus.com>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
 <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
 <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7eD-nmZPh1-WrSSO8v=imhzxr9kX4-AY7iSSVVmoYUxXQ@mail.gmail.com>
 <aa2e4d34-6192-6af2-d62a-997dbcc5e645@mrabarnett.plus.com>
Message-ID: <q2spe6$5ki7$1@blaine.gmane.org>

MRAB schrieb am 29.01.19 um 19:55:
> On 2019-01-29 13:44, Nick Coghlan wrote:
>> FWIW, we have pretty decent evidence that error messages don't have to
>> provide a wonderful explanation on their own in order to be helpful:
>> they just need to be distinctive enough that a web search will
>> reliably get you to a page that gives you relevant information.
>>
>> Pre-seeded answers on Stack Overflow are excellent for handling the
>> second half of that approach (see [1] for a specific example).
>> [1]
>> https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-parentheses-in-call-to-print-mean-in-python
> 
> I have a vague recollection that a certain computer system (Amiga?) had a
> 'why' command. If it reported an error, you could type "why" and it would
> give you more details.
> 
> I suspect that all that was happening was that when the error occurred it
> would store the additional details somewhere that the 'why' command would
> simply retrieve.

So ? are you suggesting to use the webbrowser module inside of the REPL to
look up the exception message of the previously printed stack trace in
stack overflow when a user types "why()"?

I faintly recall someone implementing something in that direction. It's
probably in some package on PyPI.

Stefan


From python at mrabarnett.plus.com  Wed Jan 30 14:33:05 2019
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 30 Jan 2019 19:33:05 +0000
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <q2spe6$5ki7$1@blaine.gmane.org>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
 <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
 <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7eD-nmZPh1-WrSSO8v=imhzxr9kX4-AY7iSSVVmoYUxXQ@mail.gmail.com>
 <aa2e4d34-6192-6af2-d62a-997dbcc5e645@mrabarnett.plus.com>
 <q2spe6$5ki7$1@blaine.gmane.org>
Message-ID: <6414be1f-4103-9d91-330f-d35e4a54e009@mrabarnett.plus.com>

On 2019-01-30 18:12, Stefan Behnel wrote:
> MRAB schrieb am 29.01.19 um 19:55:
>> On 2019-01-29 13:44, Nick Coghlan wrote:
>>> FWIW, we have pretty decent evidence that error messages don't have to
>>> provide a wonderful explanation on their own in order to be helpful:
>>> they just need to be distinctive enough that a web search will
>>> reliably get you to a page that gives you relevant information.
>>>
>>> Pre-seeded answers on Stack Overflow are excellent for handling the
>>> second half of that approach (see [1] for a specific example).
>>> [1]
>>> https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-parentheses-in-call-to-print-mean-in-python
>> 
>> I have a vague recollection that a certain computer system (Amiga?) had a
>> 'why' command. If it reported an error, you could type "why" and it would
>> give you more details.
>> 
>> I suspect that all that was happening was that when the error occurred it
>> would store the additional details somewhere that the 'why' command would
>> simply retrieve.
> 
> So ? are you suggesting to use the webbrowser module inside of the REPL to
> look up the exception message of the previously printed stack trace in
> stack overflow when a user types "why()"?
> 
No, I was just suggesting it as a possible way of providing newbies with 
more information about an error without annoying more experienced users 
with excessively long messages every time.

> I faintly recall someone implementing something in that direction. It's
> probably in some package on PyPI.
> 

From phd at phdru.name  Wed Jan 30 16:14:28 2019
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 30 Jan 2019 22:14:28 +0100
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <q2spe6$5ki7$1@blaine.gmane.org>
References: <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
 <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
 <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7eD-nmZPh1-WrSSO8v=imhzxr9kX4-AY7iSSVVmoYUxXQ@mail.gmail.com>
 <aa2e4d34-6192-6af2-d62a-997dbcc5e645@mrabarnett.plus.com>
 <q2spe6$5ki7$1@blaine.gmane.org>
Message-ID: <20190130211428.noqlsbx5uztpxbu7@phdru.name>

On Wed, Jan 30, 2019 at 07:12:21PM +0100, Stefan Behnel <stefan_ml at behnel.de> wrote:
> > I have a vague recollection that a certain computer system (Amiga?) had a
> > 'why' command. If it reported an error, you could type "why" and it would
> > give you more details.
> > 
> > I suspect that all that was happening was that when the error occurred it
> > would store the additional details somewhere that the 'why' command would
> > simply retrieve.
> 
> So ??? are you suggesting to use the webbrowser module inside of the REPL to
> look up the exception message of the previously printed stack trace in
> stack overflow when a user types "why()"?
> 
> I faintly recall someone implementing something in that direction. It's
> probably in some package on PyPI.

   It doesn't have to be all web. Compare how ``help()`` works at the
python REPL, ``pydoc name`` at the command line, and
``pydoc -p`` + a browser.

> Stefan

Oleg.
-- 
    Oleg Broytman            https://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From greg.ewing at canterbury.ac.nz  Wed Jan 30 16:46:10 2019
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 31 Jan 2019 10:46:10 +1300
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <q2spe6$5ki7$1@blaine.gmane.org>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
 <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
 <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7eD-nmZPh1-WrSSO8v=imhzxr9kX4-AY7iSSVVmoYUxXQ@mail.gmail.com>
 <aa2e4d34-6192-6af2-d62a-997dbcc5e645@mrabarnett.plus.com>
 <q2spe6$5ki7$1@blaine.gmane.org>
Message-ID: <5C521B22.3010602@canterbury.ac.nz>

Stefan Behnel wrote:
> So ? are you suggesting to use the webbrowser module inside of the REPL to
> look up the exception message of the previously printed stack trace in
> stack overflow when a user types "why()"?

"Python is searching for an answer to your question..."

-- 
Greg

From raymond.hettinger at gmail.com  Wed Jan 30 17:32:38 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 30 Jan 2019 14:32:38 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict instead
 of OrderedDict
Message-ID: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>

Now that regular dicts are ordered and compact, it makes more sense for the _asdict() method to create a regular dict (as it did in its early days) rather than an OrderedDict.  The regular dict is much smaller, much faster, and has a much cleaner looking repr.  It would also help namedtuple() stay in sync with dataclasses which already take advantage of the ordering feature of regular dicts.

The question is how to be go about making the change in a way gives the most benefit to users as soon as possible and that creates the least disruption.

Option 1) Add a deprecation notice to 3.8, make no code change in 3.8, and then update the code in 3.9.  This has several issues: a) it doesn't provide an executable DeprecationWarning in 3.8, b) it isn't really a deprecation, and c) it defers the benefits of the change for another release.

Option 2) Add a deprecation notice to 3.8, add a DeprecationWarning to the _asdict() method, and make the actual improvement in 3.9.  The main issue here is that it will create a lot of noise for normal uses of the _asdict() method which are otherwise unaffected by the change. The typical use cases for _asdict() are to create keyword arguments and to pass named tuple data into functions or methods that expect regular dictionaries.  Those use cases would benefit from seeing the change made sooner and would suffer in the interim from their code slowing down for warnings that aren't useful.

Option 3). Add a deprecation notice to 3.8 and have the _asdict() method create a subclass of OrderedDict that issues warnings only for the methods and attributes that will change (move_to_end, popitem, __eq__, __dict__, __weakref__).  This is less noisy but it adds a lot of machinery just to make a notification of a minor change.  Also, it fails to warn that the data type will change.  And it may create more confusion than options 1 and 4 which are simpler.

Option 4) Just make the change directly in 3.8,  s/OrderedDict/dict/, and be done will it.  This gives users the benefits right away and doesn't annoy them with warnings that they likely don't care about.   There is some precedent for this.  To make namedtuple class creation faster, the *verbose* option was dropped without any deprecation period.  It looks like no one missed that feature at all, but they did get the immediate benefit of faster import times.  In the case of using regular dicts in named tuples, people will get immediate and significant space savings as well as a speed benefit.

My recommendation is Option 4 as being less disruptive and more beneficial than the other options.  In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple:   nt._asdict() -> OrderedDict(nt.as_dict()).

What do you all think?


Raymond




From p.f.moore at gmail.com  Wed Jan 30 17:55:24 2019
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 30 Jan 2019 22:55:24 +0000
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
Message-ID: <CACac1F_PxspBt7cdEV=K1ytmFdf8pom5a7V3rsq29pGCx6fApg@mail.gmail.com>

On Wed, 30 Jan 2019 at 22:35, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> My recommendation is Option 4 as being less disruptive and more beneficial than the other options.  In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple:   nt._asdict() -> OrderedDict(nt.as_dict()).
>
> What do you all think?

+1 from me on option 4.

Paul

From Matthew.Lagoe at gmail.com  Wed Jan 30 17:53:16 2019
From: Matthew.Lagoe at gmail.com (Matthew Lagoe)
Date: Wed, 30 Jan 2019 15:53:16 -0700
Subject: [Python-Dev] Summer of Code 2019 Mentors
Message-ID: <CAE_HWMtAwJLsYLv5rkrJtqFi_rFekNq7+X+hd36ihDAtDecsRQ@mail.gmail.com>

Hi Python community folk!

As we've done for the past many years, Python is hoping to participate
in Google Summer of Code.  This is a neat program where students write
code over the (northern hemisphere) summer under the tutelage of open
source mentors and get paid: we provide the project ideas, mentors and
choose the students, Google provides the program framework and the money
to pay students.  You can read more about GSoC here:
https://summerofcode.withgoogle.com/

Python participates as an "umbrella org" where many different smaller
projects ("sub orgs") that use Python can take part under our banner.
You can also participate separately, but for people who've never done it
before and want help or for whom the paperwork is a hassle, you're
welcome to join up with us and let us show you the ropes!

It's really fun, and we've gotten lots of new contributors to
Python-based projects over the years, taking in as many as 70+ students
in a single year.  Last year we only had 15, though, so we've got lots
of space for new mentors and new projects.

We didn't have any projects for core python last year as there were no
mentors
for the projects, so if anyone is interested in mentoring this year let us
know asap!

You can also send questions to gsoc-admins at python.org (or just hit reply
to this email!)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190130/5eeaa33e/attachment.html>

From ethan at stoneleaf.us  Wed Jan 30 18:37:56 2019
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 30 Jan 2019 15:37:56 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <CACac1F_PxspBt7cdEV=K1ytmFdf8pom5a7V3rsq29pGCx6fApg@mail.gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
 <CACac1F_PxspBt7cdEV=K1ytmFdf8pom5a7V3rsq29pGCx6fApg@mail.gmail.com>
Message-ID: <bcce4267-8f82-429d-c3d2-b530d4d8a5b2@stoneleaf.us>

On 01/30/2019 02:55 PM, Paul Moore wrote:
> On Wed, 30 Jan 2019 at 22:35, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>> My recommendation is Option 4 as being less disruptive and more beneficial than the other options.  In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple:   nt._asdict() -> OrderedDict(nt.as_dict()).
>>
>> What do you all think?
> 
> +1 from me on option 4.

 From me as well.

--
~Ethan~

From v+python at g.nevcal.com  Wed Jan 30 18:41:15 2019
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 30 Jan 2019 15:41:15 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
Message-ID: <c87985b3-26d6-bee3-6c26-21ad835363c1@g.nevcal.com>

On 1/30/2019 2:32 PM, Raymond Hettinger wrote:
> Now that regular dicts are ordered and compact, it makes more sense for the _asdict() method to create a regular dict (as it did in its early days) rather than an OrderedDict.
...
> Option 4) Just make the change directly in 3.8,  s/OrderedDict/dict/, and be done will it.  This gives users the benefits right away and doesn't annoy them with warnings that they likely don't care about.   There is some precedent for this.  To make namedtuple class creation faster, the *verbose* option was dropped without any deprecation period.  It looks like no one missed that feature at all, but they did get the immediate benefit of faster import times.  In the case of using regular dicts in named tuples, people will get immediate and significant space savings as well as a speed benefit.
>
> My recommendation is Option 4 as being less disruptive and more beneficial than the other options.  In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple:   nt._asdict() -> OrderedDict(nt.as_dict()).
>
> What do you all think?
Option 4 sounds good to me.

Would it be practical to add deprecated methods to regular dict for the 
OrderedDict reordering methods that raise with an error suggesting "To 
use this method, convert dict to OrderedDict." (or some better wording).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190130/711f6a52/attachment-0001.html>

From greg at krypto.org  Wed Jan 30 19:50:04 2019
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 30 Jan 2019 16:50:04 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
Message-ID: <CAGE7PN+cbBg6U50XySLgjWk7_nqse-bfgLBu__s-nho1gbqQMA@mail.gmail.com>

On Wed, Jan 30, 2019 at 2:32 PM Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

> Now that regular dicts are ordered and compact, it makes more sense for
> the _asdict() method to create a regular dict (as it did in its early days)
> rather than an OrderedDict.  The regular dict is much smaller, much faster,
> and has a much cleaner looking repr.  It would also help namedtuple() stay
> in sync with dataclasses which already take advantage of the ordering
> feature of regular dicts.
>
> The question is how to be go about making the change in a way gives the
> most benefit to users as soon as possible and that creates the least
> disruption.
>
> Option 1) Add a deprecation notice to 3.8, make no code change in 3.8, and
> then update the code in 3.9.  This has several issues: a) it doesn't
> provide an executable DeprecationWarning in 3.8, b) it isn't really a
> deprecation, and c) it defers the benefits of the change for another
> release.
>
> Option 2) Add a deprecation notice to 3.8, add a DeprecationWarning to the
> _asdict() method, and make the actual improvement in 3.9.  The main issue
> here is that it will create a lot of noise for normal uses of the _asdict()
> method which are otherwise unaffected by the change. The typical use cases
> for _asdict() are to create keyword arguments and to pass named tuple data
> into functions or methods that expect regular dictionaries.  Those use
> cases would benefit from seeing the change made sooner and would suffer in
> the interim from their code slowing down for warnings that aren't useful.
>
> Option 3). Add a deprecation notice to 3.8 and have the _asdict() method
> create a subclass of OrderedDict that issues warnings only for the methods
> and attributes that will change (move_to_end, popitem, __eq__, __dict__,
> __weakref__).  This is less noisy but it adds a lot of machinery just to
> make a notification of a minor change.  Also, it fails to warn that the
> data type will change.  And it may create more confusion than options 1 and
> 4 which are simpler.
>
> Option 4) Just make the change directly in 3.8,  s/OrderedDict/dict/, and
> be done will it.  This gives users the benefits right away and doesn't
> annoy them with warnings that they likely don't care about.   There is some
> precedent for this.  To make namedtuple class creation faster, the
> *verbose* option was dropped without any deprecation period.  It looks like
> no one missed that feature at all, but they did get the immediate benefit
> of faster import times.  In the case of using regular dicts in named
> tuples, people will get immediate and significant space savings as well as
> a speed benefit.
>
> My recommendation is Option 4 as being less disruptive and more beneficial
> than the other options.  In the unlikely event that anyone is currently
> depending on the reordering methods for the output of _asdict(), the
> remediation is trivially simple:   nt._asdict() ->
> OrderedDict(nt.as_dict()).
>

+1 on option 4.  I agree with everyone else.  Because the remediation that
keeps code compatible across all versions is that simple, just go with
option 4.  We document that in What's New and be done with it. :)

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190130/5bdde236/attachment.html>

From barry at python.org  Wed Jan 30 20:07:19 2019
From: barry at python.org (Barry Warsaw)
Date: Wed, 30 Jan 2019 17:07:19 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
Message-ID: <86A7BD3D-0661-450A-964E-790E72BE5A46@python.org>

On Jan 30, 2019, at 14:32, Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> 
> Now that regular dicts are ordered and compact, it makes more sense for the _asdict() method to create a regular dict (as it did in its early days) rather than an OrderedDict.  The regular dict is much smaller, much faster, and has a much cleaner looking repr.  It would also help namedtuple() stay in sync with dataclasses which already take advantage of the ordering feature of regular dicts.

Thanks for the excellent write up, Raymond.

> Option 1) Add a deprecation notice to 3.8 [?]

In general, I don?t favor deprecation notices for things that are not actionable by the end user or the consumer of the API in question.  It ends up being just noise, and that?s one of the big reasons why they are silenced by default (i.e. the end user of a Python application can?t do anything about the deprecation, so why scare them?).

Maybe we need something like a MinorBehavioralChangeWarning which would be surfaced similar to PendingDeprecationWarning, but which says ?Hey, we changed this in the current version of Python.  Be aware of that change, but you probably don?t need to do anything about it?.  At least such warnings could be reviewed, documented, and audited.

I suppose in this case, there is something the consumer of the API can do (as you point out, they can wrap their code), but in all likelihood there?s really nothing they *need* to do.  I think it will be rare that a Python 3.8 user will get bitten by this.  Therefore...

> Option 4) Just make the change directly in 3.8,

JFDI!

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190130/71b0fd04/attachment.sig>

From mertz at gnosis.cx  Wed Jan 30 21:00:31 2019
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 30 Jan 2019 21:00:31 -0500
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <CACac1F_PxspBt7cdEV=K1ytmFdf8pom5a7V3rsq29pGCx6fApg@mail.gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
 <CACac1F_PxspBt7cdEV=K1ytmFdf8pom5a7V3rsq29pGCx6fApg@mail.gmail.com>
Message-ID: <CAEbHw4ZOQCP_YfRQs8OFVf0OJM+A2XKeA0xpFbReegc-U+f1Uw@mail.gmail.com>

Ditto +1 option 4

On Wed, Jan 30, 2019, 5:56 PM Paul Moore <p.f.moore at gmail.com wrote:

> On Wed, 30 Jan 2019 at 22:35, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
> > My recommendation is Option 4 as being less disruptive and more
> beneficial than the other options.  In the unlikely event that anyone is
> currently depending on the reordering methods for the output of _asdict(),
> the remediation is trivially simple:   nt._asdict() ->
> OrderedDict(nt.as_dict()).
> >
> > What do you all think?
>
> +1 from me on option 4.
>
> Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190130/fa55c95a/attachment.html>

From njs at pobox.com  Wed Jan 30 21:10:17 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 30 Jan 2019 18:10:17 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
Message-ID: <CAPJVwBkGKEWvEGt0UAtk5NZzuCpK60MLu4+FGTAGzEC5L12wrw@mail.gmail.com>

On Wed, Jan 30, 2019 at 2:34 PM Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> Now that regular dicts are ordered and compact, it makes more sense for the _asdict() method to create a regular dict (as it did in its early days) rather than an OrderedDict.  The regular dict is much smaller, much faster, and has a much cleaner looking repr.

How viable would it be to make OrderedDict smaller, faster, and give
it a cleaner looking repr?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From raymond.hettinger at gmail.com  Wed Jan 30 23:45:46 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 30 Jan 2019 20:45:46 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <CAPJVwBkGKEWvEGt0UAtk5NZzuCpK60MLu4+FGTAGzEC5L12wrw@mail.gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
 <CAPJVwBkGKEWvEGt0UAtk5NZzuCpK60MLu4+FGTAGzEC5L12wrw@mail.gmail.com>
Message-ID: <E05ACFC9-890B-44F9-9175-1F6636F264FC@gmail.com>


> On Jan 30, 2019, at 6:00 PM, David Mertz <mertz at gnosis.cx> wrote:
> 
> Ditto +1 option 4
> 
> On Wed, Jan 30, 2019, 5:56 PM Paul Moore <p.f.moore at gmail.com wrote:
> On Wed, 30 Jan 2019 at 22:35, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
> > My recommendation is Option 4 as being less disruptive and more beneficial than the other options.  In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple:   nt._asdict() -> OrderedDict(nt.as_dict()).
> >
> > What do you all think?
>> 
>> +1 from me on option 4.
>> 
>> Paul


Thanks everyone.  I'll move forward with option 4.  In Barry's word, JFDI :-)

> On Jan 30, 2019, at 6:10 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> How viable would it be to make OrderedDict smaller, faster, and give
> it a cleaner looking repr?

Not so much.  The implementations substantially different because they have different superpowers.  A regular dict is really good at being a dict while retaining order but it isn't good at reordering operations such as popitem(False), popitem(True), move_to_end(), and whatnot.  An OrderedDict is a heavier weight structure (a hash table augmented by a doubly-linked link) -- it is worse at being a dictionary but really good at intensive reordering operations typical in cache recency tracking and whatnot.  Also, there are long-standing API differences including weak references, ability to assign attributes, an equality operation that requires exact order when compared to another ordered dict etc, as well as the reordering methods.  If it was easy, clean, and desirable, it would have already been done :-)  Overall, I think the OrderedDict is increasingly irrelevant except for use cases requiring cross-version compatibility and for cases that need heavy reordering.  Accordingly, I mostly expect to leave it alone and fall into the not-much-used category like UserDict, UserList, and UserString.

> On Jan 30, 2019, at 3:41 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:


> Would it be practical to add deprecated methods to regular dict for the OrderedDict reordering methods that raise with an error suggesting "To use this method, convert dict to OrderedDict." (or some better wording).

That's an interesting idea.  Regular dicts aren't well suited to the reordering operations (like lists, repeated inserts at the front of the sequence wouldn't be performant relative to OrderedDict which uses double-linked lists internally).  My instinct is to leave regular dicts alone so that they can focus on their primary task (being good a fast lookups).


Raymond



From timothy.c.delaney at gmail.com  Thu Jan 31 00:11:28 2019
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Thu, 31 Jan 2019 16:11:28 +1100
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <E05ACFC9-890B-44F9-9175-1F6636F264FC@gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
 <CAPJVwBkGKEWvEGt0UAtk5NZzuCpK60MLu4+FGTAGzEC5L12wrw@mail.gmail.com>
 <E05ACFC9-890B-44F9-9175-1F6636F264FC@gmail.com>
Message-ID: <CAN8CLgnOucpyWGLh2boFA8h0+07LO+CMjSoqPxfhzMF+kM0sew@mail.gmail.com>

On Thu, 31 Jan 2019 at 15:46, Raymond Hettinger <raymond.hettinger at gmail.com>
wrote:

>
> > Would it be practical to add deprecated methods to regular dict for the
> OrderedDict reordering methods that raise with an error suggesting "To use
> this method, convert dict to OrderedDict." (or some better wording).
>
> That's an interesting idea.  Regular dicts aren't well suited to the
> reordering operations (like lists, repeated inserts at the front of the
> sequence wouldn't be performant relative to OrderedDict which uses
> double-linked lists internally).  My instinct is to leave regular dicts
> alone so that they can focus on their primary task (being good a fast
> lookups).
>

Alternatively, would it be viable to make OrderedDict work in a way that so
long as you don't use any reordering operations it's essentially just a
very thin layer on top of a dict, but if you do use any reordering
operations, it adds in the additional heavyweight structure required to
support that?

I'm pretty sure something similar has been considered before, but thought I
should bring it up in the context of this discussion (if only to have to
shot down).

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190131/638e4270/attachment.html>

From raymond.hettinger at gmail.com  Thu Jan 31 00:24:41 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 30 Jan 2019 21:24:41 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <CAN8CLgnOucpyWGLh2boFA8h0+07LO+CMjSoqPxfhzMF+kM0sew@mail.gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
 <CAPJVwBkGKEWvEGt0UAtk5NZzuCpK60MLu4+FGTAGzEC5L12wrw@mail.gmail.com>
 <E05ACFC9-890B-44F9-9175-1F6636F264FC@gmail.com>
 <CAN8CLgnOucpyWGLh2boFA8h0+07LO+CMjSoqPxfhzMF+kM0sew@mail.gmail.com>
Message-ID: <A2D36DCA-93AF-4A32-9146-7C12E79A46EF@gmail.com>



> On Jan 30, 2019, at 9:11 PM, Tim Delaney <timothy.c.delaney at gmail.com> wrote:
> 
> Alternatively, would it be viable to make OrderedDict work in a way that so long as you don't use any reordering operations it's essentially just a very thin layer on top of a dict,

There's all kinds of tricks we could do but none of them are worth it.  It took Eric Snow a long time to write the OrderedDict patch and it took years to get most of the bugs out of it.  I would really hate to go through a redesign and eat up our time for something that probably won't be much used any more.

I'm really just aiming for something as simple as s/OrderedDict/dict in namedtuple :-)  


Raymond

From v+python at g.nevcal.com  Thu Jan 31 00:18:55 2019
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 30 Jan 2019 21:18:55 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <E05ACFC9-890B-44F9-9175-1F6636F264FC@gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
 <CAPJVwBkGKEWvEGt0UAtk5NZzuCpK60MLu4+FGTAGzEC5L12wrw@mail.gmail.com>
 <E05ACFC9-890B-44F9-9175-1F6636F264FC@gmail.com>
Message-ID: <e50a4c03-9bff-f944-c9b2-2e11deb5e07b@g.nevcal.com>

On 1/30/2019 8:45 PM, Raymond Hettinger wrote:
>> On Jan 30, 2019, at 3:41 PM, Glenn Linderman<v+python at g.nevcal.com>  wrote:
>> Would it be practical to add deprecated methods to regular dict for the OrderedDict reordering methods that raise with an error suggesting "To use this method, convert dict to OrderedDict." (or some better wording).
> That's an interesting idea.  Regular dicts aren't well suited to the reordering operations (like lists, repeated inserts at the front of the sequence wouldn't be performant relative to OrderedDict which uses double-linked lists internally).  My instinct is to leave regular dicts alone so that they can focus on their primary task (being good a fast lookups).

My goal was just to give a meaningful error message if someone misses 
the implications in What's New, and has code that actually does expect 
named_tuple.as_dict to have the ordering operations.

The added, deprecated methods could be removed after a couple of versions.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190130/d4fc9f26/attachment-0001.html>

From songofacandy at gmail.com  Thu Jan 31 02:52:19 2019
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 31 Jan 2019 16:52:19 +0900
Subject: [Python-Dev] How about updating OrderedDict in csv and configparser
 to regular dict?
Message-ID: <CAEfz+TyQBiP-1HM41tz0EiT_eVtrY9jOuiTWdHnjUKnbQQfgiA@mail.gmail.com>

Hi,

csv.DictReader uses OrderedDict by default, from Python 3.6.
But it doesn't make sense anymore, like namedtuple._asdict().
How about changing default dict type back to regular dict.

Python is widely used for handling learge data.  So I think
changing default dict type to OrderedDict was performance
and memory usage regression in 3.6.

Additionally, configparser uses OrderedDict by default from Python 3.6 too.

I am not sure about `parser['section1'] == parser['section2']` is not used yet.
But we broke it once in 3.6 by changing dict to OrderedDict.  Are there any
issue report caused by this backward incompatibility?

And I think performance and memory efficiency is not so important for
configparser, unlike csv.

I'm

* +1 about changing csv.DictReader's default dict type
* +0.5 about changing configparser's default dict type.

How do you think?

Regards,
--
INADA Naoki  <songofacandy at gmail.com>

From songofacandy at gmail.com  Thu Jan 31 03:00:49 2019
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 31 Jan 2019 17:00:49 +0900
Subject: [Python-Dev] How about updating OrderedDict in csv and
 configparser to regular dict?
In-Reply-To: <CAEfz+TyQBiP-1HM41tz0EiT_eVtrY9jOuiTWdHnjUKnbQQfgiA@mail.gmail.com>
References: <CAEfz+TyQBiP-1HM41tz0EiT_eVtrY9jOuiTWdHnjUKnbQQfgiA@mail.gmail.com>
Message-ID: <CAEfz+TxtK9fJGPz+_k7mvsUHXTJYS55bQVT_3J1qAJtPOAUiTQ@mail.gmail.com>

I'm sorry, configparser is changed already.
https://bugs.python.org/issue33504

On Thu, Jan 31, 2019 at 4:52 PM INADA Naoki <songofacandy at gmail.com> wrote:
>
> Hi,
>
> csv.DictReader uses OrderedDict by default, from Python 3.6.
> But it doesn't make sense anymore, like namedtuple._asdict().
> How about changing default dict type back to regular dict.
>
> Python is widely used for handling learge data.  So I think
> changing default dict type to OrderedDict was performance
> and memory usage regression in 3.6.
>
> Additionally, configparser uses OrderedDict by default from Python 3.6 too.
>
> I am not sure about `parser['section1'] == parser['section2']` is not used yet.
> But we broke it once in 3.6 by changing dict to OrderedDict.  Are there any
> issue report caused by this backward incompatibility?
>
> And I think performance and memory efficiency is not so important for
> configparser, unlike csv.
>
> I'm
>
> * +1 about changing csv.DictReader's default dict type
> * +0.5 about changing configparser's default dict type.
>
> How do you think?
>
> Regards,
> --
> INADA Naoki  <songofacandy at gmail.com>



-- 
INADA Naoki  <songofacandy at gmail.com>

From mike at selik.org  Thu Jan 31 03:04:36 2019
From: mike at selik.org (Michael Selik)
Date: Thu, 31 Jan 2019 00:04:36 -0800
Subject: [Python-Dev] How about updating OrderedDict in csv and
 configparser to regular dict?
In-Reply-To: <CAEfz+TxtK9fJGPz+_k7mvsUHXTJYS55bQVT_3J1qAJtPOAUiTQ@mail.gmail.com>
References: <CAEfz+TyQBiP-1HM41tz0EiT_eVtrY9jOuiTWdHnjUKnbQQfgiA@mail.gmail.com>
 <CAEfz+TxtK9fJGPz+_k7mvsUHXTJYS55bQVT_3J1qAJtPOAUiTQ@mail.gmail.com>
Message-ID: <CAGgTfkPSDpe8yKWGVkfffM3r2RvKTbs-mZKNRt5-8GSTCBaXEg@mail.gmail.com>

Waiting on review
https://github.com/python/cpython/pull/8014

On Thu, Jan 31, 2019, 12:04 AM INADA Naoki <songofacandy at gmail.com wrote:

> I'm sorry, configparser is changed already.
> https://bugs.python.org/issue33504
>
> On Thu, Jan 31, 2019 at 4:52 PM INADA Naoki <songofacandy at gmail.com>
> wrote:
> >
> > Hi,
> >
> > csv.DictReader uses OrderedDict by default, from Python 3.6.
> > But it doesn't make sense anymore, like namedtuple._asdict().
> > How about changing default dict type back to regular dict.
> >
> > Python is widely used for handling learge data.  So I think
> > changing default dict type to OrderedDict was performance
> > and memory usage regression in 3.6.
> >
> > Additionally, configparser uses OrderedDict by default from Python 3.6
> too.
> >
> > I am not sure about `parser['section1'] == parser['section2']` is not
> used yet.
> > But we broke it once in 3.6 by changing dict to OrderedDict.  Are there
> any
> > issue report caused by this backward incompatibility?
> >
> > And I think performance and memory efficiency is not so important for
> > configparser, unlike csv.
> >
> > I'm
> >
> > * +1 about changing csv.DictReader's default dict type
> > * +0.5 about changing configparser's default dict type.
> >
> > How do you think?
> >
> > Regards,
> > --
> > INADA Naoki  <songofacandy at gmail.com>
>
>
>
> --
> INADA Naoki  <songofacandy at gmail.com>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190131/34c4a41e/attachment.html>

From solipsis at pitrou.net  Thu Jan 31 05:11:16 2019
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 31 Jan 2019 11:11:16 +0100
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
Message-ID: <20190131111116.424ed48f@fsol>

On Wed, 30 Jan 2019 14:32:38 -0800
Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> 
> My recommendation is Option 4 as being less disruptive and more beneficial than the other options.

Option 4 here as well.  Sometimes perfection is the enemy of the good.

Regards

Antoine.



From steve at holdenweb.com  Thu Jan 31 06:04:17 2019
From: steve at holdenweb.com (Steve Holden)
Date: Thu, 31 Jan 2019 11:04:17 +0000
Subject: [Python-Dev] How about updating OrderedDict in csv and
 configparser to regular dict?
In-Reply-To: <CAEfz+TyQBiP-1HM41tz0EiT_eVtrY9jOuiTWdHnjUKnbQQfgiA@mail.gmail.com>
References: <CAEfz+TyQBiP-1HM41tz0EiT_eVtrY9jOuiTWdHnjUKnbQQfgiA@mail.gmail.com>
Message-ID: <CAMofdRC0rQX+q3baoRGFJJHW_9sWOEV1PJqiAjhBw89KwrveCA@mail.gmail.com>

I submitted the patch to make the csv module use an OrderedDict in
DictReader. It doesn't, AFAIR, use anything but the ordering property of
ordered dict, so the reversion to a simple dict should be perfectly OK as
long as no consumers have started to make use of specific OrderedDict
properties.

regards
Steve Holden


On Thu, Jan 31, 2019 at 7:55 AM INADA Naoki <songofacandy at gmail.com> wrote:

> Hi,
>
> csv.DictReader uses OrderedDict by default, from Python 3.6.
> But it doesn't make sense anymore, like namedtuple._asdict().
> How about changing default dict type back to regular dict.
>
> Python is widely used for handling learge data.  So I think
> changing default dict type to OrderedDict was performance
> and memory usage regression in 3.6.
>
> Additionally, configparser uses OrderedDict by default from Python 3.6 too.
>
> I am not sure about `parser['section1'] == parser['section2']` is not used
> yet.
> But we broke it once in 3.6 by changing dict to OrderedDict.  Are there any
> issue report caused by this backward incompatibility?
>
> And I think performance and memory efficiency is not so important for
> configparser, unlike csv.
>
> I'm
>
> * +1 about changing csv.DictReader's default dict type
> * +0.5 about changing configparser's default dict type.
>
> How do you think?
>
> Regards,
> --
> INADA Naoki  <songofacandy at gmail.com>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190131/f311d9df/attachment.html>

From steve at holdenweb.com  Thu Jan 31 06:06:13 2019
From: steve at holdenweb.com (Steve Holden)
Date: Thu, 31 Jan 2019 11:06:13 +0000
Subject: [Python-Dev] Fwd: How about updating OrderedDict in csv and
 configparser to regular dict?
In-Reply-To: <CAMofdRA04QBfzoBO8SAci7Z7EWw+egFRx+=X8k+isHOpC_dYhQ@mail.gmail.com>
References: <CAEfz+TyQBiP-1HM41tz0EiT_eVtrY9jOuiTWdHnjUKnbQQfgiA@mail.gmail.com>
 <CAMofdRC0rQX+q3baoRGFJJHW_9sWOEV1PJqiAjhBw89KwrveCA@mail.gmail.com>
 <CAMofdRA04QBfzoBO8SAci7Z7EWw+egFRx+=X8k+isHOpC_dYhQ@mail.gmail.com>
Message-ID: <CAMofdRAQ2xxxWTw4BSST8XtEoW0O=VMHjzdMbuvu_h3xzV_wBg@mail.gmail.com>

---------- Forwarded message ---------
From: Steve Holden <steve at holdenweb.com>
Date: Thu, Jan 31, 2019 at 11:05 AM
Subject: Re: [Python-Dev] How about updating OrderedDict in csv and
configparser to regular dict?
To: INADA Naoki <songofacandy at gmail.com>


And I see that such a patch is now merged. Thanks,  Raymond!

>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190131/6a112930/attachment.html>

From ncoghlan at gmail.com  Thu Jan 31 08:16:04 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 31 Jan 2019 23:16:04 +1000
Subject: [Python-Dev] Add more SyntaxWarnings?
In-Reply-To: <q2spe6$5ki7$1@blaine.gmane.org>
References: <q2d6i0$63gm$1@blaine.gmane.org>
 <20190124220126.luvclamlqxl2paxh@python.ca>
 <20190124224035.GB26605@ando.pearwood.info>
 <CAPTjJmqPKWmPpAd8eJO3pF2kSgi3NuYJ6Z4kB8CywYX2Fm33pw@mail.gmail.com>
 <CAP7+vJL+zq=Bp_oGszfMRFwXL8HGzugbnTO=dFBRuFKBVwdvbg@mail.gmail.com>
 <CALGmxE+0iSLhrVZ_yJFZDy4WO8aBK2XL8xUPSdMkv+jodGqYKQ@mail.gmail.com>
 <23626.35650.790785.749597@turnbull.sk.tsukuba.ac.jp>
 <CALGmxELU84YF+T_ifE2VSXMFQDUj3aaiVCskk3GzxKRH3TQFxg@mail.gmail.com>
 <23631.55841.75814.366891@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7eD-nmZPh1-WrSSO8v=imhzxr9kX4-AY7iSSVVmoYUxXQ@mail.gmail.com>
 <aa2e4d34-6192-6af2-d62a-997dbcc5e645@mrabarnett.plus.com>
 <q2spe6$5ki7$1@blaine.gmane.org>
Message-ID: <CADiSq7d5jFF_Zwqs0SxW04W3jXZ9TX6+josgHMD_UTnnVkXoVA@mail.gmail.com>

On Thu, 31 Jan 2019 at 04:16, Stefan Behnel <stefan_ml at behnel.de> wrote:
> I faintly recall someone implementing something in that direction. It's
> probably in some package on PyPI.

You may be thinking of Doug Hellman's https://pypi.org/project/whatthewhat/

Cheers,
Nick.

P.S. More horrifyingly (in an amusing way), there's also
https://github.com/drathier/stack-overflow-import

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Jan 31 08:24:05 2019
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 31 Jan 2019 23:24:05 +1000
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <e50a4c03-9bff-f944-c9b2-2e11deb5e07b@g.nevcal.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
 <CAPJVwBkGKEWvEGt0UAtk5NZzuCpK60MLu4+FGTAGzEC5L12wrw@mail.gmail.com>
 <E05ACFC9-890B-44F9-9175-1F6636F264FC@gmail.com>
 <e50a4c03-9bff-f944-c9b2-2e11deb5e07b@g.nevcal.com>
Message-ID: <CADiSq7cW_5X9Z9iPMNfj7pq=5Ca0aMd9gniVr9jhYon455o9Xg@mail.gmail.com>

On Thu, 31 Jan 2019 at 16:40, Glenn Linderman <v+python at g.nevcal.com> wrote:
>> On 1/30/2019 8:45 PM, Raymond Hettinger wrote:
>>> On Jan 30, 2019, at 3:41 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
>>> Would it be practical to add deprecated methods to regular dict for the OrderedDict reordering methods that raise with an error suggesting "To use this method, convert dict to OrderedDict." (or some better wording).
>> That's an interesting idea.  Regular dicts aren't well suited to the reordering operations (like lists, repeated inserts at the front of the sequence wouldn't be performant relative to OrderedDict which uses double-linked lists internally).  My instinct is to leave regular dicts alone so that they can focus on their primary task (being good a fast lookups).
> My goal was just to give a meaningful error message if someone misses the implications in What's New, and has code that actually does expect named_tuple.as_dict to have the ordering operations.

The downside of doing that is that automated code introspection tools
don't know that the methods don't really exist, they just see the
method names in the type dictionary and offer them up for code
completion.

So in this case, the extra runtime check isn't worth the cost of
breaking static code analysis and other forms of introspection.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From raymond.hettinger at gmail.com  Thu Jan 31 12:42:54 2019
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Thu, 31 Jan 2019 09:42:54 -0800
Subject: [Python-Dev] Fwd: How about updating OrderedDict in csv and
 configparser to regular dict?
In-Reply-To: <CAMofdRAQ2xxxWTw4BSST8XtEoW0O=VMHjzdMbuvu_h3xzV_wBg@mail.gmail.com>
References: <CAEfz+TyQBiP-1HM41tz0EiT_eVtrY9jOuiTWdHnjUKnbQQfgiA@mail.gmail.com>
 <CAMofdRC0rQX+q3baoRGFJJHW_9sWOEV1PJqiAjhBw89KwrveCA@mail.gmail.com>
 <CAMofdRA04QBfzoBO8SAci7Z7EWw+egFRx+=X8k+isHOpC_dYhQ@mail.gmail.com>
 <CAMofdRAQ2xxxWTw4BSST8XtEoW0O=VMHjzdMbuvu_h3xzV_wBg@mail.gmail.com>
Message-ID: <D371BA76-B8FA-423A-AD8D-944558795B22@gmail.com>



> On Jan 31, 2019, at 3:06 AM, Steve Holden <steve at holdenweb.com> wrote:
> 
> And I see that such a patch is now merged. Thanks,  Raymond!

And thank you for getting ordering into csv.DictReader.  That was a significant improvement in usability :-)


Raymond

From njs at pobox.com  Thu Jan 31 16:25:53 2019
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 31 Jan 2019 13:25:53 -0800
Subject: [Python-Dev] How to update namedtuple asdict() to use dict
 instead of OrderedDict
In-Reply-To: <CADiSq7cW_5X9Z9iPMNfj7pq=5Ca0aMd9gniVr9jhYon455o9Xg@mail.gmail.com>
References: <DEEE3FD0-450E-4AD8-BA8E-87FDF6E0E8EB@gmail.com>
 <CAPJVwBkGKEWvEGt0UAtk5NZzuCpK60MLu4+FGTAGzEC5L12wrw@mail.gmail.com>
 <E05ACFC9-890B-44F9-9175-1F6636F264FC@gmail.com>
 <e50a4c03-9bff-f944-c9b2-2e11deb5e07b@g.nevcal.com>
 <CADiSq7cW_5X9Z9iPMNfj7pq=5Ca0aMd9gniVr9jhYon455o9Xg@mail.gmail.com>
Message-ID: <CAPJVwBnGZovkc_67=-JSd=JVq=iONZ2w0jcYC=OAKEJWLv_y=g@mail.gmail.com>

On Thu, Jan 31, 2019, 05:26 Nick Coghlan <ncoghlan at gmail.com wrote:

> On Thu, 31 Jan 2019 at 16:40, Glenn Linderman <v+python at g.nevcal.com>
> wrote:
> >> On 1/30/2019 8:45 PM, Raymond Hettinger wrote:
> >>> On Jan 30, 2019, at 3:41 PM, Glenn Linderman <v+python at g.nevcal.com>
> wrote:
> >>> Would it be practical to add deprecated methods to regular dict for
> the OrderedDict reordering methods that raise with an error suggesting "To
> use this method, convert dict to OrderedDict." (or some better wording).
> >> That's an interesting idea.  Regular dicts aren't well suited to the
> reordering operations (like lists, repeated inserts at the front of the
> sequence wouldn't be performant relative to OrderedDict which uses
> double-linked lists internally).  My instinct is to leave regular dicts
> alone so that they can focus on their primary task (being good a fast
> lookups).
> > My goal was just to give a meaningful error message if someone misses
> the implications in What's New, and has code that actually does expect
> named_tuple.as_dict to have the ordering operations.
>
> The downside of doing that is that automated code introspection tools
> don't know that the methods don't really exist, they just see the
> method names in the type dictionary and offer them up for code
> completion.
>
> So in this case, the extra runtime check isn't worth the cost of
> breaking static code analysis and other forms of introspection.
>

It's technically possible for attributes to do something custom when
accessed, without appearing in __dir__. I don't know if that's a useful
technique in this case, but sometimes it is.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190131/b8f1641f/attachment.html>