From burkhardameier at gmail.com Sun May 1 04:47:16 2016 From: burkhardameier at gmail.com (Burkhard Meier) Date: Sun, 1 May 2016 01:47:16 -0700 Subject: [Python-Dev] Memory Message-ID: ~ Die Scheibe klirrt der Spontie kichert ~ hoffentlich Alliiance versichert... ~ https://youtu.be/m3SjCzA71eM -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Sun May 1 09:08:22 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sun, 1 May 2016 08:08:22 -0500 Subject: [Python-Dev] Memory In-Reply-To: References: Message-ID: Well, I put this in Google Translate...and got this: The disk clatters the Spontie giggles ~ hopefully alliance insures ... Not sure if this a useless post or Translate just being weird. Leaning towards the latter... -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On May 1, 2016 3:48 AM, "Burkhard Meier" wrote: > ~ > Die Scheibe klirrt > der Spontie kichert > ~ > hoffentlich > Alliiance versichert... > ~ > https://youtu.be/m3SjCzA71eM > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun May 1 12:23:55 2016 From: brett at python.org (Brett Cannon) Date: Sun, 01 May 2016 16:23:55 +0000 Subject: [Python-Dev] Bug in 2to3 concerning import statements? In-Reply-To: <57255A70.3000404@canterbury.ac.nz> References: <57255A70.3000404@canterbury.ac.nz> Message-ID: On Sat, 30 Apr 2016 at 21:51 Greg Ewing wrote: > It seems that 2to3 is a bit simplistic when it comes to > translating import statements. I have a module GUI.py2exe > containing: > > import py2exe.mf as modulefinder > > 2to3 translates this into: > > from . import py2exe.mf as modulefinder > > which is a syntax error. > > It looks like 2to3 is getting confused by the fact that > there is both a submodule and a top-level module here > called py2exe. That sounds right. 2to3 has to inspect the environment to figure out if the import was meant to be relative or absolute and it's getting confused. > But the original can only be an absolute > import because it has a dot in it, so 2to3 shouldn't be > translating it into a relative one. > 2to3 is nowhere near that sophisticated for tweaking imports :) > > Putting "from __future__ import absolute_import" at the > top fixes it, but I shouldn't have to do that, should I? > There's all sorts of weird stuff going on in that import, like having a dot in the `from` part of the import instead of doing `from .py2exe import mf as modulefinder`. Try that tweak and see if that solves your SyntaxError for some reason. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun May 1 12:51:38 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 May 2016 02:51:38 +1000 Subject: [Python-Dev] Memory In-Reply-To: References: Message-ID: <20160501165136.GD13497@ando.pearwood.info> On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote: > Well, I put this in Google Translate...and got this: > > The disk clatters > the Spontie giggles > ~ > hopefully > alliance insures ... > > Not sure if this a useless post or Translate just being weird. Leaning > towards the latter... You missed the link to David Bowie's "Heroes" on YouTube. I'm pretty sure that this is a useless post. Burkhard, please speak English on this list, as best as you are able. And please avoid rambling cryptic posts and topics which have nothing to do with the development of CPython. If you want to share random thoughts, you should get a blog or a Twitter account, don't post them here. Thank you. -- Steve From cmkleffner at gmail.com Sun May 1 13:41:07 2016 From: cmkleffner at gmail.com (Carl Kleffner) Date: Sun, 1 May 2016 19:41:07 +0200 Subject: [Python-Dev] Memory In-Reply-To: <20160501165136.GD13497@ando.pearwood.info> References: <20160501165136.GD13497@ando.pearwood.info> Message-ID: AFAIK Burkhard A. Meier is the author of the Python GUI Programming Cookbook (Packtpub). I wonder if someone has hacked is gmail account? 2016-05-01 18:51 GMT+02:00 Steven D'Aprano : > On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote: > > Well, I put this in Google Translate...and got this: > > > > The disk clatters > > the Spontie giggles > > ~ > > hopefully > > alliance insures ... > > > > Not sure if this a useless post or Translate just being weird. Leaning > > towards the latter... > > You missed the link to David Bowie's "Heroes" on YouTube. > > I'm pretty sure that this is a useless post. > > Burkhard, please speak English on this list, as best as you are able. > And please avoid rambling cryptic posts and topics which have nothing to > do with the development of CPython. If you want to share random > thoughts, you should get a blog or a Twitter account, don't post them > here. > > > Thank you. > > > > -- > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun May 1 13:48:32 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 1 May 2016 10:48:32 -0700 Subject: [Python-Dev] Memory In-Reply-To: References: <20160501165136.GD13497@ando.pearwood.info> Message-ID: That would certainly be the most favorable interpretation. On Sun, May 1, 2016 at 10:41 AM, Carl Kleffner wrote: > AFAIK Burkhard A. Meier is the author of the Python GUI Programming > Cookbook (Packtpub). I wonder if someone has hacked is gmail account? > > 2016-05-01 18:51 GMT+02:00 Steven D'Aprano : > >> On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote: >> > Well, I put this in Google Translate...and got this: >> > >> > The disk clatters >> > the Spontie giggles >> > ~ >> > hopefully >> > alliance insures ... >> > >> > Not sure if this a useless post or Translate just being weird. Leaning >> > towards the latter... >> >> You missed the link to David Bowie's "Heroes" on YouTube. >> >> I'm pretty sure that this is a useless post. >> >> Burkhard, please speak English on this list, as best as you are able. >> And please avoid rambling cryptic posts and topics which have nothing to >> do with the development of CPython. If you want to share random >> thoughts, you should get a blog or a Twitter account, don't post them >> here. >> >> >> Thank you. >> >> >> >> -- >> Steve >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From burkhardameier at gmail.com Sun May 1 15:19:50 2016 From: burkhardameier at gmail.com (Burkhard Meier) Date: Sun, 1 May 2016 12:19:50 -0700 Subject: [Python-Dev] Memory In-Reply-To: References: <20160501165136.GD13497@ando.pearwood.info> Message-ID: Sorry. Somebody take the keyboard away from me after midnight. I think Google Chrome does have that feature. Definitely Not related to Python dev... Google translate doesn't correctly understand everything. My apologies (again ) for this unrelated post. And, yes, I am the author of the "Python GUI programming cookbook " Sorry, Burkhard On Sunday, May 1, 2016, Guido van Rossum wrote: > That would certainly be the most favorable interpretation. > > On Sun, May 1, 2016 at 10:41 AM, Carl Kleffner > wrote: > >> AFAIK Burkhard A. Meier is the author of the Python GUI Programming >> Cookbook (Packtpub). I wonder if someone has hacked is gmail account? >> >> 2016-05-01 18:51 GMT+02:00 Steven D'Aprano > >: >> >>> On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote: >>> > Well, I put this in Google Translate...and got this: >>> > >>> > The disk clatters >>> > the Spontie giggles >>> > ~ >>> > hopefully >>> > alliance insures ... >>> > >>> > Not sure if this a useless post or Translate just being weird. Leaning >>> > towards the latter... >>> >>> You missed the link to David Bowie's "Heroes" on YouTube. >>> >>> I'm pretty sure that this is a useless post. >>> >>> Burkhard, please speak English on this list, as best as you are able. >>> And please avoid rambling cryptic posts and topics which have nothing to >>> do with the development of CPython. If you want to share random >>> thoughts, you should get a blog or a Twitter account, don't post them >>> here. >>> >>> >>> Thank you. >>> >>> >>> >>> -- >>> Steve >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com >>> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From burkhardameier at gmail.com Sun May 1 17:11:09 2016 From: burkhardameier at gmail.com (Burkhard Meier) Date: Sun, 1 May 2016 14:11:09 -0700 Subject: [Python-Dev] Looking for a Python programming mentor Message-ID: I don't need Sigmund Freud... Burkhard -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun May 1 17:16:40 2016 From: brett at python.org (Brett Cannon) Date: Sun, 01 May 2016 21:16:40 +0000 Subject: [Python-Dev] Looking for a Python programming mentor In-Reply-To: References: Message-ID: If you mean for Python in general, then Python+list or python-tutor are a more appropriate mailing lists. If you mean for core development then core-workflow is where to ask for help. On Sun, May 1, 2016, 14:11 Burkhard Meier wrote: > I don't need Sigmund Freud... > > Burkhard > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun May 1 19:42:35 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 02 May 2016 11:42:35 +1200 Subject: [Python-Dev] Bug in 2to3 concerning import statements? In-Reply-To: References: <57255A70.3000404@canterbury.ac.nz> Message-ID: <5726946B.9050806@canterbury.ac.nz> Brett Cannon wrote: > There's all sorts of weird stuff going on in that import, like having a > dot in the `from` part of the import instead of doing `from .py2exe > import mf as modulefinder`. If I did that, it would try to import mf from the py2exe submodule rather than the global one. In hindsight it was probably a bad idea to name the submodule py2exe, but the code worked as intended. My point is that, regardless of anything else, 2to3 shouldn't be generating syntactically incorrect code. I can understand if it's not considered worth fixing, but it's still a bug. -- Greg From guido at python.org Sun May 1 20:41:19 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 1 May 2016 17:41:19 -0700 Subject: [Python-Dev] Bug in 2to3 concerning import statements? In-Reply-To: <5726946B.9050806@canterbury.ac.nz> References: <57255A70.3000404@canterbury.ac.nz> <5726946B.9050806@canterbury.ac.nz> Message-ID: I agree it's a bug. May I encourage you to file it as such on bugs.python.org? 2to3 is part of the stdlib. On Sun, May 1, 2016 at 4:42 PM, Greg Ewing wrote: > Brett Cannon wrote: > > There's all sorts of weird stuff going on in that import, like having a >> dot in the `from` part of the import instead of doing `from .py2exe import >> mf as modulefinder`. >> > > If I did that, it would try to import mf from the > py2exe submodule rather than the global one. > > In hindsight it was probably a bad idea to name > the submodule py2exe, but the code worked as > intended. > > My point is that, regardless of anything else, > 2to3 shouldn't be generating syntactically > incorrect code. I can understand if it's not > considered worth fixing, but it's still a bug. > > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue May 3 12:07:21 2016 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 03 May 2016 09:07:21 -0700 Subject: [Python-Dev] 2.7.12 schedule Message-ID: <1462291641.3972729.596903881.4CD8087B@webmail.messagingengine.com> I have been reminded that we are due for a 2.7.12 release. I think Larry's 3.5.2 schedule is perfectly cromulent and am going to piggyback. Therefore, 2.7.12rc1 will be on June 11th, and the final will be on June 25. Servus, Benjamin From stefan at bytereef.org Tue May 3 15:06:12 2016 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 3 May 2016 19:06:12 +0000 (UTC) Subject: [Python-Dev] Yearly PyPI breakage Message-ID: Hello, Could someone enlighten me which hoops I have to jump through this year in order to keep pip downloads working? Collecting cdecimal Could not find a version that satisfies the requirement cdecimal (from versions: ) No matching distribution found for cdecimal You are using pip version 7.1.2, however version 8.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. If this continues, I'm going to release a premium version that's 50% faster and only available from bytereef.org or Anaconda. Stefan Krah From lukasz at langa.pl Tue May 3 15:13:52 2016 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Tue, 3 May 2016 12:13:52 -0700 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: <3339680B-83AB-4CFB-B6B2-95AC6D055AA3@langa.pl> Why don?t you just *host* the files on PyPI? > On May 3, 2016, at 12:06 PM, Stefan Krah wrote: > > > Hello, > > Could someone enlighten me which hoops I have to jump through > this year in order to keep pip downloads working? > > Collecting cdecimal > Could not find a version that satisfies the requirement cdecimal (from > versions: ) > No matching distribution found for cdecimal > You are using pip version 7.1.2, however version 8.1.1 is available. > You should consider upgrading via the 'pip install --upgrade pip' command. > > > If this continues, I'm going to release a premium version that's > 50% faster and only available from bytereef.org or Anaconda. > > > > Stefan Krah > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/lukasz%40langa.pl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Tue May 3 15:23:41 2016 From: brett at python.org (Brett Cannon) Date: Tue, 03 May 2016 19:23:41 +0000 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: On Tue, 3 May 2016 at 12:06 Stefan Krah wrote: > > Hello, > > Could someone enlighten me which hoops I have to jump through > this year in order to keep pip downloads working? > > Collecting cdecimal > Could not find a version that satisfies the requirement cdecimal (from > versions: ) > No matching distribution found for cdecimal > You are using pip version 7.1.2, however version 8.1.1 is available. > You should consider upgrading via the 'pip install --upgrade pip' command. > The distutils-sig mailing is the best place to try and get help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue May 3 15:28:11 2016 From: donald at stufft.io (Donald Stufft) Date: Tue, 3 May 2016 15:28:11 -0400 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: > On May 3, 2016, at 3:06 PM, Stefan Krah wrote: > > > Hello, > > Could someone enlighten me which hoops I have to jump through > this year in order to keep pip downloads working? > > This is off topic for python-dev, but https://www.python.org/dev/peps/pep-0470/#i-can-t-host-my-project-on-pypi-because-of-x-what-should-i-do . ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From njs at pobox.com Tue May 3 16:09:45 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 3 May 2016 13:09:45 -0700 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: On Tue, May 3, 2016 at 12:06 PM, Stefan Krah wrote: > > Hello, > > Could someone enlighten me which hoops I have to jump through > this year in order to keep pip downloads working? > > Collecting cdecimal > Could not find a version that satisfies the requirement cdecimal (from > versions: ) > No matching distribution found for cdecimal > You are using pip version 7.1.2, however version 8.1.1 is available. > You should consider upgrading via the 'pip install --upgrade pip' command. > > > If this continues, I'm going to release a premium version that's > 50% faster and only available from bytereef.org or Anaconda. There's no point in making threats -- you're threatening the air. PyPI is maintained by one overloaded developer (Donald Stufft, sponsored by HPE as part of their openstack work) with help from a few overloaded, burned-out volunteers. Everyone wants PyPI to be awesome and useful; your frustration is totally understandable, and lots of us share it. But the limitation here is that we have ~one person running a website that serves ~100,000,000 requests/day, and running as fast as they can just to keep things from breaking more than they already have. As long as that's the case it doesn't matter what "should" happen, there's no-one to do it. -n -- Nathaniel J. Smith -- https://vorpus.org From stefan at bytereef.org Tue May 3 16:42:46 2016 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 3 May 2016 20:42:46 +0000 (UTC) Subject: [Python-Dev] Yearly PyPI breakage References: Message-ID: Nathaniel Smith pobox.com> writes: > > If this continues, I'm going to release a premium version that's > > 50% faster and only available from bytereef.org or Anaconda. > > There's no point in making threats -- you're threatening the air. PyPI > is maintained by one overloaded developer (Donald Stufft, sponsored by > HPE as part of their openstack work) with help from a few overloaded, > burned-out volunteers. Everyone wants PyPI to be awesome and useful; > your frustration is totally understandable, and lots of us share it. > But the limitation here is that we have ~one person running a website > that serves ~100,000,000 requests/day, and running as fast as they can > just to keep things from breaking more than they already have. As long > as that's the case it doesn't matter what "should" happen, there's > no-one to do it. This wasn't so much of a threat, it was resignation. At some point it seems like the rational thing to do. I don't fully understand your explanation. Judging by the link that Donald posted (thanks!) it seems that PEP 470 introduced extra work for him that would not have been present had things been left in place. Also, I did not get any notification and now that I searched for PEP 470 it seems that it wasn't announced here: https://mail.python.org/pipermail/python-dev/2015-October/141838.html But if the majority prefers PyPI that way, I'll stop arguing. Stefan Krah From lukasz at langa.pl Tue May 3 17:21:28 2016 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Tue, 3 May 2016 14:21:28 -0700 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> > On May 3, 2016, at 1:42 PM, Stefan Krah wrote: > > I don't fully understand your explanation. Judging by the link that > Donald posted (thanks!) it seems that PEP 470 introduced extra work > for him that would not have been present had things been left in place. IIRC the PyPI maintainers were constantly nagged about ?PyPI reliability issues? that were instead external hosting issues. Everybody was affected every now and then whenever tummy.com or other external servers for popular packages were down. Or at least I know *I was*. Way too often. > But if the majority prefers PyPI that way, I'll stop arguing. I?m not sure what you mean here but if you want to argue for reverting PEP 470, I wouldn?t hold my breath. -- ? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan at bytereef.org Tue May 3 17:38:13 2016 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 3 May 2016 21:38:13 +0000 (UTC) Subject: [Python-Dev] Yearly PyPI breakage References: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> Message-ID: ?ukasz Langa langa.pl> writes: > > I don't fully understand your explanation. Judging by the link that > > Donald posted (thanks!) it seems that PEP 470 introduced extra work > > for him that would not have been present had things been left in place. > > IIRC the PyPI maintainers were constantly nagged about ?PyPI reliability issues? that were instead > external hosting issues. Everybody was affected every now and then whenever tummy.com or other external > servers for popular packages were down. Or at least I know *I was*. Way too often. But making them completely unreachable does not increase reliability. :) > > But if the majority prefers PyPI that way, I'll stop arguing. > > I?m not sure what you mean here but if you want to argue for reverting PEP 470, I wouldn?t hold my breath. No, I mean what we're doing now. Stefan Krah From glyph at twistedmatrix.com Tue May 3 20:56:13 2016 From: glyph at twistedmatrix.com (Glyph) Date: Tue, 3 May 2016 17:56:13 -0700 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> Message-ID: <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com> On May 3, 2016, at 2:38 PM, Stefan Krah wrote: > > But making them completely unreachable does not increase reliability. :) But it does increase security. The other motivation, besides reliability, listed in this section , is that: "transparently including external links [is] a security hazard (given that in most cases it allowed a MITM to execute arbitrary Python code on the end users machine)". And, indeed, the URL presently listed on PyPI for the cdecimal upload is an unverified http URL. This means that any evil barista with access to a coffee-shop wifi router could instantly execute user-privileged code on any Python programmer's laptop if they were to `pip install? this externally hosted package, which is one of the reasons why neither `pip? nor `pypi? allow such a thing any more. Please believe me when I say I do not mean the following to be insulting - information security is incredibly confusing, difficult, and rapidly evolving, and I don't blame you for getting it wrong - but maintaining a popular package in this way is dangerously irresponsible. There are solid social reasons to centralize the control of the default package repository in the hands of dedicated experts who can scale their security expertise to a large audience, so that package authors like you and I don't need to do this in order to prevent Python from gaining a reputation as a vector for malware; this package is a case in point. Separately from the issue of how PyPI works, even if you have some reason you need to host it externally (which I seriously doubt), please take the trouble to set up a server with properly verified TLS, or use a '.github.io' hostname that can be verified that way. In the meanwhile, just to demonstrate that it's a trivial amount of work to just host it on PyPI, I checked out this package via a verified mechanism ("git clone https://github.com/bytereef/bytereef.github.io") and created a new pypi-cdecimal package >, via editing the setup.py to change the name, 'python setup.py register', 'python setup.py sdist', 'pip wheel' (for some reason direct 'python setup.py bdist_wheel' didn't work), and 'twine upload'. `pip install pypi-cdecimal? should now work and get you an importable `cdecimal?, and if you happen to be lucky enough to run the same OS version I am, you won't even need to build C code. cdecimal users may wish to retrieve it via this mechanism until there's a secure way to get the proper upstream distribution. If anyone wants package-index access to this name to upload Windows or manylinux wheels just let me know; however, as this is just a proof of concept, I do not intend to maintain it long-term. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue May 3 22:07:45 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 3 May 2016 22:07:45 -0400 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com> References: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com> Message-ID: On 5/3/2016 8:56 PM, Glyph wrote: > setup.py bdist_wheel' didn't work), and 'twine upload'. `pip install > pypi-cdecimal? should now work and get you an importable `cdecimal?, and > if you happen to be lucky enough to run the same OS version I am, you > won't even need to build C code. cdecimal users may wish to retrieve it > via this mechanism until there's a secure way to get the proper upstream > distribution. > > If anyone wants package-index access to this name to upload Windows or > manylinux wheels just let me know; however, as this is just a proof of > concept, I do not intend to maintain it long-term. For Windows, http://www.lfd.uci.edu/~gohlke/pythonlibs/ has 32 and 64 bit builds of cdecimal 2.3 for 2.7, 3.4, and 3.5. (Is cdecimal substantially different from the _decimal added in 3.5?) -- Terry Jan Reedy From tritium-list at sdamon.com Tue May 3 23:44:02 2016 From: tritium-list at sdamon.com (tritium-list at sdamon.com) Date: Tue, 3 May 2016 23:44:02 -0400 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com> Message-ID: <001e01d1a5b7$33978020$9ac68060$@hotmail.com> > > (Is cdecimal substantially different from the _decimal added in 3.5?) > AFAICT, they are unrelated codebases that do about the same thing with the same amount of performance, with the main exception that _decimal in 3.5 does not require one to change their import (or to compile the package themselves.) From stefan at bytereef.org Wed May 4 00:15:10 2016 From: stefan at bytereef.org (Stefan Krah) Date: Wed, 4 May 2016 04:15:10 +0000 (UTC) Subject: [Python-Dev] Yearly PyPI breakage References: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com> Message-ID: > [cut overlong post] Glyph, nice sneaky way to try to divert from the original issue. Your whole post is invalidated by the simple fact that the URL was protected by a hash (which I repeatedly asked to be upgraded to sha256). This was the official scheme promoted by PEP-438, which you should know. But of course your actual intention here is character assassination, pretending to "rescue" cdecimal and trying to divert from the fact that the transition to PEP 470 was handled suboptimally. The very reason for this thread is that the security was silently disabled WITHOUT me getting a notification. What is on PyPI *now* is not what I configured! Please believe me when I say I do not mean the following to be insulting -- people who have done *actual* cryptography to varying degrees often tend to focus on the important parts and aren't impressed by regurgitating catch phrases like SSL and man-in-the-middle: http://cr.yp.to/ecdh.html The amount of security "experts" in the Python community that pontificate on any occasion is pretty annoying. What do you think djb thinks of Twisted? > If anyone wants package-index access to this name to upload Windows or manylinux wheels just let me know; however, as this is just a proof of concept, I do not intend to maintain it long-term. That apparently all you can do: Move bits from place A to place B and not care how long it took to produce them. You are a real hero. Stefan Krah From tritium-list at sdamon.com Wed May 4 02:39:51 2016 From: tritium-list at sdamon.com (tritium-list at sdamon.com) Date: Wed, 4 May 2016 02:39:51 -0400 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com> Message-ID: <003c01d1a5cf$c33133d0$49939b70$@hotmail.com> Are you for real? I honestly do not understand your hostility. You posted a mean-spirited complaint about a policy that is nearly exactly two years old, to the wrong list, and call out the people calmly trying to explain what happened and why, and how you can mitigate it for your own work and organization. What do you intend to accomplish? * PyPI is no longer and index, it is a repository * The decision to disable the index-only features of PyPI were made 2 years ago, including pep438 - plenty of time to make alternate arrangements. * The tooling for hosting your own repository is available, should you actually need to host the files outside of PyPI * The tooling exists to use other indexes * The tooling exists to host your own index that serves your own packages (that you develop or third party packages that you package for your own use), that defaults to PyPI for packages not in your own repository I understand that you are upset that a feature you used was removed; posting with hostility to a list of people who do not even have control over the repository is not a legitimate way to solve your problems. > -----Original Message----- > From: Python-Dev [mailto:python-dev-bounces+tritium- > list=sdamon.com at python.org] On Behalf Of Stefan Krah > Sent: Wednesday, May 04, 2016 00:15 > To: python-dev at python.org > Subject: Re: [Python-Dev] Yearly PyPI breakage > > > > [cut overlong post] > > Glyph, > > nice sneaky way to try to divert from the original issue. Your whole post > is invalidated by the simple fact that the URL was protected by a hash > (which I repeatedly asked to be upgraded to sha256). > > This was the official scheme promoted by PEP-438, which you should know. > But of course your actual intention here is character assassination, > pretending to "rescue" cdecimal and trying to divert from the fact that > the transition to PEP 470 was handled suboptimally. > > > The very reason for this thread is that the security was silently disabled > WITHOUT me getting a notification. What is on PyPI *now* is not what I > configured! > > > Please believe me when I say I do not mean the following to be insulting -- > people who have done *actual* cryptography to varying degrees often tend > to focus on the important parts and aren't impressed by regurgitating > catch phrases like SSL and man-in-the-middle: > > http://cr.yp.to/ecdh.html > > > The amount of security "experts" in the Python community that pontificate > on any occasion is pretty annoying. What do you think djb thinks of Twisted? > > > > If anyone wants package-index access to this name to upload Windows or > manylinux wheels just let me know; however, as this is just a proof of > concept, I do not intend to maintain it long-term. > > That apparently all you can do: Move bits from place A to place B and not > care how long it took to produce them. > > You are a real hero. > > > > Stefan Krah > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > list%40sdamon.com From stefan at bytereef.org Wed May 4 03:23:50 2016 From: stefan at bytereef.org (Stefan Krah) Date: Wed, 4 May 2016 07:23:50 +0000 (UTC) Subject: [Python-Dev] Yearly PyPI breakage References: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com> <003c01d1a5cf$c33133d0$49939b70$@hotmail.com> Message-ID: sdamon.com> writes: > You posted a mean-spirited complaint about a policy that is nearly exactly > two years old, to the wrong list, and call out the people calmly trying to > explain what happened and why, and how you can mitigate it for your own work Yeah, right. This is distutils-sig level. Read Glyph's "calm" and "nice" post again. Also note that *I* had already bowed out of the thread when Glyph unnecessarily decided to spread complete misinformation. Stefan Krah From glyph at twistedmatrix.com Wed May 4 03:38:05 2016 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 4 May 2016 00:38:05 -0700 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com> Message-ID: <53C056CE-390B-4473-A383-DBC2599A5071@twistedmatrix.com> On May 3, 2016, at 9:15 PM, Stefan Krah wrote: > >> [cut overlong post] > > Glyph, > > nice sneaky way to try to divert from the original issue. The original issue, as I understood it, at the start of the thread was "which hoops I have to jump through this year in order to keep pip downloads working". So I showed you the hoops. Your question implied, to me, that you were not presently aware of how easy it is to simply build and upload your packages with sdist and twine. After years and years of horrible setuptools bugs, it certainly seemed plausible to me that if you had long-standing experience with python packaging, you might have perhaps developed a well-deserved low opinion of them in the past. Therefore, you might not be aware of relatively recent improvements to the packaging ecosystem which made this problem trivial to solve. My intent was, therefore, simply to demonstrate that things have improved, and that this was not a hard thing for you to do and could be resolved with a minimum of fuss. I confess that before posting I was made aware that you'd had some personality conflicts with some PyPI maintainers in the past. But that sentence was about the extent and detail level of my understanding. I was not aware to what extent, and the reason I jumped into this particular thread, when I rarely participate in python-dev, was that I hoped a simple explanation of the facts of the matter from someone you hadn't previously interacted with could address your concerns. > Your whole post is invalidated by the simple fact that the URL was protected by a hash (which I repeatedly asked to be upgraded to sha256). Based only on previous discussion here, I had no way to know either of those things. You didn't reference it in the post I was replying to, or in your original post. And, as you say later, PyPI's download URL doesn't include the hash any more, so it wasn't there for me to observe. (There were some manual instructions in your package description but no automated tooling will honor that.) In any case, fragment hashes are not really a suitable general-purpose mechanism as they are only honored by specific tools (like pip) whereas HTTPS verification ought to be universally supported, so IMHO it is a good thing that PyPI is discouraging their use for this purpose. > This was the official scheme promoted by PEP-438, which you should know. But of course your actual intention here is character assassination, pretending to "rescue" cdecimal In the "overlong" post that you elided, I specifically said I didn't intend to maintain it for long. If this wasn't clear, what I meant to say by that comment was that I would keep the index entry available until you had the opportunity to upload some sdists and wheels yourself to PyPI. If you don't intend to, I am not the right person to "rescue" the package; someone else who is more invested in cdecimal should provide an alternate PyPI entry, or take over this one. > and trying to divert from the fact that > the transition to PEP 470 was handled suboptimally. I don't see any need to divert attention from this fact, because you appear to be in a minority of one in considering it so. > The very reason for this thread is that the security was silently disabled WITHOUT me getting a notification. What is on PyPI *now* is not what I configured! If that was the reason for the thread, you would have been better served by making that specific complaint rather than asking for information, and then yelling at the people who provided it to you. You might also consider reporting these issues to an appropriate forum, since python-dev is not the bugtracker for PyPI. You can find that here: >. You might also want to continue this thread on distutils-sig; I'm sorry for contributing to the noise on python-dev, but I thought getting a high-profile package such as cdecimal integrated into the modern packaging ecosystem would be worth the off-topic digression. > [various spurious and irrelevant ad-hominem attacks redacted] Perhaps naively, given the level of hostility on display here, I still hope that you might see the wisdom in simply uploading build artifacts to PyPI. But I won't try to convince you further. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From torkvemada at sorokdva.net Wed May 4 14:37:40 2016 From: torkvemada at sorokdva.net (=?UTF-8?B?0JLRgdC10LLQvtC70L7QtCDQktC10LvQuNGH0LrQvg==?=) Date: Wed, 4 May 2016 21:37:40 +0300 Subject: [Python-Dev] Restore functions pickling Message-ID: Hello, I was recently porting my application from Py2 to Py3 and encountered error, when I accidentally became unable to define my own pickling mechanism for functions. Before that I was using it to pickle pure lambdas, registering my own handler for cPickle with copy_reg, but after [1] (and release 3.4) it seems that feature has been removed. As it mentioned in commit message, this was done for compatibility with pure-python pickle. So I have to ask, may be it would be better to revert the commit and add the corresponding feature to pure-python pickle implementation? [1] https://hg.python.org/cpython/rev/6bd1f0a27e8e -- Best wishes, Vsevolod Velichko -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 4 14:55:32 2016 From: brett at python.org (Brett Cannon) Date: Wed, 04 May 2016 18:55:32 +0000 Subject: [Python-Dev] Restore functions pickling In-Reply-To: References: Message-ID: On Wed, 4 May 2016 at 11:39 ???????? ??????? wrote: > Hello, > > I was recently porting my application from Py2 to Py3 and encountered > error, when I accidentally became unable to define my own pickling > mechanism for functions. > Before that I was using it to pickle pure lambdas, registering my own > handler for cPickle with copy_reg, but after [1] (and release 3.4) it seems > that feature has been removed. > > As it mentioned in commit message, this was done for compatibility with > pure-python pickle. So I have to ask, may be it would be better to revert > the commit and add the corresponding feature to pure-python pickle > implementation? > > > [1] https://hg.python.org/cpython/rev/6bd1f0a27e8e > If you want to provide a patch to implement function pickling in pure Python code and make this work with both pickle.py and _pickle.c we might be up for accepting the patch. You can also open an issue requesting pickling support for functions at bugs.python.org to keep track of the request and see if anyone else is interested in the feature. -------------- next part -------------- An HTML attachment was scrubbed... URL: From torkvemada at sorokdva.net Wed May 4 20:23:15 2016 From: torkvemada at sorokdva.net (=?UTF-8?B?0JLRgdC10LLQvtC70L7QtCDQktC10LvQuNGH0LrQvg==?=) Date: Thu, 5 May 2016 03:23:15 +0300 Subject: [Python-Dev] Restore functions pickling In-Reply-To: References: Message-ID: Super, since you are not against this feature, I created the issue with the pach[1]. Hopefully you or one of your colleagues will have some time to review it. (I signed the contributor form, changes just not reached the bugtracker yet) [1] http://bugs.python.org/issue26959 2016-05-04 21:55 GMT+03:00 Brett Cannon : > > > On Wed, 4 May 2016 at 11:39 ???????? ??????? > wrote: > >> Hello, >> >> I was recently porting my application from Py2 to Py3 and encountered >> error, when I accidentally became unable to define my own pickling >> mechanism for functions. >> Before that I was using it to pickle pure lambdas, registering my own >> handler for cPickle with copy_reg, but after [1] (and release 3.4) it seems >> that feature has been removed. >> >> As it mentioned in commit message, this was done for compatibility with >> pure-python pickle. So I have to ask, may be it would be better to revert >> the commit and add the corresponding feature to pure-python pickle >> implementation? >> >> >> [1] https://hg.python.org/cpython/rev/6bd1f0a27e8e >> > > If you want to provide a patch to implement function pickling in pure > Python code and make this work with both pickle.py and _pickle.c we might > be up for accepting the patch. You can also open an issue requesting > pickling support for functions at bugs.python.org to keep track of the > request and see if anyone else is interested in the feature. > -- Best wishes, Vsevolod Velichko -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 4 21:28:29 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 May 2016 11:28:29 +1000 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: On 4 May 2016 at 05:06, Stefan Krah wrote: > > Hello, > > Could someone enlighten me which hoops I have to jump through > this year in order to keep pip downloads working? Stefan, I know you're not happy with myself and the other distutils-sig folks regarding the decision to deprecate and remove automatic link spidering, nor with the PSF regarding the current Terms of Service around uploads to PyPI, but that doesn't make it OK to start off-topic threads on python-dev just because you're a CPython core developer in addition to being a PyPI user. It *definitely* doesn't make it OK to accuse people of conspiring against you when they answer your question in good faith, just because their answer is the official distutils-sig/PyPA one (which was approved through the PEP process in PEP 470). Depending on which aspect is most bothering you right now, potential courses of action worth pursuing include: - writing to psf-legal to let them know whether or not Van Lindberg's draft updates to the Terms of Service would be sufficient to make you comfortable with uploading cdecimal to PyPI in addition to bundling it with the standard library under your existing Contributor Licensing Agreement: https://bitbucket.org/vanl/pypi/src/default/templates/confirm.pt - writing to distutils-sig to see what opportunities exist to assist with the migration from the fragile legacy PyPI codebase to the new Warehouse implementation (which is a technical debt reduction activity currently blocking a lot of other enhancements to the PyPI ecosystem) - writing to the PSF Board asking us to increase the PSF's level of investment in PyPI infrastructure support (while we're already aware of that need, we're also the right venue to escalate these kinds of concerns for folks that don't have the time or inclination to get directly involved in improving the situation themselves) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed May 4 22:12:19 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 May 2016 12:12:19 +1000 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: <001e01d1a5b7$33978020$9ac68060$@hotmail.com> References: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl> <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com> <001e01d1a5b7$33978020$9ac68060$@hotmail.com> Message-ID: On 4 May 2016 at 13:44, wrote: >> >> (Is cdecimal substantially different from the _decimal added in 3.5?) >> > AFAICT, they are unrelated codebases that do about the same thing with the same amount of performance, with the main exception that _decimal in 3.5 does not require one to change their import (or to compile the package themselves.) cdecimal and the standard library's _decimal module are built around the same decimal arithmetic library (libmpdec), and Stefan is the maintainer for all of them. Similar to other standard library modules with a PyPI counterpart, end users can choose between using the standard library version (and avoiding the external dependency) and using the independently updated version (and gaining increased consistency across Python versions, including availability on 2.7). More info on that can be found in the Python 3.3 What's New doc: https://docs.python.org/3/whatsnew/3.3.html#new-decimal Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From d.srivastava15 at gmail.com Thu May 5 04:24:04 2016 From: d.srivastava15 at gmail.com (Deepak Srivastava) Date: Thu, 5 May 2016 13:54:04 +0530 Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1 or pyCharm Community edition References: Message-ID: <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com> > Dear All, > I am new to python and very much excited to learn this technology. > I have done setup of python 3.5.1 with PyCharm community edition on windows 7(64bit service pack 1). > I am trying to execute some piece of code but it fails . Requesting you to please help. > > My Piece of code is > import pytesseract > from PIL import Image > from PIL import ImageFilter > import os > print (os.getcwd()) > url="Polish.png" > def process_image(url): > image = _get_image(url) > image = image.filter(ImageFilter.SHARPEN) > # image = image.convert('1') > print (pytesseract.image_to_string(image)) > > def _get_image(url): > return Image.open(url) > > process_image(url) > > Error which I am getting , > C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\python.exe C:/Temp/Opt/src/ocr.py > C:\Temp\Opt\src > in subprocess > None > 0 > None > None > > Traceback (most recent call last): > File "C:/Temp/Opt/src/ocr.py", line 17, in > process_image(url) > File "C:/Temp/Opt/src/ocr.py", line 12, in process_image > print (pytesseract.image_to_string(image)) > File "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py", line 161, in image_to_string > config=config) > File "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py", line 94, in run_tesseract > stderr=subprocess.PIPE) > File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 950, in __init__ > restore_signals, start_new_session) > File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 1226, in _execute_child > startupinfo) > FileNotFoundError: [WinError 2] The system cannot find the file specified > > Process finished with exit code 1 > > Thanks and Regards > Deepak Srivastava > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu May 5 11:26:47 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 5 May 2016 10:26:47 -0500 Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1 or pyCharm Community edition In-Reply-To: <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com> References: <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com> Message-ID: On Thu, May 5, 2016 at 3:24 AM, Deepak Srivastava wrote: > > Dear All, > > I am new to python and very much excited to learn this technology. > > I have done setup of python 3.5.1 with PyCharm community edition on > windows 7(64bit service pack 1). > > I am trying to execute some piece of code but it fails . Requesting you to > please help. > > > > My Piece of code is > > *import *pytesseract > *from *PIL *import *Image > *from *PIL *import *ImageFilter > *import *os > print (os.getcwd()) > url= > *"Polish.png"**def *process_image(*url*): > image = _get_image(*url*) > image = image.filter(ImageFilter.SHARPEN) > # image = image.convert('1') > print (pytesseract.image_to_string(image)) > > *def *_get_image(*url*): > *return *Image.open(*url*) > > process_image(url) > > > > Error which I am getting , > > *C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\python.exe > C:/Temp/Opt/src/ocr.py* > > *C:\Temp\Opt\src* > > *in subprocess* > > *None* > > *0* > > *None* > > *None* > > ** > > *Traceback (most recent call last):* > > * File "C:/Temp/Opt/src/ocr.py", line 17, in * > > * process_image(url)* > > * File "C:/Temp/Opt/src/ocr.py", line 12, in process_image* > > * print (pytesseract.image_to_string(image))* > > * File > "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py", > line 161, in image_to_string* > > * config=config)* > > * File > "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py", > line 94, in run_tesseract* > > * stderr=subprocess.PIPE)* > > * File > "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", > line 950, in __init__* > > * restore_signals, start_new_session)* > > * File > "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", > line 1226, in _execute_child* > > * startupinfo)* > > *FileNotFoundError: [WinError 2] The system cannot find the file specified* > > > > *Process finished with exit code 1* > > > > Thanks and Regards > > Deepak Srivastava > > > > Questions like this are better suited for python-list. Regardless, you need to install Tesseract first: https://github.com/tesseract-ocr/tesseract/wiki That should fix the error. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu May 5 12:23:02 2016 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 5 May 2016 17:23:02 +0100 Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1 or pyCharm Community edition In-Reply-To: References: <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com> Message-ID: On 2016-05-05 16:26, Ryan Gonzalez wrote: > On Thu, May 5, 2016 at 3:24 AM, Deepak Srivastava > > wrote: > > >> Dear All, >> >> I am new to python and very much excited to learn this technology. >> >> I have done setup of python 3.5.1 with PyCharm community edition >> on windows 7(64bit service pack 1). >> >> I am trying to execute some piece of code but it fails . >> Requesting you to please help. >> >> My Piece of code is >> >> /import /pytesseract >> /from /PIL /import /Image >> /from /PIL /import /ImageFilter >> /import /os >> print (os.getcwd()) >> url=*"Polish.png" >> */def /process_image(/url/): >> image = _get_image(/url/) >> image = image.filter(ImageFilter.SHARPEN) >> # image = image.convert('1') >> print (pytesseract.image_to_string(image)) >> >> /def /_get_image(/url/): >> /return /Image.open(/url/) >> >> process_image(url) >> >> Error which I am getting , >> >> *C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\python.exe >> C:/Temp/Opt/src/ocr.py* >> >> *C:\Temp\Opt\src* >> >> *in subprocess* >> >> *None* >> >> *0* >> >> *None* >> >> *None* >> >> ** >> >> *Traceback (most recent call last):* >> >> * File "C:/Temp/Opt/src/ocr.py", line 17, in * >> >> *process_image(url)* >> >> * File "C:/Temp/Opt/src/ocr.py", line 12, in process_image* >> >> * print (pytesseract.image_to_string(image))* >> >> * File >> "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py", >> line 161, in image_to_string* >> >> *config=config)* >> >> * File >> "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py", >> line 94, in run_tesseract* >> >> *stderr=subprocess.PIPE)* >> >> * File >> "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", >> line 950, in __init__* >> >> *restore_signals, start_new_session)* >> >> * File >> "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", >> line 1226, in _execute_child* >> >> * startupinfo)* >> >> *FileNotFoundError: [WinError 2] The system cannot find the file >> specified* >> >> ** >> >> *Process finished with exit code 1* >> >> Thanks and Regards >> >> Deepak Srivastava >> > > Questions like this are better suited for python-list. Regardless, you > need to install Tesseract first: > > https://github.com/tesseract-ocr/tesseract/wiki > > That should fix the error. It looks to me that it's the url="Polish.png" that's the real cause of the problem. It's not clear where the file "Polish.png" is actually stored. It would be better to use a full file path. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu May 5 12:51:35 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 6 May 2016 02:51:35 +1000 Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1 or pyCharm Community edition In-Reply-To: References: <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com> Message-ID: <20160505165135.GQ12028@ando.pearwood.info> On Thu, May 05, 2016 at 05:23:02PM +0100, MRAB wrote: > On 2016-05-05 16:26, Ryan Gonzalez wrote: > >On Thu, May 5, 2016 at 3:24 AM, Deepak Srivastava [ ... about three or four pages of quoting ... ] > >Questions like this are better suited for python-list. [...] > It looks to me that it's the url="Polish.png" [...] Guys, it's bad enough that you're engaging in problem-solving mode for an off-topic question, but could you at least trim your replies first? There's no need to quote the entire body of the OP's post (three or four pages worth) only to add a two or three line comment to the end. Have pity on your readers and use the delete key on some of the unnecessary text, please? -- Steve From zreed at fastmail.com Thu May 5 14:28:22 2016 From: zreed at fastmail.com (zreed at fastmail.com) Date: Thu, 05 May 2016 13:28:22 -0500 Subject: [Python-Dev] Wordcode: new regular bytecode using 16-bit units Message-ID: <1462472902.2822020.599321113.046AABAF@webmail.messagingengine.com> Here is something I wrote because I was also unsatisfied with byteplay's API: https://github.com/zachariahreed/byteasm. Maybe it's useful in a discussion of "minimum viable" api for bytecode manipulation. From rymg19 at gmail.com Thu May 5 15:18:10 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 5 May 2016 14:18:10 -0500 Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1 or pyCharm Community edition In-Reply-To: References: <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com> Message-ID: Well, the stack trace was pointing to the line that called Tesseract, so I figured that was the problem. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On May 5, 2016 11:24 AM, "MRAB" wrote: > > It looks to me that it's the url="Polish.png" that's the real cause of the problem. It's not clear where the file "Polish.png" is actually stored. It would be better to use a full file path. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Thu May 5 18:22:01 2016 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 5 May 2016 22:22:01 +0000 (UTC) Subject: [Python-Dev] Yearly PyPI breakage References: Message-ID: Nick Coghlan gmail.com> writes: > I know you're not happy with myself and the other distutils-sig folks > regarding the decision to deprecate and remove automatic link > spidering, More accurately: Not happy with the removal of the checksummed "explicit" mode. What I would have preferred is a FreeBSD-like ports system. FreeBSD has been used in huge and secure installations, so the I don't buy the reliability and security arguments that are used in favor of centralization. But centralization seems to be a goal in and of itself these days (and that isn't limited to Python). > nor with the PSF regarding the current Terms of Service > around uploads to PyPI, but that doesn't make it OK to start off-topic > threads on python-dev just because you're a CPython core developer in > addition to being a PyPI user. Alexander thought otherwise: https://mail.python.org/pipermail/python-dev/2015-October/141840.html "Given that ensurepip is part of stdlib, I am not sure this is an accurate statement. Even if it was, did you make any effort to discuss the proposal outside of a small group subscribed to distutils ML?" I completely agree with that. Fredrik Lundh is also affected (and might not have received any mail, same as me): https://pypi.python.org/pypi/PIL > It *definitely* doesn't make it OK to accuse people of conspiring > against you when they answer your question in good faith, just because > their answer is the official distutils-sig/PyPA one (which was > approved through the PEP process in PEP 470). I'm not sure what you are referring to. Donald posted a link to PEP 470, in my response to Nathaniel I acknowledged this. In my exchange with ?ukasz we both came to the conclusion (I think) that further discussion would be futile. IMO all responses from Brett, Donald, Nathaniel and ?ukasz were reasonable and I haven't accused them of conspiring in the slightest. I see that the PEP was accepted by Paul Moore. I couldn't even dream of accusing Paul Moore of any kind of conspiracy. He's one of the most reasonable (and *genuinely* polite) people on these mailing lists. Or are you referring to a super-condescending flame bait where someone cloned a private website, assumed general ignorance and then proceeded to offer a hostile fork to anyone who would be interested? Well, I accepted the flame bait. > - writing to psf-legal to let them know whether or not Van Lindberg's > draft updates to the Terms of Service would be sufficient to make you > comfortable with uploading cdecimal to PyPI in addition to bundling it > with the standard library under your existing Contributor Licensing > Agreement: https://bitbucket.org/vanl/pypi/src/default/templates Okay, so there was recent progress here. This is actual news to me. Do you remember what kind of derision I went through for just suggesting something like that a couple of years ago? Yes, you supported me, but what about the others? Or that Van Lindberg was in the merry group of Twitter heroes who were gloating about the fact that (in their opinion) I could not do anything about the first hostile fork? Technically, he didn't gloat, but suddenly legal advice was apparently readily available. https://pypi.python.org/pypi/m3-cdecimal I can assure you that CoCs or "diversity statements" won't help you at all. Stefan Krah From breamoreboy at yahoo.co.uk Thu May 5 18:58:18 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Thu, 5 May 2016 23:58:18 +0100 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: On 05/05/2016 23:22, Stefan Krah wrote > > Fredrik Lundh is also affected (and might not have received any mail, > same as me): > > https://pypi.python.org/pypi/PIL > He might be, but clearly the Python community as a whole is not impacted. From what I see the latest version of PIL that is available is 1.1.6, which requires Python 1.5.2 or higher, and has the following stats:- 0 downloads in the last day 0 downloads in the last week 0 downloads in the last month I wish I could vent my feelings regarding your comments earlier in this thread but I won't, as apparently core developers can say what they like with no comeback, whereas plebs like me can and will get hammered by the Python community, despite having explained that a combination of anxiety, depression, chronic fatigue syndrome, insomnia, autism and diplopia makes life rather difficult. Unless of course you have similar problems, in which case please say so. The rest of the community might not understand, I certainly will. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From chris.barker at noaa.gov Thu May 5 19:03:58 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 5 May 2016 16:03:58 -0700 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: > > Fredrik Lundh is also affected (and might not have received any mail, >> same as me): >> >> https://pypi.python.org/pypi/PIL >> >> > He might be, but clearly the Python community as a whole is not impacted. > From what I see the latest version of PIL that is available is 1.1.6, which > requires Python 1.5.2 or higher, Indeed -- Fredrik never made any effort to support pypi, pip, etc. -- that's why the Pillow fork was started in the first place. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu May 5 19:06:57 2016 From: barry at python.org (Barry Warsaw) Date: Thu, 5 May 2016 18:06:57 -0500 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: <20160505180657.7be52a5e.barry@wooz.org> On May 05, 2016, at 11:58 PM, Mark Lawrence via Python-Dev wrote: >On 05/05/2016 23:22, Stefan Krah wrote >> >> Fredrik Lundh is also affected (and might not have received any mail, >> same as me): >> >> https://pypi.python.org/pypi/PIL Maybe, but then there's the friendly fork: https://pypi.python.org/pypi/Pillow/3.2.0 -Barry From donald at stufft.io Thu May 5 19:07:52 2016 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 May 2016 19:07:52 -0400 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: > On May 5, 2016, at 6:22 PM, Stefan Krah wrote: > > Nick Coghlan gmail.com> writes: >> I know you're not happy with myself and the other distutils-sig folks >> regarding the decision to deprecate and remove automatic link >> spidering, > > More accurately: Not happy with the removal of the checksummed "explicit" > mode. What I would have preferred is a FreeBSD-like ports system. FreeBSD > has been used in huge and secure installations, so the I don't buy the > reliability and security arguments that are used in favor of centralization. > But centralization seems to be a goal in and of itself these days (and > that isn't limited to Python). Let me say outright that there is nothing wrong with a well designed, ports like system. However, PyPI was not that and the vast bulk of packages were not using it in that fashion. At the time of PEP 470 being written a mere 59 out of 65,232 (at the time) projects were utilizing the feature that allowed people to safely do a ports like system. That's roughly 0.09% of all of the projects on PyPI which meant those projects were in a very serious minority. Thus it is only natural that the expectations end users had when installing software was that it was coming from PyPI, not from some external host and they made decisions based around those expectations and when those expectations were violated they came to the people operating PyPI and developing pip and were at best very confused, and at worst irate at *us*. It's also true that there are benefits and cons to both a repository centric approach and a ports style approach, however since we were attempting to do both in one repository it typically meant that we got all of the cons of both systems, but we couldn't ever take advantage of any of the unique benefits of either system since it would only apply to some fraction of the projects. This meant that improvements that would help people installing 99.11% of projects would end up getting blocked by 0.09% of projects. At the end of the day, attempting to both things in one repository just wasn't working out very well in the general case, so we needed (IMO) to pick one way and stop supporting the other way. I personally think it's a pretty obvious choice to go the way that we did, given that the overwhelming majority of projects were already being used that way. That being said, I don't think the characterization of "centralizing" is really accurate here. People are still free to host their projects wherever they want and pip (and easy_install) can be used to install them. All it requires is telling pip where it needs to look in addition to the default location (PyPI), which can be configured via the CLI, environment variables, or a number of configuration files at the system, user, and virtual environment level. You're free to continue providing a page on PyPI to enable discovery of your project and you can include in that page instructions on how to install your project. > > >> nor with the PSF regarding the current Terms of Service >> around uploads to PyPI, but that doesn't make it OK to start off-topic >> threads on python-dev just because you're a CPython core developer in >> addition to being a PyPI user. > > Alexander thought otherwise: > > https://mail.python.org/pipermail/python-dev/2015-October/141840.html > > "Given that ensurepip is part of stdlib, I am not sure this is an accurate > statement. Even if it was, did you make any effort to discuss the proposal > outside of a small group subscribed to distutils ML?" > > I completely agree with that. We've sort of hijacked the PEP process to do something that it wasn't really meant to do. Changes like PEP 470 have nothing to do with what Python itself does and thus it's really outside of the realm of where python-dev really needs a say or notification. In Alexander's example of PEP 495 it was talking about adding a new API to Python itself, thus it makes sense that python-dev should be told about that PEP, and in cases where we needed to change the standard library we've done that discussion on python-dev (such as with PEP 453). The python-dev list is not the list to notify people of changes to things in all corners of the Python ecosystem. If people want to get involved or keep abreast of the changes in the external projects of PyPI, pip, setuptools, etc then they should subscribe to the places that those changes get discussed-- which is distutils-sig. Packaging already has enough problems with getting bogged down in "why wasn't I consulted" and bikeshedding, I don't think that inviting folks (and nothing against python-dev here) a chance to try and re-legislate an agreed upon change after the fact is going to provide much in the way of benefits and will instead just contribute further to stymying efforts to move packaging forward. If it was decided that we really need all PEPs announced on python-dev and python-dev given a chance to weigh in, I would advocate for packaging to stop using the PEP process all together and create our own process to mirror the PEP process because again, we're not using the PEP process because these things naturally fall under what the PEP process was designed to cover, we're using it because it was there already and close enough that we could hijack it and it was moderately easier to just do that. If the downsides of using the PEP process become greater than the effort to come up with our own process then I think it's a no brainer to just do our own thing. > > > Fredrik Lundh is also affected (and might not have received any mail, > same as me): > > https://pypi.python.org/pypi/PIL I've personally emailed the PIL offer several times through various email addresses over several years and have never gotten a response. At this point I can only assume either they are willfully ignoring those messages or they've completely changed their contact information in a way that I have not been able to find updated contacted information for at all. > Technically, he > didn't gloat, but suddenly legal advice was apparently readily available. > > https://pypi.python.org/pypi/m3-cdecimal > I'm not going to respond to the rest of this, because I don't think it's possible for you and me to have a constructive discussion on these other issues. However, I do want to point out that the PSF is the legal entity behind PyPI itself and Van is both a board member and a lawyer. I don't recall what exactly was discussed with who about m3-cdecimal, but it's completely reasonable that when a legal question comes up with PyPI that we consult the PSF board members, particularly the ones who are lawyers. I do recall that the last time m3-cdecimal came up [1] you (again) brought up issues you had with PyPI in an inappropriate venue and as far as I know, you never actually used any of the contact channels that you were offered to try and resolve the issue that you had with a project on PyPI. [1] https://bugs.python.org/issue22483 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stefan at bytereef.org Thu May 5 19:13:24 2016 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 5 May 2016 23:13:24 +0000 (UTC) Subject: [Python-Dev] Yearly PyPI breakage References: Message-ID: Chris Barker noaa.gov> writes: > Indeed -- Fredrik never made any effort to support pypi, pip, etc. -- that's why the Pillow fork was started in the first place. Maybe, but he created PIL, so thank you, Fredrik! Stefan Krah From breamoreboy at yahoo.co.uk Thu May 5 19:24:57 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Fri, 6 May 2016 00:24:57 +0100 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: <20160505180657.7be52a5e.barry@wooz.org> References: <20160505180657.7be52a5e.barry@wooz.org> Message-ID: On 06/05/2016 00:06, Barry Warsaw wrote: > On May 05, 2016, at 11:58 PM, Mark Lawrence via Python-Dev wrote: > >> On 05/05/2016 23:22, Stefan Krah wrote >>> >>> Fredrik Lundh is also affected (and might not have received any mail, >>> same as me): >>> >>> https://pypi.python.org/pypi/PIL > > Maybe, but then there's the friendly fork: > > https://pypi.python.org/pypi/Pillow/3.2.0 > > -Barry > Where is the relevance to my words that you've snipped? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From dw+python-dev at hmmz.org Thu May 5 19:42:30 2016 From: dw+python-dev at hmmz.org (David Wilson) Date: Thu, 5 May 2016 23:42:30 +0000 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: <20160505234230.GB14936@k3> This is mostly just observational, and not meant primarily as criticism of the fabulous work of Donald and others (ignoring pypa, also the prompt, reliable, and skilled support responses common on such places as IRC), however I can't help but notice that PyPI governance seems to come under fire vastly more often than similar and much more popular packaging systems, and some choices that have been made particularly in recent years have caused a noticeable amount of dissent with what might be considered the typical developer. When a contributor to the core language is having repeat issues maintaining some basic element of the function of the packaging system, might it be fair to reflect on how changes to those functions are being managed? There are PEPs covering a great deal of the work done to PyPI recently, but, and I say this as someone who has bumped into friction with the packaging tooling in the relatively recent past, even I despite my motivations to the contrary, I have not read most of them. It seems the current process is observed by few, does not sufficiently address the range of traditional use cases that were possible in the past, and the first the common user learns of a change is when pip (after insisting it must be upgraded) fails to function as it previously did. The usual course then is some complaint, that leads to distutils-sig, which ultimately leads to pointing at some design work that was only observed by perhaps 50 people max that turns out had some edge cases that hurt in a common use case. Is there something to contemplate in here? I dislike posting questions instead of answers, but it seems apparent there is a problem here and it continues to remain unaddressed. David On Tue, May 03, 2016 at 07:06:12PM +0000, Stefan Krah wrote: > > Hello, > > Could someone enlighten me which hoops I have to jump through > this year in order to keep pip downloads working? > > Collecting cdecimal > Could not find a version that satisfies the requirement cdecimal (from > versions: ) > No matching distribution found for cdecimal > You are using pip version 7.1.2, however version 8.1.1 is available. > You should consider upgrading via the 'pip install --upgrade pip' command. > > > If this continues, I'm going to release a premium version that's > 50% faster and only available from bytereef.org or Anaconda. > > > > Stefan Krah > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/dw%2Bpython-dev%40hmmz.org From brett at python.org Thu May 5 20:03:48 2016 From: brett at python.org (Brett Cannon) Date: Fri, 06 May 2016 00:03:48 +0000 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: <20160505234230.GB14936@k3> References: <20160505234230.GB14936@k3> Message-ID: On Thu, 5 May 2016 at 16:54 David Wilson wrote: > This is mostly just observational, and not meant primarily as criticism > of the fabulous work of Donald and others (ignoring pypa, also the > prompt, reliable, and skilled support responses common on such places as > IRC), however I can't help but notice that PyPI governance seems to come > under fire vastly more often than similar and much more popular > packaging systems, and some choices that have been made particularly in > recent years have caused a noticeable amount of dissent with what might > be considered the typical developer. > > When a contributor to the core language is having repeat issues > maintaining some basic element of the function of the packaging system, > might it be fair to reflect on how changes to those functions are being > managed? > > There are PEPs covering a great deal of the work done to PyPI recently, > but, and I say this as someone who has bumped into friction with the > packaging tooling in the relatively recent past, even I despite my > motivations to the contrary, I have not read most of them. It seems the > current process is observed by few, does not sufficiently address the > range of traditional use cases that were possible in the past, and the > first the common user learns of a change is when pip (after insisting it > must be upgraded) fails to function as it previously did. > > The usual course then is some complaint, that leads to distutils-sig, > which ultimately leads to pointing at some design work that was only > observed by perhaps 50 people max that turns out had some edge cases > that hurt in a common use case. > > Is there something to contemplate in here? I dislike posting questions > instead of answers, but it seems apparent there is a problem here and it > continues to remain unaddressed. > This is whole thread is off-topic precisely because all of this is discussed -- in the open -- on distutils-sig and decided there. If people feel changes need to be made like broadcasting to a wider audience when a change occurs, then please bring it up on distutils-sig. But if people choose not to participate then they are implicitly delegating decision powers to those who do participate (which is totally fine and I'm not implying that if people don't participate they are doing something wrong). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu May 5 20:13:11 2016 From: brett at python.org (Brett Cannon) Date: Fri, 06 May 2016 00:13:11 +0000 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: Message-ID: On Thu, 5 May 2016 at 16:00 Mark Lawrence via Python-Dev < python-dev at python.org> wrote: > On 05/05/2016 23:22, Stefan Krah wrote > > > > Fredrik Lundh is also affected (and might not have received any mail, > > same as me): > > > > https://pypi.python.org/pypi/PIL > > > > He might be, but clearly the Python community as a whole is not > impacted. From what I see the latest version of PIL that is available > is 1.1.6, which requires Python 1.5.2 or higher, and has the following > stats:- > > 0 downloads in the last day > 0 downloads in the last week > 0 downloads in the last month > > I wish I could vent my feelings regarding your comments earlier in this > thread but I won't, as apparently core developers can say what they like > with no comeback, I don't think that's fair. Several people pointed out that Stefan's initial email was off-topic and somewhat rude, but was probably given some slack due to the fact that having deployments to the Cheeseshop fail can be frustrating (leeway I think anyone posting here would have received). At this point I think there's nothing new to be said and unless someone wants to take a more drastic step like a formal CoC complaint or calling for the heads of the management of distutils-sig on spikes, this thread has run its course. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Thu May 5 20:24:57 2016 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 6 May 2016 00:24:57 +0000 (UTC) Subject: [Python-Dev] Yearly PyPI breakage References: Message-ID: Donald Stufft stufft.io> writes: > > Technically, he > > didn't gloat, but suddenly legal advice was apparently readily available. > > > > https://pypi.python.org/pypi/m3-cdecimal > > > > I'm not going to respond to the rest of this, because I don't think it's > possible for you and me to have a constructive discussion on these other > issues. However, I do want to point out that the PSF is the legal entity behind > PyPI itself and Van is both a board member and a lawyer. I don't recall what > exactly was discussed with who about m3-cdecimal, but it's completely > reasonable that when a legal question comes up with PyPI that we consult the > PSF board members, particularly the ones who are lawyers. It is not reasonable at all. A normal human being would try to consider first if an author has moral rights as opposed to legal rights and maybe pick up the phone or use private email instead of escalating everything to Twitter. Also, what was discussed was not whether the PSF had any right to remove m3-cdecimal but rather whether *I* had any rights to *demand* the removal. People concluded gleefully that I hadn't any (I still think they're mistaken and an enterprising Oracle lawyer could come to a different conclusion, but that's besides the point). Stefan Krah From dw+python-dev at hmmz.org Thu May 5 20:35:27 2016 From: dw+python-dev at hmmz.org (David Wilson) Date: Fri, 6 May 2016 00:35:27 +0000 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: References: <20160505234230.GB14936@k3> Message-ID: <20160506003527.GA30649@k3> On Fri, May 06, 2016 at 12:03:48AM +0000, Brett Cannon wrote: > Is there something to contemplate in here? I dislike posting questions > instead of answers, but it seems apparent there is a problem here and it > continues to remain unaddressed. > This is whole thread is off-topic precisely because all of this is > discussed -- in the open -- on distutils-sig and decided there. If > people feel changes need to be made like broadcasting to a wider > audience when a change occurs, then please bring it up on > distutils-sig. I respectfully disagree, as this has been the default response applied for years, and it seems friction and dissemination have not been improved by it. Packaging is not some adjunct technicality, anyone learning Python in the past few years at least has been taught pip within the first week. > But if people choose not to participate then they are implicitly > delegating decision powers to those who do participate I believe this is also practically rhetorical in nature. I've watched the wars on distutils-sig for many years now, and the general strategy is that beyond minor outside influence, the process there is occupied by few individuals who are resistant to outside change. Outside influence is regularly met with essay-length reponses and tangential minutia until the energy of the challenge is expended. As an example, one common argument is that "Donald is overworked", however as an example, I offered a very long time ago to implement full text indexing for PyPI search. At the time I belive I was told such things weren't necessary, only to learn a few years later that Donald himself implemented the same function, and it suffers from huge latency and accuracy issues in the meantime. The solution to those problems is of course the ever-delayed rewrite. Over on distutils-sig, one will learn that a large amount of effort has been poured into a rewrite of PyPI (an effort going on years now), however the original codebase was not far from rescue (I had a local copy almost entirely ported to Flask in a few days). There is no reason why this effort nor any other (like full text search) should be used, as it often is, as an argument in the decisionmaking process that largely governs how PyPI and pip have worked in the recent years, yet it only takes a few glances at the archives to demonstrate that it regularly is. David From stefan at bytereef.org Thu May 5 21:28:20 2016 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 6 May 2016 01:28:20 +0000 (UTC) Subject: [Python-Dev] Yearly PyPI breakage References: <20160505234230.GB14936@k3> Message-ID: Brett Cannon python.org> writes: > This is whole thread is off-topic precisely because all of this is discussed -- in the open -- on distutils-sig and decided there. If people feel changes need to be made like broadcasting to a wider audience when a change occurs, then please bring it up on distutils-sig. But if people choose not to participate then they are implicitly delegating decision powers to those who do participate (which is totally fine and I'm not implying that ?if people don't participate they are doing something wrong). Participating there, especially for a non-native speaker, is basically a full-time job. You're met with a large number of extremely lengthy posts that would require an incredible amount of time to respond to in a careful manner. And ultimately it is the persons that Guido delegated the authority to who decide everything. I think many people have concluded that the influence/time ratio is too low to be worth it. Also I don't know any other development group who is a) that quick in trying to suppress any "off-topic" discussions and b) constantly uses venues outside of the Python mailing lists to steer and manipulate public opinion. Stefan Krah From donald at stufft.io Thu May 5 22:31:48 2016 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 May 2016 22:31:48 -0400 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: <20160506003527.GA30649@k3> References: <20160505234230.GB14936@k3> <20160506003527.GA30649@k3> Message-ID: <9F53FF72-6CCB-482A-B4EB-E8165E404362@stufft.io> > On May 5, 2016, at 8:35 PM, David Wilson wrote: > > On Fri, May 06, 2016 at 12:03:48AM +0000, Brett Cannon wrote: > >> Is there something to contemplate in here? I dislike posting questions >> instead of answers, but it seems apparent there is a problem here and it >> continues to remain unaddressed. > >> This is whole thread is off-topic precisely because all of this is >> discussed -- in the open -- on distutils-sig and decided there. If >> people feel changes need to be made like broadcasting to a wider >> audience when a change occurs, then please bring it up on >> distutils-sig. > > I respectfully disagree, as this has been the default response applied > for years, and it seems friction and dissemination have not been > improved by it. Packaging is not some adjunct technicality, anyone > learning Python in the past few years at least has been taught pip > within the first week. > > >> But if people choose not to participate then they are implicitly >> delegating decision powers to those who do participate > > I believe this is also practically rhetorical in nature. I've watched > the wars on distutils-sig for many years now, and the general strategy > is that beyond minor outside influence, the process there is occupied by > few individuals who are resistant to outside change. Outside influence > is regularly met with essay-length reponses and tangential minutia until > the energy of the challenge is expended. Honestly, I just don't think this is an honest characterization. It is true that in general there are few people who bother to put in the effort to take a proposal from start to finish including actually writing the code to make such a thing happen. The problem of packaging is a particularly hard one where it's difficult to make trade offs because unlike other systems, people are sort of locked into using whatever the popular system is. If asyncio doesn't suite everyone that's fine it's not hard for them to go and switch and use Twisted, Tornado, gevent, eventlet, curio, etc etc but it's not very realistic for someone to opt out of the packaging ecosystem. In addition, it's much like something like HTTP or SMTP or the such where once you add some feature, it becomes incredibly difficult to ever remove it if you end up needing to (case in point, this thread and off site hosting), so we tend to over scrutinize changes wherever we can to try and make sure we *really* understand what the tradeoffs we're making are. Just as an example, it took a year and a half for PEP 440 to be standardized which is for something as bite sized as version numbers and PEP 440 was a continuation of the stalled PEP 386. The most time consuming part of PEP 440 was trying our new rules against every single version that existed on PyPI and minimizing the breakage. Even then, we've had several recent PEPs go through (manylinux1, environment markers, requirements) from people willing to do so. > > As an example, one common argument is that "Donald is overworked", > however as an example, I offered a very long time ago to implement full > text indexing for PyPI search. At the time I belive I was told such > things weren't necessary, only to learn a few years later that Donald > himself implemented the same function, and it suffers from huge latency > and accuracy issues in the meantime. The solution to those problems is > of course the ever-delayed rewrite. I don't remember the specific details around your proposal, but I'm pretty sure that *I* never told you that "such things were not necessary" since I've never held that opinion to my memory. What I have found [1][2] (to try and refresh my memory) are threads from you that went largely unanswered in April of 2013, which was right at the time I was heavily focused on getting PyPI setup behind Fastly and wasn't really paying attention to much else. I don't see anything else from you about it until September of 2015 when you offered your help again and at that point I told you that we had already switched to using Elasticsearch. In those two years since your initial offer, and then your follow up in 2015 I do not see any pull request to the PyPI code base from you (which, assuming they were reasonable would have been merged), however I do see the PR [3] from Ernest (No, I didn't implement the search) which got merged and deployed. My experience is that people are often willing to *offer* help with PyPI, but then they quickly disappear once they start to actually try and hack on it's code base and realize how difficult it is to work with. That experience tends to mean that I don't really get super excited when someone shows interest in helping because it rarely actually manifests. I think we've had more people contribute to Warehouse in the last year than we've *ever* had contribute to the legacy PyPI code base, which I think says a lot about the decision to switch. > > Over on distutils-sig, one will learn that a large amount of effort has > been poured into a rewrite of PyPI (an effort going on years now), > however the original codebase was not far from rescue (I had a local > copy almost entirely ported to Flask in a few days). There is no reason > why this effort nor any other (like full text search) should be used, as > it often is, as an argument in the decisionmaking process that largely > governs how PyPI and pip have worked in the recent years, yet it only > takes a few glances at the archives to demonstrate that it regularly is. > I find the statement that "the original code base is not far from rescue" a bit interesting, since you had previously stated [2] that: Again PyPI has been growing organically for a very long time, and dumping even more features in there doesn't seem a great idea. I looked at retrofitting PyPI with Flask, but there is simply too much custom code to be sure things won't be broken by doing it in a hurry. In any case, yes we're rewriting PyPI and it's been something that I've put a lot of effort into. That also means that I'm not particularly enthused about spending a bunch of time and effort working on legacy PyPI because doing so is incredibly demotivating to me to the point that the more time I spent in that code base the more I want to quit all together. Even given that I still have and would review PRs to that code base, it's just that very few people ever suffer through that code base long enough to actually submit one. I don't believe we've ever told someone that something can't happen because of Warehouse, only that *I* won't implement something until after Warehouse. That often times means that something won't happen until after Warehouse because of the severe shortage of people with enough time and motivation to work on this stuff but if someone did step up more things would get done. [1] https://mail.python.org/pipermail/distutils-sig/2013-April/020553.html [2] https://groups.google.com/d/msg/pypa-dev/ZjUNkczsKos/SzwNOckisXUJ [3] https://bitbucket.org/pypa/pypi/pull-requests/81/implement-an-elasticsearch-index-for-pypi/diff ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From josephnunn at gmail.com Fri May 6 05:42:06 2016 From: josephnunn at gmail.com (Joseph Nunn) Date: Fri, 6 May 2016 09:42:06 +0000 (UTC) Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems Message-ID: Hello Everyone, my first post. I would also like to subscribe but haven't seen how yet. So I've an issue with how the 3.5.1 OSX package installs on OSX. The problem was first made apparent when trying to create a virtual environment and `pip install ZODB`, which failed as clang and the linker could not find either the include or lib directories for a dependency. By adding a -v option to the compiler options I was able to see in the compiler flags the following: -isysroot /Developer/SDKs/MacOSX10.6.sdk Which was not present in the compiler flags of other people who I was talking with. The isysroot flag changes the location of includes and libs from the system default, which would be why they can't be found. After grepping around a bit I found this file: /Library/Frameworks/Python.framework/Versions/3.5 /lib/python3.5/_sysconfigdata.py Within which are many mentions of this flag, as so: 'CONFIGURE_CFLAGS': '-arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -g', 'CONFIGURE_CFLAGS_NODIST': '-Werror=declaration-after -statement', 'CONFIGURE_CPPFLAGS': '-isysroot /Developer/SDKs/MacOSX10.6.sdk', 'CONFIGURE_LDFLAGS': '-arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -g', 'CONFIG_ARGS': "'-C' '--enable-framework' '--enable- universalsdk=/Developer/SDKs/MacOSX10.6.sdk' '--with-universal-archs=intel' '--with-computed-gotos' '--without-ensurepip' 'LDFLAGS=-g' 'CFLAGS=-g' 'CC=gcc-4.2'", among others. By running Python 3.5's sysconfig.py as a module, via `python -m sysconfig`, I see the following at the beginning of the output: Platform: "macosx-10.6-intel" Python version: "3.5" Current installation scheme: "posix_prefix" And when I simply go into python interactive mode, import sysconfig, and type sysconfig.get_platform() it returns macosx-10.6-intel. I believe somehow the platform being reported is incorrect, but looking at my environment, I do not have any kind of compiler flags or anything else like that set in my bash init files. Can someone who has a working OSX 3.5 installation check their reported platform to see what it is and if anyone has an idea as to how this information is getting into the installation and where I might change it please let me know. Sincerely, Joseph josephnunn at gmail.com From nad at python.org Fri May 6 07:20:12 2016 From: nad at python.org (Ned Deily) Date: Fri, 6 May 2016 07:20:12 -0400 Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems In-Reply-To: References: Message-ID: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org> On May 6, 2016, at 05:42, Joseph Nunn wrote: > Hello Everyone, my first post. I would also like to subscribe but > haven't seen how yet. > > So I've an issue with how the 3.5.1 OSX > package installs on OSX. The problem was first made apparent > when trying to create a virtual environment and `pip install ZODB`, > which failed as clang and the linker could not find either the > include or lib directories for a dependency. By adding a -v option > to the compiler options I was able to see in the compiler flags > the following: > > -isysroot /Developer/SDKs/MacOSX10.6.sdk > > Which was not present in the compiler flags of other people > who I was talking with. [...] Hi Joseph! Welcome! Note this list is for the development of Python itself, not for usage or installation issues. The Python Developer's Guide (https://docs.python.org/devguide/) including its FAQ gives information about mailing lists and other support forums to get help with general Python issues. That said, without more information, it would be difficult to know exactly what problem you are running into but, since you mention missing headers, my guess is that you need to install Apple's Command Line Tools as described here: https://docs.python.org/devguide/setup.html#build-dependencies Installing Xcode by itself is not sufficient. The sysroot and flag values you note in _sysconfigdata.py are normal and there for compatibility with installing on older versions of OS X; they should not cause a problem on newer releases. For what it's worth, I was able to install ZODB with the current python.org 3.5.1 on OS X 10.11.4 using either venv or virtualenv. Good luck! --Ned P.S. Mailing list info is available here: https://mail.python.org. The bug tracker is here: https://bugs.python.org -- Ned Deily nad at python.org -- [] From larry at hastings.org Fri May 6 11:34:08 2016 From: larry at hastings.org (Larry Hastings) Date: Fri, 6 May 2016 08:34:08 -0700 Subject: [Python-Dev] 3.4.4 Release Schedule In-Reply-To: <1462291641.3972729.596903881.4CD8087B@webmail.messagingengine.com> References: <1462291641.3972729.596903881.4CD8087B@webmail.messagingengine.com> Message-ID: <572CB970.3030601@hastings.org> On 05/03/2016 09:07 AM, Benjamin Peterson wrote: > 2.7.12rc1 will be on June 11th, and the final will be on June 25. That's perfectly cromulent! I'm going to piggyback on that for 3.4. Therefore, 3.4.4 rc1 will be on June 11th, and 3.4.4 final will be on June 25. Reminder: 3.4.4 will be a source-only release. June's coming up Python, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Fri May 6 11:40:52 2016 From: larry at hastings.org (Larry Hastings) Date: Fri, 6 May 2016 08:40:52 -0700 Subject: [Python-Dev] 3.4.5 Release Schedule (not 3.4.4, whoops) In-Reply-To: <572CB970.3030601@hastings.org> References: <1462291641.3972729.596903881.4CD8087B@webmail.messagingengine.com> <572CB970.3030601@hastings.org> Message-ID: <572CBB04.8090501@hastings.org> On 05/06/2016 08:34 AM, Larry Hastings wrote: > That's perfectly cromulent! I'm going to piggyback on that for 3.4. > Therefore, 3.4.4 rc1 will be on June 11th, and 3.4.4 final will be on > June 25. Whoops, a whole bunch of off-by-one bugs. First, I'll be releasing 3.4.5 in June. 3.4.4 was released in December 2015. D'oh! Second, Benjamin and I have slightly different approaches--I like to tag on Saturday and release on Sunday. Therefore 3.4.5 rc1 will be released June 12th, and 3.4.5 final will be released June 26th. (3.4.5 rc1 will be tagged Saturday June 11, 3.4.5 final will be tagged June 25th.) The release schedule PEP has all the correct numbers and dates and such. That'll teach me to send these emails while half-asleep, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri May 6 12:08:31 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 6 May 2016 18:08:31 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20160506160831.0491F567BA@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-04-29 - 2016-05-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5504 (+29) closed 33223 (+56) total 38727 (+85) Open issues with patches: 2388 Issues opened (58) ================== #24225: Idlelib: changing file names http://bugs.python.org/issue24225 reopened by serhiy.storchaka #26156: Bad name into power operator syntax http://bugs.python.org/issue26156 reopened by r.david.murray #26163: FAIL: test_hash_effectiveness (test.test_set.TestFrozenSet) http://bugs.python.org/issue26163 reopened by berker.peksag #26888: Multiple memory leaks after raw Py_Initialize and Py_Finalize. http://bugs.python.org/issue26888 opened by Aleksander Gajewski #26889: Improve Doc/library/xmlrpc.client.rst http://bugs.python.org/issue26889 opened by serhiy.storchaka #26890: inspect.getsource gets source copy on disk even when module ha http://bugs.python.org/issue26890 opened by sebastien.bourdeauducq #26891: CPython doesn't work when you disable refcounting http://bugs.python.org/issue26891 opened by larry #26892: debuglevel not honored in urllib http://bugs.python.org/issue26892 opened by Chi Hsuan Yen #26894: Readline not aborting line edition on sigint http://bugs.python.org/issue26894 opened by memeplex #26896: mix-up with the terms 'importer', 'finder', 'loader' in the im http://bugs.python.org/issue26896 opened by Oren Milman #26897: Clarify Popen stdin, stdout, stderr http://bugs.python.org/issue26897 opened by Yclept.Nemo #26899: struct.pack_into(), struct.unpack_from() don't document suppor http://bugs.python.org/issue26899 opened by pfalcon #26900: Exclude the private API from the stable API http://bugs.python.org/issue26900 opened by serhiy.storchaka #26901: Argument Clinic test is broken http://bugs.python.org/issue26901 opened by serhiy.storchaka #26903: ProcessPoolExecutor(max_workers=64) crashes on Windows http://bugs.python.org/issue26903 opened by diogocp #26904: Difflib quick_ratio() could use Counter() http://bugs.python.org/issue26904 opened by Michael Cuthbert #26906: format(object.__reduce__) fails intermittently http://bugs.python.org/issue26906 opened by ztane #26907: Add missing getsockopt constants http://bugs.python.org/issue26907 opened by christian.heimes #26909: Serious performance loss (10 times) when NOT using .drain() http://bugs.python.org/issue26909 opened by mmarkk #26911: lib2to3/tests/pytree_idempotency.py has a broken import http://bugs.python.org/issue26911 opened by imz #26912: test/test_email/torture_test.py (and test_asian_codecs.py) has http://bugs.python.org/issue26912 opened by imz #26917: unicodedata.normalize(): bug in Hangul Composition http://bugs.python.org/issue26917 opened by arigo #26919: android: test_cmd_line fails http://bugs.python.org/issue26919 opened by xdegaye #26920: android: test_sys fails http://bugs.python.org/issue26920 opened by xdegaye #26923: asyncio.gather drops cancellation http://bugs.python.org/issue26923 opened by JohannesEbke #26924: android: test_concurrent_futures fails http://bugs.python.org/issue26924 opened by xdegaye #26925: android: test_multiprocessing_main_handling fails http://bugs.python.org/issue26925 opened by xdegaye #26926: Large files are not supported on Android http://bugs.python.org/issue26926 opened by xdegaye #26928: android: test_site fails http://bugs.python.org/issue26928 opened by xdegaye #26929: android: test_strptime fails http://bugs.python.org/issue26929 opened by xdegaye #26930: Upgrade installers to OpenSSL 1.0.2h http://bugs.python.org/issue26930 opened by alex #26931: android: test_distutils fails http://bugs.python.org/issue26931 opened by xdegaye #26934: android: test_faulthandler fails http://bugs.python.org/issue26934 opened by xdegaye #26935: android: test_os fails http://bugs.python.org/issue26935 opened by xdegaye #26936: android: test_socket fails http://bugs.python.org/issue26936 opened by xdegaye #26937: android: test_tarfile fails http://bugs.python.org/issue26937 opened by xdegaye #26938: android: test_concurrent_futures hangs on armv7 http://bugs.python.org/issue26938 opened by xdegaye #26939: android: test_functools hangs on armv7 http://bugs.python.org/issue26939 opened by xdegaye #26940: android: test_importlib hangs on armv7 http://bugs.python.org/issue26940 opened by xdegaye #26941: android: test_threading hangs on armv7 http://bugs.python.org/issue26941 opened by xdegaye #26942: android: test_ctypes crashes on armv7 http://bugs.python.org/issue26942 opened by xdegaye #26944: android: test_posix fails http://bugs.python.org/issue26944 opened by xdegaye #26945: difflib.HtmlDiff().make_file() treat few change as whole line http://bugs.python.org/issue26945 opened by Fairuz Zack #26947: Documentation improvement needed http://bugs.python.org/issue26947 opened by mPython #26948: Simplify PyImport_ImportModuleLevelObject: avoid temporary tup http://bugs.python.org/issue26948 opened by haypo #26949: IDLE restarts when one debugs code raising SystemExit http://bugs.python.org/issue26949 opened by ppperry #26951: Unintuitive error when using generator expression in class pro http://bugs.python.org/issue26951 opened by corey #26952: argparse help formatter crashes http://bugs.python.org/issue26952 opened by Endre #26954: Add Guido's rejection notice to the "with" FAQ http://bugs.python.org/issue26954 opened by steven.daprano #26955: Implement equivalent to `pip.locations.distutils_scheme` in di http://bugs.python.org/issue26955 opened by sylvain.corlay #26959: pickle: respect dispatch for functions again http://bugs.python.org/issue26959 opened by torkve #26960: urllib hangs when closing connection http://bugs.python.org/issue26960 opened by Chris Beaumont #26961: Provide an API to help debug super().__getattribute__ results http://bugs.python.org/issue26961 opened by ncoghlan #26964: Incorrect documentation for `-u`-flag http://bugs.python.org/issue26964 opened by olavmrk #26967: argparse: allow_abbrev=False stops -vv from working http://bugs.python.org/issue26967 opened by meinersbur #26968: glob.glob incorrect results under windows when pathname exists http://bugs.python.org/issue26968 opened by kwarzecha7 #26969: ascynio should provide a policy to address pass-loop-everywher http://bugs.python.org/issue26969 opened by Ilya.Kulakov #26970: Replace OpenSSL's CPRNG with system entropy source http://bugs.python.org/issue26970 opened by christian.heimes Most recent 15 issues with no replies (15) ========================================== #26964: Incorrect documentation for `-u`-flag http://bugs.python.org/issue26964 #26960: urllib hangs when closing connection http://bugs.python.org/issue26960 #26955: Implement equivalent to `pip.locations.distutils_scheme` in di http://bugs.python.org/issue26955 #26954: Add Guido's rejection notice to the "with" FAQ http://bugs.python.org/issue26954 #26942: android: test_ctypes crashes on armv7 http://bugs.python.org/issue26942 #26937: android: test_tarfile fails http://bugs.python.org/issue26937 #26936: android: test_socket fails http://bugs.python.org/issue26936 #26935: android: test_os fails http://bugs.python.org/issue26935 #26934: android: test_faulthandler fails http://bugs.python.org/issue26934 #26930: Upgrade installers to OpenSSL 1.0.2h http://bugs.python.org/issue26930 #26920: android: test_sys fails http://bugs.python.org/issue26920 #26909: Serious performance loss (10 times) when NOT using .drain() http://bugs.python.org/issue26909 #26907: Add missing getsockopt constants http://bugs.python.org/issue26907 #26903: ProcessPoolExecutor(max_workers=64) crashes on Windows http://bugs.python.org/issue26903 #26899: struct.pack_into(), struct.unpack_from() don't document suppor http://bugs.python.org/issue26899 Most recent 15 issues waiting for review (15) ============================================= #26970: Replace OpenSSL's CPRNG with system entropy source http://bugs.python.org/issue26970 #26964: Incorrect documentation for `-u`-flag http://bugs.python.org/issue26964 #26960: urllib hangs when closing connection http://bugs.python.org/issue26960 #26959: pickle: respect dispatch for functions again http://bugs.python.org/issue26959 #26948: Simplify PyImport_ImportModuleLevelObject: avoid temporary tup http://bugs.python.org/issue26948 #26939: android: test_functools hangs on armv7 http://bugs.python.org/issue26939 #26931: android: test_distutils fails http://bugs.python.org/issue26931 #26926: Large files are not supported on Android http://bugs.python.org/issue26926 #26924: android: test_concurrent_futures fails http://bugs.python.org/issue26924 #26917: unicodedata.normalize(): bug in Hangul Composition http://bugs.python.org/issue26917 #26906: format(object.__reduce__) fails intermittently http://bugs.python.org/issue26906 #26904: Difflib quick_ratio() could use Counter() http://bugs.python.org/issue26904 #26900: Exclude the private API from the stable API http://bugs.python.org/issue26900 #26896: mix-up with the terms 'importer', 'finder', 'loader' in the im http://bugs.python.org/issue26896 #26892: debuglevel not honored in urllib http://bugs.python.org/issue26892 Top 10 most discussed issues (10) ================================= #26156: Bad name into power operator syntax http://bugs.python.org/issue26156 12 msgs #26906: format(object.__reduce__) fails intermittently http://bugs.python.org/issue26906 11 msgs #26219: implement per-opcode cache in ceval http://bugs.python.org/issue26219 10 msgs #26929: android: test_strptime fails http://bugs.python.org/issue26929 9 msgs #26939: android: test_functools hangs on armv7 http://bugs.python.org/issue26939 9 msgs #26858: android: setting SO_REUSEPORT fails http://bugs.python.org/issue26858 7 msgs #26926: Large files are not supported on Android http://bugs.python.org/issue26926 7 msgs #26765: Factor out common bytes and bytearray implementation http://bugs.python.org/issue26765 6 msgs #26814: [WIP] Add a new _PyObject_FastCall() function which avoids the http://bugs.python.org/issue26814 6 msgs #26917: unicodedata.normalize(): bug in Hangul Composition http://bugs.python.org/issue26917 6 msgs Issues closed (58) ================== #10435: Document unicode C-API in reST http://bugs.python.org/issue10435 closed by berker.peksag #12135: The spawn function should return stderr. http://bugs.python.org/issue12135 closed by berker.peksag #13173: Default values for string.Template http://bugs.python.org/issue13173 closed by serhiy.storchaka #13436: compile() doesn't work on ImportFrom with level=None http://bugs.python.org/issue13436 closed by berker.peksag #14882: Link/Compile Error on Sun Sparc Solaris10 with gcc3.4.3----pyt http://bugs.python.org/issue14882 closed by berker.peksag #18688: Document undocumented Unicode object API http://bugs.python.org/issue18688 closed by berker.peksag #18841: math.isfinite fails with Decimal sNAN http://bugs.python.org/issue18841 closed by berker.peksag #18916: Various out-of-date Lock text in 3.2+ http://bugs.python.org/issue18916 closed by berker.peksag #18956: Document useful functions in ???pydoc??? module http://bugs.python.org/issue18956 closed by berker.peksag #20120: Percent-signs (%) in .pypirc should not be interpolated http://bugs.python.org/issue20120 closed by lukasz.langa #20554: Use specific asserts in optparse test http://bugs.python.org/issue20554 closed by serhiy.storchaka #20971: HTML output of difflib http://bugs.python.org/issue20971 closed by berker.peksag #21367: multiprocessing.JoinableQueue requires new kwarg http://bugs.python.org/issue21367 closed by berker.peksag #21746: urlparse.BaseResult no longer exists http://bugs.python.org/issue21746 closed by berker.peksag #23960: PyErr_SetImportError doesn't clean up on some errors http://bugs.python.org/issue23960 closed by berker.peksag #24114: ctypes.utils uninitialized variable 'paths' http://bugs.python.org/issue24114 closed by meador.inge #24950: FAIL: test_expanduser when $HOME=/ http://bugs.python.org/issue24950 closed by serhiy.storchaka #25177: OverflowError in statistics.mean when summing large floats http://bugs.python.org/issue25177 closed by berker.peksag #25757: Subclasses of property lose docstring http://bugs.python.org/issue25757 closed by berker.peksag #25974: Fix statistics.py after the Decimal.as_integer_ratio() change http://bugs.python.org/issue25974 closed by berker.peksag #26002: make statistics.median_grouped more efficient http://bugs.python.org/issue26002 closed by berker.peksag #26710: ConfigParser: Values in DEFAULT section override defaults pass http://bugs.python.org/issue26710 closed by Marc.Abramowitz #26711: Fix comparison of plistlib.Data http://bugs.python.org/issue26711 closed by serhiy.storchaka #26756: fileinput handling of unicode errors from standard input http://bugs.python.org/issue26756 closed by serhiy.storchaka #26811: segfault due to null pointer in tuple http://bugs.python.org/issue26811 closed by serhiy.storchaka #26849: android does not support versioning in SONAME http://bugs.python.org/issue26849 closed by xdegaye #26864: urllib.request no_proxy check differs from curl http://bugs.python.org/issue26864 closed by martin.panter #26873: xmlrpclib raises when trying to convert an int to string when http://bugs.python.org/issue26873 closed by serhiy.storchaka #26886: Cross-compiling python:3.5.x fails with "Parser/pgen: Parser/ http://bugs.python.org/issue26886 closed by Peter L2 #26887: Erratum in https://docs.python.org/2.6/library/multiprocessing http://bugs.python.org/issue26887 closed by zach.ware #26893: ValueError exception raised when using IntEnum with an attribu http://bugs.python.org/issue26893 closed by berker.peksag #26895: regex matching on bytes considers zero byte as end http://bugs.python.org/issue26895 closed by serhiy.storchaka #26898: Typo in the documentation of math.isclose() http://bugs.python.org/issue26898 closed by berker.peksag #26902: Argument Clinic incorrectly works with custom converter and re http://bugs.python.org/issue26902 closed by larry #26905: from multiprocessing.pool import Pool bug on linux http://bugs.python.org/issue26905 closed by SilentGhost #26908: "1and 0" evaluated a zero, instead of SyntaxError http://bugs.python.org/issue26908 closed by berker.peksag #26910: dictionary literal should not allow duplicate keys http://bugs.python.org/issue26910 closed by r.david.murray #26913: statistics.mean of bools http://bugs.python.org/issue26913 closed by r.david.murray #26914: Fix formatting of lists in PEP 420 http://bugs.python.org/issue26914 closed by georg.brandl #26915: Test identity first in membership operation of ItemsView, Valu http://bugs.python.org/issue26915 closed by rhettinger #26916: Word duplications in PEPs http://bugs.python.org/issue26916 closed by serhiy.storchaka #26918: android: test_pipes fails http://bugs.python.org/issue26918 closed by serhiy.storchaka #26921: Incorrect usage of a/an articles in PEPs http://bugs.python.org/issue26921 closed by serhiy.storchaka #26922: build from fresh checkout fails http://bugs.python.org/issue26922 closed by pitrou #26927: android: test_mmap fails http://bugs.python.org/issue26927 closed by serhiy.storchaka #26932: RTLD_* macros are not defined on Android http://bugs.python.org/issue26932 closed by serhiy.storchaka #26933: android: test_posixpath fails http://bugs.python.org/issue26933 closed by serhiy.storchaka #26943: datetime.datetime.strptime() raises an exception http://bugs.python.org/issue26943 closed by SilentGhost #26946: os.path.realpath.__name__ is 'abspath' http://bugs.python.org/issue26946 closed by brett.cannon #26950: Typo in unittest.mock > Autospeccing documentation http://bugs.python.org/issue26950 closed by berker.peksag #26953: Failed building wheel for greenlet http://bugs.python.org/issue26953 closed by berker.peksag #26956: About Idle-x version not updated for 1.13 http://bugs.python.org/issue26956 closed by terry.reedy #26957: Docs: remove duplicate "the" in description of `%z` in datetim http://bugs.python.org/issue26957 closed by berker.peksag #26958: Suggestion to imporve queue.full reliability http://bugs.python.org/issue26958 closed by rhettinger #26962: curses documenation- keys for Page Up and Page Down are revers http://bugs.python.org/issue26962 closed by berker.peksag #26963: compile() expected string without null bytes http://bugs.python.org/issue26963 closed by serhiy.storchaka #26965: spam http://bugs.python.org/issue26965 closed by ned.deily #26966: Spam http://bugs.python.org/issue26966 closed by berker.peksag From amk at amk.ca Fri May 6 13:11:34 2016 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 6 May 2016 13:11:34 -0400 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: <9F53FF72-6CCB-482A-B4EB-E8165E404362@stufft.io> References: <20160505234230.GB14936@k3> <20160506003527.GA30649@k3> <9F53FF72-6CCB-482A-B4EB-E8165E404362@stufft.io> Message-ID: <20160506171134.GA841@DATLANDREWK.local> On Thu, May 05, 2016 at 10:31:48PM -0400, Donald Stufft wrote: > I don't believe we've ever told someone that something can't happen because of > Warehouse, only that *I* won't implement something until after Warehouse. That > often times means that something won't happen until after Warehouse because of > the severe shortage of people with enough time and motivation to work on this > stuff but if someone did step up more things would get done. Could the PSF help with this, whether by paying for Donald's (or a consultant's) time to whatever do final implementation, polishing, testing, or sysadmin work is required? --amk From josephnunn at gmail.com Fri May 6 13:43:12 2016 From: josephnunn at gmail.com (Joseph Lee Nunn III) Date: Fri, 6 May 2016 10:43:12 -0700 Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems In-Reply-To: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org> References: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org> Message-ID: Then I have found the right list to post on, for my problem is a bug in the Python 3.5.1 distribution. Albeit one which will only manifest in certain situations. The presence of numerous `-isysroot /Developer/SDKs/MacOSX10.6.sdk` directives in the following file /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_sysconfigdata.py Will cause errors for people who have kept around the old /Developer directory tree, and old SDKs like 10.6, for building backwards compatible old code. Newer versions of the SDKs are now kept inside the Xcode application itself. And then they will only have a problem when trying to install a package with C extensions, which in my case was Persistence, a dependency of ZODB. And even more so, ONLY when trying to pip install said package into a virtual environment created by 3.5?s pyvenv command, if installing the package into the system wide Python there will be no error. Such is this corner-case hell I have found myself, until I decided to dump my older /Developer directory tree and SDKs, as those were for a project long defunct. You see the -isysroot flag has the unusual behavior where instead of reporting an error when it cannot find the newly specified system root it simply has no effect. So if you don?t have the SDK specified in the location specified then the flag does nothing, as I assume is the case for 99% of people. I understand the need to make Python 3.5 backwards compatible, but doing it this way would appear to have the side effect of breaking the ability to install packages using C extensions in virtual environments, or at least it did for me. Joseph Nunn josephnunn at gmail.com joseph.nunn at uci.edu > On May 06 , 2016, at 4:20 AM, Ned Deily wrote: > > > On May 6, 2016, at 05:42, Joseph Nunn wrote: >> Hello Everyone, my first post. I would also like to subscribe but >> haven't seen how yet. >> >> So I've an issue with how the 3.5.1 OSX >> package installs on OSX. The problem was first made apparent >> when trying to create a virtual environment and `pip install ZODB`, >> which failed as clang and the linker could not find either the >> include or lib directories for a dependency. By adding a -v option >> to the compiler options I was able to see in the compiler flags >> the following: >> >> -isysroot /Developer/SDKs/MacOSX10.6.sdk >> >> Which was not present in the compiler flags of other people >> who I was talking with. [...] > > Hi Joseph! > > Welcome! Note this list is for the development of Python itself, not for usage or installation issues. The Python Developer's Guide (https://docs.python.org/devguide/) including its FAQ gives information about mailing lists and other support forums to get help with general Python issues. That said, without more information, it would be difficult to know exactly what problem you are running into but, since you mention missing headers, my guess is that you need to install Apple's Command Line Tools as described here: > > https://docs.python.org/devguide/setup.html#build-dependencies > > Installing Xcode by itself is not sufficient. The sysroot and flag values you note in _sysconfigdata.py are normal and there for compatibility with installing on older versions of OS X; they should not cause a problem on newer releases. For what it's worth, I was able to install ZODB with the current python.org 3.5.1 on OS X 10.11.4 using either venv or virtualenv. > > Good luck! > > --Ned > > P.S. Mailing list info is available here: https://mail.python.org. The bug tracker is here: https://bugs.python.org > > -- > Ned Deily > nad at python.org -- [] > From donald at stufft.io Fri May 6 15:31:40 2016 From: donald at stufft.io (Donald Stufft) Date: Fri, 6 May 2016 15:31:40 -0400 Subject: [Python-Dev] Yearly PyPI breakage In-Reply-To: <20160506171134.GA841@DATLANDREWK.local> References: <20160505234230.GB14936@k3> <20160506003527.GA30649@k3> <9F53FF72-6CCB-482A-B4EB-E8165E404362@stufft.io> <20160506171134.GA841@DATLANDREWK.local> Message-ID: <2D6D5D97-3695-450D-9C3E-60E8EFFE7D5B@stufft.io> > On May 6, 2016, at 1:11 PM, A.M. Kuchling wrote: > > On Thu, May 05, 2016 at 10:31:48PM -0400, Donald Stufft wrote: >> I don't believe we've ever told someone that something can't happen because of >> Warehouse, only that *I* won't implement something until after Warehouse. That >> often times means that something won't happen until after Warehouse because of >> the severe shortage of people with enough time and motivation to work on this >> stuff but if someone did step up more things would get done. > > Could the PSF help with this, whether by paying for Donald's (or a > consultant's) time to whatever do final implementation, polishing, > testing, or sysadmin work is required? Personally, my time is already paid for by HPE to work on this stuff but I?m only one person. The PSF could pay for others to help with that though. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From camillamon at gmail.com Fri May 6 17:14:32 2016 From: camillamon at gmail.com (Camilla) Date: Fri, 6 May 2016 22:14:32 +0100 Subject: [Python-Dev] Anyone want to lead the sprints at PyCon US 2016? In-Reply-To: References: Message-ID: I was thinking about holding a Patch Review Party/Sprint, which would provide people unfamiliar with the Python dev process a way to contribute to the project and get familiar with running tests, applying patches and so forth. I have a list of easy-ish patches that I wanted to take a look at and I could expand that and use those as a starting for people who don't have any particular bug tracker issues in mind. I'm not a patch review guru by any means, though. Also not sure if this is a good idea or if this is just late night caffeine talking. 2016-04-28 20:07 GMT+01:00 Brett Cannon : > No one stepped forward to lead the sprints this year, so I will put myself > as the sprint leader and lean on everyone else who appears to help. :) > > > On Tue, 5 Apr 2016 at 09:36 Brett Cannon wrote: > >> The call has started to go out for sprint groups to list themselves >> online. Anyone want to specifically lead the core sprint this year? If no >> one specifically does then I will sign us up and do my usual thing of >> pointing people at the devguide and encourage people to ask questions but >> not do a lot of hand-holding (I'm expecting to be busy either working on >> GitHub migration stuff or doing other things that I have been neglecting >> due to my GitHub migration work). >> >> ---------- Forwarded message --------- >> From: Ewa Jodlowska >> Date: Mon, 4 Apr 2016 at 07:14 >> Subject: [PSF-Community] Sprinting at PyCon US 2016 >> To: >> >> >> Are you coming to PyCon US? Have you thought about sprinting? >> >> The coding Sprints are the hidden gem of PyCon, up to 4 days (June 2-5) >> of coding with many Python projects and their maintainers. And if you're >> coming to PyCon, taking part in the Sprints is easy! >> >> You don?t need to change your registration* to join the Sprints. There?s >> no additional registration fee, and you even get lunch. You do need to >> cover the additional lodging and other meals, but that?s it. If you?ve >> booked a room through the PyCon registration system, you'll need to contact >> the registration team at pycon2016 at cteusa.com as soon as possible to >> request the extra nights. The sprinting itself (along with lunch every day) >> is free, so your only expenses are your room and other meals. >> >> If you're interested in what projects will be sprinting, just keep an eye >> on the sprints page on the PyCon web site at >> https://us.pycon.org/2016/community/sprints/ Be sure to check back, as >> groups are being added all the time. >> >> If you haven't sprinted before, or if you just need to brush up on >> sprinting tools and techniques, there will again be an 'Intro to Sprinting' >> session the evening of June 1, lead by Shauna Gordon-McKeon and other >> members of Python community. To grab a free ticket for this session, just >> visit >> https://www.eventbrite.com/e/introduction-to-open-source-the-pycon-sprints-tickets-22435151141 >> . >> >> *Please note that conference registration is sold out, but you do not >> need a conference registration to come to the Sprints. >> >> _______________________________________________ >> PSF-Community mailing list >> PSF-Community at python.org >> https://mail.python.org/mailman/listinfo/psf-community >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/camillamon%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 6 17:38:22 2016 From: brett at python.org (Brett Cannon) Date: Fri, 06 May 2016 21:38:22 +0000 Subject: [Python-Dev] Anyone want to lead the sprints at PyCon US 2016? In-Reply-To: References: Message-ID: On Fri, 6 May 2016 at 14:14 Camilla wrote: > I was thinking about holding a Patch Review Party/Sprint, which would > provide people unfamiliar with the Python dev process a way to contribute > to the project and get familiar with running tests, applying patches and so > forth. I have a list of easy-ish patches that I wanted to take a look at > and I could expand that and use those as a starting for people who don't > have any particular bug tracker issues in mind. > I'm not a patch review guru by any means, though. Also not sure if this is > a good idea or if this is just late night caffeine talking. > I have absolutely no problem if you want to pitch this idea to new contributors who show up at the sprints! -Brett > > > 2016-04-28 20:07 GMT+01:00 Brett Cannon : > >> No one stepped forward to lead the sprints this year, so I will put >> myself as the sprint leader and lean on everyone else who appears to help. >> :) >> >> >> On Tue, 5 Apr 2016 at 09:36 Brett Cannon wrote: >> >>> The call has started to go out for sprint groups to list themselves >>> online. Anyone want to specifically lead the core sprint this year? If no >>> one specifically does then I will sign us up and do my usual thing of >>> pointing people at the devguide and encourage people to ask questions but >>> not do a lot of hand-holding (I'm expecting to be busy either working on >>> GitHub migration stuff or doing other things that I have been neglecting >>> due to my GitHub migration work). >>> >>> ---------- Forwarded message --------- >>> From: Ewa Jodlowska >>> Date: Mon, 4 Apr 2016 at 07:14 >>> Subject: [PSF-Community] Sprinting at PyCon US 2016 >>> To: >>> >>> >>> Are you coming to PyCon US? Have you thought about sprinting? >>> >>> The coding Sprints are the hidden gem of PyCon, up to 4 days (June 2-5) >>> of coding with many Python projects and their maintainers. And if you're >>> coming to PyCon, taking part in the Sprints is easy! >>> >>> You don?t need to change your registration* to join the Sprints. There?s >>> no additional registration fee, and you even get lunch. You do need to >>> cover the additional lodging and other meals, but that?s it. If you?ve >>> booked a room through the PyCon registration system, you'll need to contact >>> the registration team at pycon2016 at cteusa.com as soon as possible to >>> request the extra nights. The sprinting itself (along with lunch every day) >>> is free, so your only expenses are your room and other meals. >>> >>> If you're interested in what projects will be sprinting, just keep an >>> eye on the sprints page on the PyCon web site at >>> https://us.pycon.org/2016/community/sprints/ Be sure to check back, as >>> groups are being added all the time. >>> >>> If you haven't sprinted before, or if you just need to brush up on >>> sprinting tools and techniques, there will again be an 'Intro to Sprinting' >>> session the evening of June 1, lead by Shauna Gordon-McKeon and other >>> members of Python community. To grab a free ticket for this session, just >>> visit >>> https://www.eventbrite.com/e/introduction-to-open-source-the-pycon-sprints-tickets-22435151141 >>> . >>> >>> *Please note that conference registration is sold out, but you do not >>> need a conference registration to come to the Sprints. >>> >>> _______________________________________________ >>> PSF-Community mailing list >>> PSF-Community at python.org >>> https://mail.python.org/mailman/listinfo/psf-community >>> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/camillamon%40gmail.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From willingc at willingconsulting.com Fri May 6 17:49:24 2016 From: willingc at willingconsulting.com (Carol Willing) Date: Fri, 06 May 2016 14:49:24 -0700 Subject: [Python-Dev] Anyone want to lead the sprints at PyCon US 2016? In-Reply-To: References: Message-ID: On 6 May 2016, at 14:38, Brett Cannon wrote: > On Fri, 6 May 2016 at 14:14 Camilla wrote: > >> I was thinking about holding a Patch Review Party/Sprint, which would >> provide people unfamiliar with the Python dev process a way to >> contribute >> to the project and get familiar with running tests, applying patches >> and so >> forth. I have a list of easy-ish patches that I wanted to take a look >> at >> and I could expand that and use those as a starting for people who >> don't >> have any particular bug tracker issues in mind. >> I'm not a patch review guru by any means, though. Also not sure if >> this is >> a good idea or if this is just late night caffeine talking. >> > > I have absolutely no problem if you want to pitch this idea to new > contributors who show up at the sprints! > > -Brett Camilla, I would be happy to support your effort. I find it a wonderful idea! Carol Carol Willing Research Software Engineer, Project Jupyter @ Cal Poly Director, Python Software Foundation >> >> >> 2016-04-28 20:07 GMT+01:00 Brett Cannon : >> >>> No one stepped forward to lead the sprints this year, so I will put >>> myself as the sprint leader and lean on everyone else who appears to >>> help. >>> :) >>> >>> >>> On Tue, 5 Apr 2016 at 09:36 Brett Cannon wrote: >>> >>>> The call has started to go out for sprint groups to list themselves >>>> online. Anyone want to specifically lead the core sprint this year? >>>> If no >>>> one specifically does then I will sign us up and do my usual thing >>>> of >>>> pointing people at the devguide and encourage people to ask >>>> questions but >>>> not do a lot of hand-holding (I'm expecting to be busy either >>>> working on >>>> GitHub migration stuff or doing other things that I have been >>>> neglecting >>>> due to my GitHub migration work). >>>> >>>> ---------- Forwarded message --------- >>>> From: Ewa Jodlowska >>>> Date: Mon, 4 Apr 2016 at 07:14 >>>> Subject: [PSF-Community] Sprinting at PyCon US 2016 >>>> To: >>>> >>>> >>>> Are you coming to PyCon US? Have you thought about sprinting? >>>> >>>> The coding Sprints are the hidden gem of PyCon, up to 4 days (June >>>> 2-5) >>>> of coding with many Python projects and their maintainers. And if >>>> you're >>>> coming to PyCon, taking part in the Sprints is easy! >>>> >>>> You don?t need to change your registration* to join the Sprints. >>>> There?s >>>> no additional registration fee, and you even get lunch. You do need >>>> to >>>> cover the additional lodging and other meals, but that?s it. If >>>> you?ve >>>> booked a room through the PyCon registration system, you'll need to >>>> contact >>>> the registration team at pycon2016 at cteusa.com as soon as possible >>>> to >>>> request the extra nights. The sprinting itself (along with lunch >>>> every day) >>>> is free, so your only expenses are your room and other meals. >>>> >>>> If you're interested in what projects will be sprinting, just keep >>>> an >>>> eye on the sprints page on the PyCon web site at >>>> https://us.pycon.org/2016/community/sprints/ Be sure to check back, >>>> as >>>> groups are being added all the time. >>>> >>>> If you haven't sprinted before, or if you just need to brush up on >>>> sprinting tools and techniques, there will again be an 'Intro to >>>> Sprinting' >>>> session the evening of June 1, lead by Shauna Gordon-McKeon and >>>> other >>>> members of Python community. To grab a free ticket for this >>>> session, just >>>> visit >>>> https://www.eventbrite.com/e/introduction-to-open-source-the-pycon-sprints-tickets-22435151141 >>>> . >>>> >>>> *Please note that conference registration is sold out, but you do >>>> not >>>> need a conference registration to come to the Sprints. >>>> >>>> _______________________________________________ >>>> PSF-Community mailing list >>>> PSF-Community at python.org >>>> https://mail.python.org/mailman/listinfo/psf-community >>>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/camillamon%40gmail.com >>> >>> >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/willingc%40willingconsulting.com From victor.stinner at gmail.com Fri May 6 17:58:52 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 6 May 2016 23:58:52 +0200 Subject: [Python-Dev] Wordcode: new regular bytecode using 16-bit units In-Reply-To: <1462472902.2822020.599321113.046AABAF@webmail.messagingengine.com> References: <1462472902.2822020.599321113.046AABAF@webmail.messagingengine.com> Message-ID: Oh nice. Did you see my recent "bytecode" project? http://bytecode.readthedocs.io/ Victor Le 5 mai 2016 8:30 PM, a ?crit : > Here is something I wrote because I was also unsatisfied with byteplay's > API: https://github.com/zachariahreed/byteasm. Maybe it's useful in a > discussion of "minimum viable" api for bytecode manipulation. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat May 7 18:39:13 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 8 May 2016 01:39:13 +0300 Subject: [Python-Dev] Return type of alternative constructors Message-ID: Some types have alternative constructors -- class methods used to create an instance of the class. For example: int.from_bytes(), float.fromhex(), dict.fromkeys(), Decimal.from_float(). But what should return these methods for subclasses? Should they return an instance of base class or an instance of subclass? Almost all alternative constructors return an instance of subclass (exceptions are new in 3.6 bytes.fromhex() and bytearray.fromhex() that return bare bytes and bytearray). But there is a problem, because this allows to break invariants provided by the main constructor. For example, there are only two instances of the bool class: False and True. But with the from_bytes() method inherited from int you can create new boolean values! >>> Confusion = bool.from_bytes(b'\2', 'big') >>> isinstance(Confusion, bool) True >>> Confusion == True False >>> bool(Confusion) True >>> Confusion False >>> not Confusion False bool is just the most impressive example, the same problem exists with IntEnum and other enums derived from float, Decimal, datetime. [1] The simplest solution is to return an instance of base class. But this can breaks a code, and for this case we should be use static method (like str.maketrans), not class method. Should alternative constructor call __new__ and __init__ methods? Thay can change signature in derived class. Should it complain if __new__ or __init__ were overridden? [1] http://bugs.python.org/issue23640 From ethan at stoneleaf.us Sat May 7 19:06:51 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 07 May 2016 16:06:51 -0700 Subject: [Python-Dev] Tracker Etiquette Message-ID: <572E750B.80706@stoneleaf.us> I just reviewed an issue on the tracker, and found that many, if not all, of my comments made at the time I completely disagree with (both now, and before -- I'm going to claim sleep deprivation). Obviously I should post a new comment explaining my thoughts about the issue, but what should I do with the old comments? Is it acceptable to delete them? -- ~Ethan~ From brett at python.org Sat May 7 19:10:43 2016 From: brett at python.org (Brett Cannon) Date: Sat, 07 May 2016 23:10:43 +0000 Subject: [Python-Dev] Tracker Etiquette In-Reply-To: <572E750B.80706@stoneleaf.us> References: <572E750B.80706@stoneleaf.us> Message-ID: I think if you leave a comment that to deleted your own comments and why you did it then it's fine. On Sat, May 7, 2016, 16:07 Ethan Furman wrote: > I just reviewed an issue on the tracker, and found that many, if not > all, of my comments made at the time I completely disagree with (both > now, and before -- I'm going to claim sleep deprivation). > > Obviously I should post a new comment explaining my thoughts about the > issue, but what should I do with the old comments? Is it acceptable to > delete them? > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat May 7 19:14:12 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 07 May 2016 16:14:12 -0700 Subject: [Python-Dev] Return type of alternative constructors In-Reply-To: References: Message-ID: <572E76C4.3020907@stoneleaf.us> On 05/07/2016 03:39 PM, Serhiy Storchaka wrote: > Some types have alternative constructors -- class methods used to create > an instance of the class. For example: int.from_bytes(), > float.fromhex(), dict.fromkeys(), Decimal.from_float(). > > But what should return these methods for subclasses? Should they return > an instance of base class or an instance of subclass? Almost all > alternative constructors return an instance of subclass (exceptions are > new in 3.6 bytes.fromhex() and bytearray.fromhex() that return bare > bytes and bytearray). But there is a problem, because this allows to > break invariants provided by the main constructor. Please ignore my comments in that issue. I actually prefer that class constructors go through the subclass' __new__ and __init__. Overriding parent class methods for the sole purpose of getting the subclass's type is quite irritating. -- ~Ethan~ From python at mrabarnett.plus.com Sat May 7 19:17:19 2016 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 8 May 2016 00:17:19 +0100 Subject: [Python-Dev] Tracker Etiquette In-Reply-To: <572E750B.80706@stoneleaf.us> References: <572E750B.80706@stoneleaf.us> Message-ID: <8d44f36f-e18d-5540-e6af-c13392bd7e44@mrabarnett.plus.com> On 2016-05-08 00:06, Ethan Furman wrote: > I just reviewed an issue on the tracker, and found that many, if not > all, of my comments made at the time I completely disagree with (both > now, and before -- I'm going to claim sleep deprivation). > > Obviously I should post a new comment explaining my thoughts about the > issue, but what should I do with the old comments? Is it acceptable to > delete them? > I think you shouldn't delete them. It would be better just to say that you've changed your mind and explain why. From senthil at uthcode.com Sat May 7 19:23:32 2016 From: senthil at uthcode.com (Senthil Kumaran) Date: Sat, 7 May 2016 16:23:32 -0700 Subject: [Python-Dev] Tracker Etiquette In-Reply-To: <8d44f36f-e18d-5540-e6af-c13392bd7e44@mrabarnett.plus.com> References: <572E750B.80706@stoneleaf.us> <8d44f36f-e18d-5540-e6af-c13392bd7e44@mrabarnett.plus.com> Message-ID: On Sat, May 7, 2016 at 4:17 PM, MRAB wrote: > I think you shouldn't delete them. It would be better just to say that > you've changed your mind and explain why. > I support this. Please leave your new comments correcting previous one and support your current stance. I think, it is alright to make revisions in comments. -- Senthil -------------- next part -------------- An HTML attachment was scrubbed... URL: From vadmium+py at gmail.com Sat May 7 19:57:05 2016 From: vadmium+py at gmail.com (Martin Panter) Date: Sat, 7 May 2016 23:57:05 +0000 Subject: [Python-Dev] Tracker Etiquette In-Reply-To: References: <572E750B.80706@stoneleaf.us> <8d44f36f-e18d-5540-e6af-c13392bd7e44@mrabarnett.plus.com> Message-ID: On 7 May 2016 at 23:23, Senthil Kumaran wrote: > > On Sat, May 7, 2016 at 4:17 PM, MRAB wrote: >> >> I think you shouldn't delete them. It would be better just to say that >> you've changed your mind and explain why. > > > I support this. Please leave your new comments correcting previous one and > support your current stance. I think, it is alright to make revisions in > comments. Taking a middle ground between this and what Brett said, I would say it is okay to delete stuff if it is fresh and hasn?t been replied to (but maybe leave a message if it is not obvious why you delete it). On the other hand, avoid deleting stuff if it would harm the overall thread. From chris.barker at noaa.gov Sun May 8 01:59:16 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 7 May 2016 22:59:16 -0700 Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems In-Reply-To: References: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org> Message-ID: On Fri, May 6, 2016 at 10:43 AM, Joseph Lee Nunn III wrote: > Then I have found the right list to post on, for my problem is a bug in > the Python 3.5.1 distribution. Albeit one which will only manifest in > certain situations. > > The presence of numerous `-isysroot /Developer/SDKs/MacOSX10.6.sdk` > directives in the following file > > > /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_sysconfigdata.py > As I understand it, that is all there so that when extensions are built, they are linked against that SDK -- so you can build extensions that can then be run on any system 10.6 and greater. > Will cause errors for people who have kept around the old /Developer > directory tree, and old SDKs like 10.6, for building backwards compatible > old code. what are the errors -- wouldn't the old SDK be found, and used, and that's exactly what's intended? > Newer versions of the SDKs are now kept inside the Xcode application > itself. And then they will only have a problem when trying to install a > package with C extensions, well, sure. of course this is an issue for building C extensions. > which in my case was Persistence, a dependency of ZODB. And even more so, > ONLY when trying to pip install said package into a virtual environment > created by 3.5?s pyvenv command, if installing the package into the system > wide Python there will be no error. > hmm, As I understand it the building process should be exactly the same whether you are on a virtualenv or not. but if so -- maybe the "solution" is to build a wheel outside virtualenv, and then install that wheel in the virtualenv. But I agree, it should work, of course. > You see the -isysroot flag has the unusual behavior where instead of > reporting an error when it cannot find the newly specified system root it > simply has no effect. So if you don?t have the SDK specified in the > location specified then the flag does nothing, as I assume is the case for > 99% of people. > Indeed, that's been working fine for me for a while. However it's this behavior that is actually the bug -- extensions are supposed to be build against the 10.6 SDK -- and that's not what's happening in this case > I understand the need to make Python 3.5 backwards compatible, but doing > it this way would appear to have the side effect of breaking the ability to > install packages using C extensions in virtual environments, or at least it > did for me. > It does sound like you've hit a bug, but I actually think you've misidentified the bug -- I'm trying to understand why it's not working in a virtualenv -- THAT's the bug. And I have no idea what to do about it -- thanks Apple for moving things -- that's a great way to keep backward compatibility! Ned: I'm trying to explore this a bit now (on a 10.9 system), so I followed the instructions in the Python developers guide, and did: xcode-select --install And was a bit surprised that It did a substantial download -- I thought I already had the XCode command line tools -- but whatever. After running that, I found I had the 10.9 and 10.10 SDKs (I think that installed the 10.10 one, I previously had only 10.9). And they are buried deep in XCode.app .... OK, but how would I go about getting the 10.6 SDK??? This SO post: http://stackoverflow.com/questions/11989007/where-to-get-macos-sdk-10-6-for-xcode indicates that it's a serious pain in the $%^% -- thank Apple! But, as far as I can tell, if you install an old XCode, you will get the older SDKs, and installed into the old location. (this is why I kept a 10.6 system around for a long time -- to build python extensions...) But what should we do? IF you are building extensions only for yourself, then I think whatever the heck SDK you have will work fine -- but how to build for distribution? .... Trying to decide if I want to go through the hassle of installing an old XCode to see if i can replicate the bug the OP is reporting.... -CHB > https://docs.python.org/devguide/setup.html#build-dependencies > > > > > Installing Xcode by itself is not sufficient. The sysroot and flag > values you note in _sysconfigdata.py are normal and there for compatibility > with installing on older versions of OS X; they should not cause a problem > on newer releases. For what it's worth, I was able to install ZODB with > the current python.org 3.5.1 on OS X 10.11.4 using either venv or > virtualenv. > > > > Good luck! > > > > --Ned > > > > P.S. Mailing list info is available here: https://mail.python.org. The > bug tracker is here: https://bugs.python.org > > > > -- > > Ned Deily > > nad at python.org -- [] > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sun May 8 02:52:48 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 7 May 2016 23:52:48 -0700 Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems In-Reply-To: References: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org> Message-ID: On Sat, May 7, 2016 at 10:59 PM, Chris Barker wrote: > > .... Trying to decide if I want to go through the hassle of installing an > old XCode to see if i can replicate the bug the OP is reporting.... > OK, I found this gitHub repo with all the old SDKs: https://github.com/phracker/MacOSX-SDKs I put a copy of the 10.6 SDk in it's old location: /Developer/SDKs/MacOSX10.6.sdk/ /before I did that, I did a pip install of zodb in a virtualenv, and all seemed to go fine. THe extensions rae linked against : $ otool -L cPersistence.cpython-35m-darwin.so cPersistence.cpython-35m-darwin.so: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1) where that comes from -- who knows?? then I tried again, after putting the sdk in place: [note: serious pain the $%^# to get pip to stop using it's cache! ] and everything seemed to build fine, and at least one simple test of zodb works fine. And this is how the extensions are linked: $ otool -L cPersistence.cpython-35m-darwin.so cPersistence.cpython-35m-darwin.so: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11) which is the same as above, but the "current version" is much older -- so does that mean it used the 10.6 SDK???? Anyway, I can't reproduce the OP's bug..... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun May 8 03:03:45 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 8 May 2016 00:03:45 -0700 Subject: [Python-Dev] Return type of alternative constructors In-Reply-To: <572E76C4.3020907@stoneleaf.us> References: <572E76C4.3020907@stoneleaf.us> Message-ID: IMO bool is a special case because it's meant to be a final class, and the implementation of int (which is in C and so can violate most rules) doesn't respect that. But in general I think the only reasonable approach is that a construction class method should return an instance of the subclass; these class methods have a signature that's constrained by their signature in the base class. OTOH operators like __add__ cannot be expected to return an instance of the subclass, because these typically construct an instance using __new__/__init__, whose signatures are *not* constrained by the base class. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun May 8 06:56:34 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 8 May 2016 20:56:34 +1000 Subject: [Python-Dev] Tracker Etiquette In-Reply-To: <572E750B.80706@stoneleaf.us> References: <572E750B.80706@stoneleaf.us> Message-ID: <20160508105633.GW12028@ando.pearwood.info> On Sat, May 07, 2016 at 04:06:51PM -0700, Ethan Furman wrote: > I just reviewed an issue on the tracker, and found that many, if not > all, of my comments made at the time I completely disagree with (both > now, and before -- I'm going to claim sleep deprivation). > > Obviously I should post a new comment explaining my thoughts about the > issue, but what should I do with the old comments? Is it acceptable to > delete them? My preference would be not to delete them. Certainly don't delete them if others have explicitly replied to them. Even if you no longer agree with the conclusion, there may be something of value in the comments. -- Steve From ncoghlan at gmail.com Sun May 8 07:49:20 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 May 2016 21:49:20 +1000 Subject: [Python-Dev] Return type of alternative constructors In-Reply-To: References: Message-ID: On 8 May 2016 at 08:39, Serhiy Storchaka wrote: > Should alternative constructor call __new__ and __init__ methods? Thay can > change signature in derived class. I think this is typically the way to go (although, depending on the specific type, the unpickling related methods may be a more appropriate way for the alternate constructor to populate the instance state) > Should it complain if __new__ or __init__ were overridden? If there are alternate constructors that depend on either the signature of __new__/__init__, unpickling support, or some other mechanism for creating new instances, this should be mentioned in the class documentation as a constraint on subclasses - if subclasses don't want to meet the constraint, they'll need to override the affected alternate constructors. Cheers, Nick. P.S. The potential complexity of that is one of the reasons the design philosophy of "prefer composition to inheritance" has emerged - subclassing is a powerful tool, but it does mean you often end up needing to care about more interactions between the subclass and the base class than you really wanted to. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun May 8 07:58:02 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 May 2016 21:58:02 +1000 Subject: [Python-Dev] Tracker Etiquette In-Reply-To: <20160508105633.GW12028@ando.pearwood.info> References: <572E750B.80706@stoneleaf.us> <20160508105633.GW12028@ando.pearwood.info> Message-ID: On 8 May 2016 at 20:56, Steven D'Aprano wrote: > On Sat, May 07, 2016 at 04:06:51PM -0700, Ethan Furman wrote: >> I just reviewed an issue on the tracker, and found that many, if not >> all, of my comments made at the time I completely disagree with (both >> now, and before -- I'm going to claim sleep deprivation). >> >> Obviously I should post a new comment explaining my thoughts about the >> issue, but what should I do with the old comments? Is it acceptable to >> delete them? > > My preference would be not to delete them. Certainly don't delete them > if others have explicitly replied to them. Even if you no longer agree > with the conclusion, there may be something of value in the comments. I generally only delete mine if I've posted to the wrong issue (which happens sometimes if I have multiple issues open and lose track of which tab is which). For other discussions, I figure realising I'd made a mistake on my own isn't that different from someone else replying to point out the error - I'm sure there a few tracker comments floating around from Nick-of-several-hours-later suggesting that Nick-of-several-hours-earlier probably shouldn't be commenting on tracker issues when he's in dire need of sleep ;) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Sun May 8 11:51:22 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 May 2016 08:51:22 -0700 Subject: [Python-Dev] Tracker Etiquette In-Reply-To: <572E750B.80706@stoneleaf.us> References: <572E750B.80706@stoneleaf.us> Message-ID: <572F607A.4070904@stoneleaf.us> On 05/07/2016 04:06 PM, Ethan Furman wrote: > I just reviewed an issue on the tracker, and found that many, if not > all, of my comments made at the time I completely disagree with (both > now, and before -- I'm going to claim sleep deprivation). > > Obviously I should post a new comment explaining my thoughts about the > issue, but what should I do with the old comments? Is it acceptable to > delete them? Thank you all for your comments. Old ones kept, new one posted. -- ~Ethan~ From ethan at stoneleaf.us Sun May 8 16:10:05 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 May 2016 13:10:05 -0700 Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2? Message-ID: <572F9D1D.8090309@stoneleaf.us> Currently, the Enum creation process ignores __dunder__ attributes, and blocks all _sunder_ attributes. Because of this, the enum34 backport used __order__ instead of _order_ to provide a mechanism for ordering the enum members (which I never really liked). However, I've been working on my aenum [1] package, which uses several other _sunder_ attributes (for python2 compatibility) so I enabled _order_ instead and promote that spelling in the docs. Unlike the other _sunder_ attributes, _order_ has no meaningful affect in Python 3 so I'd like to change the stdlib Enum to allow it (and either ignore completely, or check it is the same as the definition order). My question is: Should I put this change in 3.5.2? - Yes means 3.5.2+ will work with _order_, 3.4, 3.5.0, and 3.5.1 will not; - No means 3.4 and all of 3.5 will not. -- ~Ethan~ [1] https://pypi.python.org/pypi/aenum From brett at python.org Sun May 8 16:41:04 2016 From: brett at python.org (Brett Cannon) Date: Sun, 08 May 2016 20:41:04 +0000 Subject: [Python-Dev] devinabox has moved to GitHub Message-ID: https://github.com/python/devinabox The single issue for devinabox has moved to its own issue tracker, so there's no need to worry about those issues cluttering b.p.o in the future. I have made the Python core team I created on GitHub last week have write privileges and Nick and I as admins on the repository. I have also turned on the CLA bot for the repository (FYI https://github.com/python/the-knights-who-say-ni has that bot's code). I've asked Georg, Antoine, and Benjamin to tell me what I need to do to shut off -- either by making it read-only or just deleting -- hg.python.org/devinabox. Now that the migration has seriously begun, the next repos will be the peps and devguide (slightly more complicated thanks to needing to update commands for building their online versions). There's also the benchmarks repo, but that might not get migrated if we start from scratch (see the speed@ ML about that). As for cpython, I've been asked to talk about it at the language summit where I will start a conversation about what the minimal feature set will need to be to migrate. Once we have settled on what has to be in place to migrate the cpython repo then we can start working on that TODO list. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Sun May 8 18:29:20 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun, 8 May 2016 15:29:20 -0700 Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2? In-Reply-To: <572F9D1D.8090309@stoneleaf.us> References: <572F9D1D.8090309@stoneleaf.us> Message-ID: If enum were provisional it would be okay, but since it isn't, I think this change can't go into 3.5.2. Think if this: could any code that works in 3.5.1 be broken by the change? --Guido (mobile) On May 8, 2016 1:11 PM, "Ethan Furman" wrote: > Currently, the Enum creation process ignores __dunder__ attributes, and > blocks all _sunder_ attributes. > > Because of this, the enum34 backport used __order__ instead of _order_ to > provide a mechanism for ordering the enum members (which I never really > liked). > > However, I've been working on my aenum [1] package, which uses several > other _sunder_ attributes (for python2 compatibility) so I enabled _order_ > instead and promote that spelling in the docs. > > Unlike the other _sunder_ attributes, _order_ has no meaningful affect in > Python 3 so I'd like to change the stdlib Enum to allow it (and either > ignore completely, or check it is the same as the definition order). > > My question is: Should I put this change in 3.5.2? > > - Yes means 3.5.2+ will work with _order_, 3.4, 3.5.0, and 3.5.1 > will not; > > - No means 3.4 and all of 3.5 will not. > > -- > ~Ethan~ > > > [1] https://pypi.python.org/pypi/aenum > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun May 8 18:43:38 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 May 2016 15:43:38 -0700 Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2? In-Reply-To: References: <572F9D1D.8090309@stoneleaf.us> Message-ID: <572FC11A.9060104@stoneleaf.us> On 05/08/2016 03:29 PM, Guido van Rossum wrote: > If enum were provisional it would be okay, but since it isn't, I think > this change can't go into 3.5.2. Think if this: could any code that > works in 3.5.1 be broken by the change? No, but with the change code that works in 3.5.2 could break in 3.5.1 or 3.5.0. It's a 2/3 compatibility issue with enum34 and aenum which support _order_, and Python3.4+ which does not. The work-around is to use __order__ instead (or use enum34 or aenum instead ;) . Either way, it's only similarity to a bug is I should have named it _order_ in the beginning, and put the compatibility shim into the stdlib version at the same time. -- ~Ethan~ From guido at python.org Sun May 8 18:50:10 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 8 May 2016 15:50:10 -0700 Subject: [Python-Dev] Return type of alternative constructors In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 4:49 AM, Nick Coghlan wrote: > On 8 May 2016 at 08:39, Serhiy Storchaka wrote: > > Should alternative constructor call __new__ and __init__ methods? Thay > can > > change signature in derived class. > > I think this is typically the way to go (although, depending on the > specific type, the unpickling related methods may be a more > appropriate way for the alternate constructor to populate the instance > state) > Honestly, either of these sounds like fragile, even though I really want the alternative constructor to return an instance of the subclass (else why invoke it through the subclass). > > Should it complain if __new__ or __init__ were overridden? > > If there are alternate constructors that depend on either the > signature of __new__/__init__, unpickling support, or some other > mechanism for creating new instances, this should be mentioned in the > class documentation as a constraint on subclasses - if subclasses > don't want to meet the constraint, they'll need to override the > affected alternate constructors. > Putting this constraint in the docs sounds fragile too. :-( > Cheers, > Nick. > > P.S. The potential complexity of that is one of the reasons the design > philosophy of "prefer composition to inheritance" has emerged - > subclassing is a powerful tool, but it does mean you often end up > needing to care about more interactions between the subclass and the > base class than you really wanted to. > Indeed! We could also consider this a general weakness of the "alternative constructors are class methods" pattern. If instead these alternative constructors were folded into the main constructor (e.g. via special keyword args) it would be altogether clearer what a subclass should do. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun May 8 18:52:25 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 8 May 2016 15:52:25 -0700 Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2? In-Reply-To: <572FC11A.9060104@stoneleaf.us> References: <572F9D1D.8090309@stoneleaf.us> <572FC11A.9060104@stoneleaf.us> Message-ID: On Sun, May 8, 2016 at 3:43 PM, Ethan Furman wrote: > On 05/08/2016 03:29 PM, Guido van Rossum wrote: > > If enum were provisional it would be okay, but since it isn't, I think >> this change can't go into 3.5.2. Think if this: could any code that >> works in 3.5.1 be broken by the change? >> > > No, but with the change code that works in 3.5.2 could break in 3.5.1 or > 3.5.0. > That's bad too (and it's one reason why we're generally strict about the "no new features in bugfix releases" rule. > It's a 2/3 compatibility issue with enum34 and aenum which support > _order_, and Python3.4+ which does not. The work-around is to use > __order__ instead (or use enum34 or aenum instead ;) . > > Either way, it's only similarity to a bug is I should have named it > _order_ in the beginning, and put the compatibility shim into the stdlib > version at the same time. > I think it's a case of water under the bridge and learning to live with your mistakes. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun May 8 19:39:13 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 May 2016 16:39:13 -0700 Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2? In-Reply-To: References: <572F9D1D.8090309@stoneleaf.us> <572FC11A.9060104@stoneleaf.us> Message-ID: <572FCE21.9020309@stoneleaf.us> On 05/08/2016 03:52 PM, Guido van Rossum wrote: > I think it's a case of water under the bridge and learning to live with > your mistakes. Better than dying from them. :) -- ~Ethan~ From greg.ewing at canterbury.ac.nz Sun May 8 20:55:09 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 09 May 2016 12:55:09 +1200 Subject: [Python-Dev] Return type of alternative constructors In-Reply-To: References: Message-ID: <572FDFED.1000502@canterbury.ac.nz> Guido van Rossum wrote: > We could also consider this a general weakness of the "alternative > constructors are class methods" pattern. If instead these alternative > constructors were folded into the main constructor (e.g. via special > keyword args) it would be altogether clearer what a subclass should do. A useful guideline might be that class methods can be provided as sugar for alternative constructors, but they should all funnel through the main constructor. There's a convention like this in the Objective-C world where one of an object's "init" methods is supposed to be documented as the "designated initialiser", so that there is just one thing that subclasses need to override. -- Greg From josephnunn at gmail.com Sun May 8 21:49:11 2016 From: josephnunn at gmail.com (Joseph Lee Nunn III) Date: Sun, 8 May 2016 18:49:11 -0700 Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems In-Reply-To: References: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org> Message-ID: Thank you for looking into this for me, but since I fixed my problem by simply deleting the old SDKs I?ve moved on. However let me clarify exactly what was going on one more time, just in case you guys decide to pursue further. I had the old 10.6 SDK in /Developer/SDKs, it could have been ?special? in that I did alot of weird things for a former project and I just don?t recall. For the sake of explanation though lets say it wasn?t, what then? I installed OSX Python 3.5.1, created a virtual environment using the pyvenv command from that newly installed Python 3.5.1, and sourced its bin/activate. Then when running pip install ZODB, the compiler would be passed the -isysroot flag for the 10.6 SDK, which would change its include paths and it would find no includes or libs (if you sym link the includes into the include directory of the virtual env you will get farther and see that the linker does not find libs either). By suppling adding -v to CFLAGS you can have the compiler output the include paths and compiler call when running pip install. That is how I noticed the alternate system root supplied by the isysroot flag. Grepping around found me the _sysconfigdata.py file where that stuff was specified, it took a bit to realize that file came with the distribution and wasn?t generated on my system. When not in a virtual environment, ZODB would install just fine into the system python (2.7). Not sure why that is, or what would make the virtual environment different, but there you go. Joseph Nunn josephnunn at gmail.com joseph.nunn at uci.edu > On May 07 , 2016, at 11:52 PM, Chris Barker wrote: > > On Sat, May 7, 2016 at 10:59 PM, Chris Barker > wrote: > > .... Trying to decide if I want to go through the hassle of installing an old XCode to see if i can replicate the bug the OP is reporting.... > > OK, I found this gitHub repo with all the old SDKs: > > https://github.com/phracker/MacOSX-SDKs > > I put a copy of the 10.6 SDk in it's old location: > > /Developer/SDKs/MacOSX10.6.sdk/ > > /before I did that, I did a pip install of zodb in a virtualenv, and all seemed to go fine. THe extensions rae linked against : > > $ otool -L cPersistence.cpython-35m-darwin.so > cPersistence.cpython-35m-darwin.so : > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1) > > where that comes from -- who knows?? > > then I tried again, after putting the sdk in place: > > [note: serious pain the $%^# to get pip to stop using it's cache! ] > > and everything seemed to build fine, and at least one simple test of zodb works fine. And this is how the extensions are linked: > > $ otool -L cPersistence.cpython-35m-darwin.so > cPersistence.cpython-35m-darwin.so : > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11) > > which is the same as above, but the "current version" is much older -- so does that mean it used the 10.6 SDK???? > > Anyway, I can't reproduce the OP's bug..... > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun May 8 22:15:59 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 May 2016 12:15:59 +1000 Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2? In-Reply-To: <572FC11A.9060104@stoneleaf.us> References: <572F9D1D.8090309@stoneleaf.us> <572FC11A.9060104@stoneleaf.us> Message-ID: On 9 May 2016 at 08:43, Ethan Furman wrote: > On 05/08/2016 03:29 PM, Guido van Rossum wrote: > >> If enum were provisional it would be okay, but since it isn't, I think >> this change can't go into 3.5.2. Think if this: could any code that >> works in 3.5.1 be broken by the change? > > > No, but with the change code that works in 3.5.2 could break in 3.5.1 or > 3.5.0. > > It's a 2/3 compatibility issue with enum34 and aenum which support _order_, > and Python3.4+ which does not. The work-around is to use __order__ instead > (or use enum34 or aenum instead ;) . Needing to use a PyPI alternative to a stdlib module for increased cross-version consistency is a pretty common experience these days, so I think that's OK - end users can choose for themselves between the stability of the stdlib version and the reduced update latency of the PyPI version. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From gvanrossum at gmail.com Sun May 8 22:27:40 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun, 8 May 2016 19:27:40 -0700 Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2? In-Reply-To: References: <572F9D1D.8090309@stoneleaf.us> <572FC11A.9060104@stoneleaf.us> Message-ID: I wonder if we should add something for this? --Guido (mobile) On May 8, 2016 7:18 PM, "Nick Coghlan" wrote: > On 9 May 2016 at 08:43, Ethan Furman wrote: > > On 05/08/2016 03:29 PM, Guido van Rossum wrote: > > > >> If enum were provisional it would be okay, but since it isn't, I think > >> this change can't go into 3.5.2. Think if this: could any code that > >> works in 3.5.1 be broken by the change? > > > > > > No, but with the change code that works in 3.5.2 could break in 3.5.1 or > > 3.5.0. > > > > It's a 2/3 compatibility issue with enum34 and aenum which support > _order_, > > and Python3.4+ which does not. The work-around is to use __order__ > instead > > (or use enum34 or aenum instead ;) . > > Needing to use a PyPI alternative to a stdlib module for increased > cross-version consistency is a pretty common experience these days, so > I think that's OK - end users can choose for themselves between the > stability of the stdlib version and the reduced update latency of the > PyPI version. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun May 8 22:39:51 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 May 2016 19:39:51 -0700 Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2? In-Reply-To: References: <572F9D1D.8090309@stoneleaf.us> <572FC11A.9060104@stoneleaf.us> Message-ID: <572FF877.7090308@stoneleaf.us> On 05/08/2016 07:15 PM, Nick Coghlan wrote: > On 9 May 2016 at 08:43, Ethan Furman wrote: >> It's a 2/3 compatibility issue with enum34 and aenum which support _order_, >> and Python3.4+ which does not. The work-around is to use __order__ instead >> (or use enum34 or aenum instead ;) . > > Needing to use a PyPI alternative to a stdlib module for increased > cross-version consistency is a pretty common experience these days, so > I think that's OK - end users can choose for themselves between the > stability of the stdlib version and the reduced update latency of the > PyPI version. Are you saying I shouldn't bother updating the 3.6 Enum to ignore _order_? -- ~Ethan~ From ncoghlan at gmail.com Sun May 8 22:52:30 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 May 2016 12:52:30 +1000 Subject: [Python-Dev] Return type of alternative constructors In-Reply-To: References: Message-ID: On 9 May 2016 at 08:50, Guido van Rossum wrote: > On Sun, May 8, 2016 at 4:49 AM, Nick Coghlan wrote: >> P.S. The potential complexity of that is one of the reasons the design >> philosophy of "prefer composition to inheritance" has emerged - >> subclassing is a powerful tool, but it does mean you often end up >> needing to care about more interactions between the subclass and the >> base class than you really wanted to. > > Indeed! > > We could also consider this a general weakness of the "alternative > constructors are class methods" pattern. If instead these alternative > constructors were folded into the main constructor (e.g. via special keyword > args) it would be altogether clearer what a subclass should do. Unfortunately, even that approach gets tricky when the inheritance relationship crosses the boundary between components with independent release cycles. In my experience, this timeline is the main one that causes the pain: * Base class is released in Component A (e.g. CPython) * Subclass is released in Component B (e.g. PyPI module) * Component A releases a new base class construction feature Question: does the new construction feature work with the existing subclass in Component B if you combine it with the new version of Component A? When alternate constructors can be implemented as class methods that work by creating a default instance and using existing public API methods to mutate it, then the answer to that question is "yes", since the default constructor hasn't changed, and the new convenience constructor isn't relying on any other new features. The answer is also "yes" for existing subclasses that only add new behaviour without adding any new state, and hence just use the base class __new__ and __init__ without overriding either of them. It's when the existing subclasses overrides __new__ or __init__ and one or both of the following is true that things can get tricky: - you're working with an immutable type - the API implementing the post-creation mutation is a new one In both of those cases, the new construction feature of the base class probably won't work right without updates to the affected subclass to support the new capability (whether that's supporting a new parameter in __new__ and __init__, or adding their own implementation of the new alternate constructor). I'm genuinely unsure that's a solvable problem in the general case - it seems to be an inherent consequence of the coupling between subclasses and base classes during instance construction, akin to the challenges with subclass compatibility of the unpickling APIs when a base class adds new state. However, from a pragmatic perspective, the following approach seems to work reasonably well: * assume subclasses don't change the signature of __new__ or __init__ * note the assumptions about the default constructor signature in the alternate constructor docs to let implementors of subclasses that change the signature know they'll need to explicitly test compatibility and perhaps provide their own implementation of the alternate constructor You *do* still end up with some cases where a subclass needs to be upgraded before a new base class feature works properly for that particular subclass, but subclasses that *don't* change the constructor signature "just work". Cheers, Nick. P.S. It occurs to me that a sufficiently sophisticated typechecker might be able to look at all of the calls to "cls(*args, **kwds)" in class methods and "type(self)(*args, **kwds)" in instance methods, and use those to define a set of type constraints for the expected constructor signatures in subclassses, even if the current code base never actually invokes those code paths. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon May 9 06:21:06 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 May 2016 20:21:06 +1000 Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2? In-Reply-To: <572FF877.7090308@stoneleaf.us> References: <572F9D1D.8090309@stoneleaf.us> <572FC11A.9060104@stoneleaf.us> <572FF877.7090308@stoneleaf.us> Message-ID: On 9 May 2016 at 12:39, Ethan Furman wrote: > On 05/08/2016 07:15 PM, Nick Coghlan wrote: >> Needing to use a PyPI alternative to a stdlib module for increased >> cross-version consistency is a pretty common experience these days, so >> I think that's OK - end users can choose for themselves between the >> stability of the stdlib version and the reduced update latency of the >> PyPI version. > > Are you saying I shouldn't bother updating the 3.6 Enum to ignore _order_? I don't have an opinion on the change itself (although keeping PyPI and 3.6-to-be aligned seems reasonable), just that it's OK to suggest folks use a backport module even on earlier 3.x versions if they want the behaviour of the latest (or even the next upcoming) stdlib release, rather than the baseline behaviour of their current runtime. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Mon May 9 12:30:06 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 9 May 2016 09:30:06 -0700 Subject: [Python-Dev] Return type of alternative constructors In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 7:52 PM, Nick Coghlan wrote: > On 9 May 2016 at 08:50, Guido van Rossum wrote: > > On Sun, May 8, 2016 at 4:49 AM, Nick Coghlan wrote: > >> P.S. The potential complexity of that is one of the reasons the design > >> philosophy of "prefer composition to inheritance" has emerged - > >> subclassing is a powerful tool, but it does mean you often end up > >> needing to care about more interactions between the subclass and the > >> base class than you really wanted to. > > > > Indeed! > > > > We could also consider this a general weakness of the "alternative > > constructors are class methods" pattern. If instead these alternative > > constructors were folded into the main constructor (e.g. via special > keyword > > args) it would be altogether clearer what a subclass should do. > > Unfortunately, even that approach gets tricky when the inheritance > relationship crosses the boundary between components with independent > release cycles. > > In my experience, this timeline is the main one that causes the pain: > > * Base class is released in Component A (e.g. CPython) > * Subclass is released in Component B (e.g. PyPI module) > * Component A releases a new base class construction feature > > Question: does the new construction feature work with the existing > subclass in Component B if you combine it with the new version of > Component A? > > When alternate constructors can be implemented as class methods that > work by creating a default instance and using existing public API > methods to mutate it, then the answer to that question is "yes", since > the default constructor hasn't changed, and the new convenience > constructor isn't relying on any other new features. > > The answer is also "yes" for existing subclasses that only add new > behaviour without adding any new state, and hence just use the base > class __new__ and __init__ without overriding either of them. > > It's when the existing subclasses overrides __new__ or __init__ and > one or both of the following is true that things can get tricky: > > - you're working with an immutable type > - the API implementing the post-creation mutation is a new one > > In both of those cases, the new construction feature of the base class > probably won't work right without updates to the affected subclass to > support the new capability (whether that's supporting a new parameter > in __new__ and __init__, or adding their own implementation of the new > alternate constructor). > OTOH it's not the end of the world -- until B is updated, you can't use the new construction feature with subclass B, you have to use the old way of constructing instances of B. Presumably that old way is still supported, otherwise the change to A has just broken all of B, regardless of the new construction feature. Which is possible, but it's a choice that A's author has to make after careful deliberation. Or maybe the "class construction" machinery in A is so prominent that it really is part of the interface between A and any of its subclasses, and then that API had better be documented. Just saying "you can subclass this" won't be sufficient. > I'm genuinely unsure that's a solvable problem in the general case - > it seems to be an inherent consequence of the coupling between > subclasses and base classes during instance construction, akin to the > challenges with subclass compatibility of the unpickling APIs when a > base class adds new state. > Yup. My summary of it is that versioning sucks. > However, from a pragmatic perspective, the following approach seems to > work reasonably well: > > * assume subclasses don't change the signature of __new__ or __init__ > I still find that a distasteful choice, because *in general* there is no requirement like that and there are good reasons why subclasses might have a different __init__/__new__ signature. (For example dict and defaultdict.) > * note the assumptions about the default constructor signature in the > alternate constructor docs to let implementors of subclasses that > change the signature know they'll need to explicitly test > compatibility and perhaps provide their own implementation of the > alternate constructor > Yup, you end up having to design the API for subclasses carefully and then document it precisely. This is what people too often forget when they complain e.g. "why can't I subclass EventLoop more easily" -- we don't want to have a public API for that, so we discourage it, but people mistakenly believe that anything that's a class should be subclassable. > You *do* still end up with some cases where a subclass needs to be > upgraded before a new base class feature works properly for that > particular subclass, but subclasses that *don't* change the > constructor signature "just work". > The key is that there's an API requirement and that you have to design and document that API with future evolution in mind. If you don't do and let people write subclasses that just happen to work, you're in a lot of pain. The interface between a base class and a subclass just is very complex so must designers of base classes get this wrong initially. > Cheers, > Nick. > > P.S. It occurs to me that a sufficiently sophisticated typechecker > might be able to look at all of the calls to "cls(*args, **kwds)" in > class methods and "type(self)(*args, **kwds)" in instance methods, and > use those to define a set of type constraints for the expected > constructor signatures in subclassses, even if the current code base > never actually invokes those code paths. > Could you restate that as a concrete code example? (Examples of the problems with "construction features" would also be helpful, probably -- abstract descriptions of problems often lead me astray.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon May 9 14:03:31 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 9 May 2016 11:03:31 -0700 Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems In-Reply-To: References: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org> Message-ID: Thanks Joseph, I say this isn't worth pursuing until someone else reports the issue, as I couldn't replicate it with a "clean" system. I had the old 10.6 SDK in /Developer/SDKs, it could have been ?special? in > that I did alot of weird things for a former project and I just don?t > recall. > I suspect your 10.6 SDK was cnfigured oddly, or ???? For the sake of explanation though lets say it wasn?t, what then? > > I installed OSX Python 3.5.1, created a virtual environment using the > pyvenv command from that newly installed Python 3.5.1, and sourced its > bin/activate. > Then when running pip install ZODB, the compiler would be passed the > -isysroot flag for the 10.6 SDK, which would change its include paths and > it would find no includes or libs > this is the odd part -- shouldn't the new sysroot (i.e. the SDK) have all the includes and libs needed? Not sure why that is, or what would make the virtual environment different, > but there you go. > yeah, me neither .... Anyway, unless someone else sees this, I think we're done. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephnunn at gmail.com Mon May 9 15:08:10 2016 From: josephnunn at gmail.com (Joseph Lee Nunn III) Date: Mon, 9 May 2016 12:08:10 -0700 Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems In-Reply-To: References: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org> Message-ID: <66EF7E49-D14C-42CD-B68A-10AC63566213@gmail.com> > On May 09 , 2016, at 11:03 AM, Chris Barker wrote: > > I installed OSX Python 3.5.1, created a virtual environment using the pyvenv command from that newly installed Python 3.5.1, and sourced its bin/activate. > Then when running pip install ZODB, the compiler would be passed the -isysroot flag for the 10.6 SDK, which would change its include paths and it would find no includes or libs > > this is the odd part -- shouldn't the new sysroot (i.e. the SDK) have all the includes and libs needed? > That is the weird part, if installing into the system python the new sysroot worked just fine. It was only when trying to install into a virtual environment that it failed. What the difference is I just don?t know. Thanks again for you efforts! Joseph -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Mon May 9 22:13:48 2016 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Mon, 9 May 2016 19:13:48 -0700 Subject: [Python-Dev] PyGC_Collect ignores state of `enabled` Message-ID: Is this deliberate? Could we change it for 3.6 so PyGC_Collect does take `enabled` into account? Context: we?re experimenting with disabling the gc entirely for some workloads because it doesn?t fare too well with Copy-on-Write on Linux. What ends up happening is that disabling the gc actually drops memory usage significantly if you?re using hundreds of the same processes on one box :-) However, because of PyGC_Collect() called in Py_Finalize(), during interpreter shutdown the collection is done anyway, Linux does CoW and the memory usage spikes. Which is ironic on process shutdown. -- Lukasz Langa | Facebook Production Engineer | The Ministry of Silly Walks (+1) 650-681-7811 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From benjamin at python.org Tue May 10 01:55:51 2016 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 09 May 2016 22:55:51 -0700 Subject: [Python-Dev] PyGC_Collect ignores state of `enabled` In-Reply-To: References: Message-ID: <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com> Adding PyGC_CollectIfEnabled() and calling it in Py_Finalize is probably fine. I don't think the contract of PyGC_Collect itself (or gc.collect() for that matter) should be changed. You might want to disable GC but invoke it yourself. On Mon, May 9, 2016, at 19:13, ?ukasz Langa wrote: > Is this deliberate? Could we change it for 3.6 so PyGC_Collect does take > `enabled` into account? > > Context: we?re experimenting with disabling the gc entirely for some > workloads because it doesn?t fare too well with Copy-on-Write on Linux. > What ends up happening is that disabling the gc actually drops memory > usage significantly if you?re using hundreds of the same processes on one > box :-) > > However, because of PyGC_Collect() called in Py_Finalize(), during > interpreter shutdown the collection is done anyway, Linux does CoW and > the memory usage spikes. Which is ironic on process shutdown. > > -- > Lukasz Langa | Facebook > Production Engineer | The Ministry of Silly Walks > (+1) 650-681-7811 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org > Email had 1 attachment: > + signature.asc > 1k (application/pgp-signature) From vadmium+py at gmail.com Tue May 10 04:17:58 2016 From: vadmium+py at gmail.com (Martin Panter) Date: Tue, 10 May 2016 08:17:58 +0000 Subject: [Python-Dev] ctypes compatibility with 2.3 Message-ID: I am working on , to fix shell injection problems with ctypes.util.find_library(). The proposal for Python 3 is to change os.popen(shell-script) calls to use subprocess.Popen(). However the Python 2.7 version of the module has a comment which says ?This file should be kept compatible with Python 2.3, see PEP 291.? Looking at , it is not clear why we have to maintain this compatibility. My best guess is that there may be an external ctypes package that people want(ed) to keep compatible with 2.3, and also keep synchronized with 2.7. I would like to lift this restriction to at least 2.4, because that is when the ?subprocess? module was added. I notice that there is already code that relies on the list.sort(key=...) feature, which was added in 2.4. Ideally I would prefer to drop the restriction and only require 2.7 compatibility. Would either of these options be a problem? If it is a problem, I think it is still possible to avoid the shell by passing a sequence of program arguments to os.popen(). But I prefer not to do this, because the function is marked as deprecated, and the code would be substantially different to Python 3. -Martin From ncoghlan at gmail.com Tue May 10 09:21:57 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 May 2016 23:21:57 +1000 Subject: [Python-Dev] Return type of alternative constructors In-Reply-To: References: Message-ID: On 10 May 2016 at 02:30, Guido van Rossum wrote: > On Sun, May 8, 2016 at 7:52 PM, Nick Coghlan wrote: >> P.S. It occurs to me that a sufficiently sophisticated typechecker >> might be able to look at all of the calls to "cls(*args, **kwds)" in >> class methods and "type(self)(*args, **kwds)" in instance methods, and >> use those to define a set of type constraints for the expected >> constructor signatures in subclassses, even if the current code base >> never actually invokes those code paths. > > Could you restate that as a concrete code example? (Examples of the problems > with "construction features" would also be helpful, probably -- abstract > descriptions of problems often lead me astray.) Rectangle/Square is a classic example of the constructor signature changing, so I'll try to use that to illustrate the point with a "displaced_copy" alternate constructor: class Rectangle: def __init__(self, top_left_point, width, height): self.top_left_point = top_left_point self.width = width self.height = height @classmethod def displaced_copy(cls, other_rectangle, offset): """Create a new instance from an existing one""" return cls(other.top_left_point + offset, other.width, other.height) class Square: def __init__(self, top_left_point, side_length): super().__init__(top_left_point, side_length, side_length) At this point, a typechecker *could* have enough info to know that "Square.displaced_copy(some_rectangle, offset)" is necessarily going to fail, even if nothing in the application actually *calls* Square.displaced_copy. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Tue May 10 13:39:44 2016 From: brett at python.org (Brett Cannon) Date: Tue, 10 May 2016 17:39:44 +0000 Subject: [Python-Dev] ctypes compatibility with 2.3 In-Reply-To: References: Message-ID: On Tue, 10 May 2016 at 01:18 Martin Panter wrote: > I am working on , to fix shell > injection problems with ctypes.util.find_library(). The proposal for > Python 3 is to change os.popen(shell-script) calls to use > subprocess.Popen(). > > However the Python 2.7 version of the module has a comment which says > ?This file should be kept compatible with Python 2.3, see PEP 291.? > Looking at , it is not > clear why we have to maintain this compatibility. My best guess is > that there may be an external ctypes package that people want(ed) to > keep compatible with 2.3, and also keep synchronized with 2.7. > That's correct and the maintainer is/was Thomas Heller who I have cc'ed to see if he's okay with lifting the restriction. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 10 14:02:13 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 10 May 2016 11:02:13 -0700 Subject: [Python-Dev] Return type of alternative constructors In-Reply-To: References: Message-ID: On Tue, May 10, 2016 at 6:21 AM, Nick Coghlan wrote: > On 10 May 2016 at 02:30, Guido van Rossum wrote: > > On Sun, May 8, 2016 at 7:52 PM, Nick Coghlan wrote: > >> P.S. It occurs to me that a sufficiently sophisticated typechecker > >> might be able to look at all of the calls to "cls(*args, **kwds)" in > >> class methods and "type(self)(*args, **kwds)" in instance methods, and > >> use those to define a set of type constraints for the expected > >> constructor signatures in subclassses, even if the current code base > >> never actually invokes those code paths. > > > > Could you restate that as a concrete code example? (Examples of the > problems > > with "construction features" would also be helpful, probably -- abstract > > descriptions of problems often lead me astray.) > > Rectangle/Square is a classic example of the constructor signature > changing, so I'll try to use that to illustrate the point with a > "displaced_copy" alternate constructor: > > class Rectangle: > def __init__(self, top_left_point, width, height): > self.top_left_point = top_left_point > self.width = width > self.height = height > > @classmethod > def displaced_copy(cls, other_rectangle, offset): > """Create a new instance from an existing one""" > return cls(other.top_left_point + offset, other.width, > other.height) > (But why is it a class method? I guess the example could also use an instance method and it would still have the same properties relevant for this discussion.) > class Square: > def __init__(self, top_left_point, side_length): > super().__init__(top_left_point, side_length, side_length) > > At this point, a typechecker *could* have enough info to know that > "Square.displaced_copy(some_rectangle, offset)" is necessarily going > to fail, even if nothing in the application actually *calls* > Square.displaced_copy. > The question remains of course whether the type checker should flag Square to be an invalid subclass or merely as not implementing displaced_copy(). Anyway, at this point I believe we're just violently agreeing, so no need for another response. Though Serhiy may be unhappy with the lack of guidance he's received... -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From theller at ctypes.org Wed May 11 07:35:23 2016 From: theller at ctypes.org (Thomas Heller) Date: Wed, 11 May 2016 13:35:23 +0200 Subject: [Python-Dev] ctypes compatibility with 2.3 In-Reply-To: References: Message-ID: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org> Am 10.05.2016 um 19:39 schrieb Brett Cannon: > > > On Tue, 10 May 2016 at 01:18 Martin Panter > wrote: > > I am working on , to fix shell > injection problems with ctypes.util.find_library(). The proposal for > Python 3 is to change os.popen(shell-script) calls to use > subprocess.Popen(). > > However the Python 2.7 version of the module has a comment which says > ?This file should be kept compatible with Python 2.3, see PEP 291.? > Looking at , it is not > clear why we have to maintain this compatibility. My best guess is > that there may be an external ctypes package that people want(ed) to > keep compatible with 2.3, and also keep synchronized with 2.7. > > > That's correct and the maintainer is/was Thomas Heller who I have cc'ed > to see if he's okay with lifting the restriction. For me it is totally ok to lift this restriction. Thomas From brett at python.org Wed May 11 12:04:27 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 16:04:27 +0000 Subject: [Python-Dev] ctypes compatibility with 2.3 In-Reply-To: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org> References: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org> Message-ID: On Wed, 11 May 2016 at 04:35 Thomas Heller wrote: > Am 10.05.2016 um 19:39 schrieb Brett Cannon: > > > > > > On Tue, 10 May 2016 at 01:18 Martin Panter > > wrote: > > > > I am working on , to fix shell > > injection problems with ctypes.util.find_library(). The proposal for > > Python 3 is to change os.popen(shell-script) calls to use > > subprocess.Popen(). > > > > However the Python 2.7 version of the module has a comment which says > > ?This file should be kept compatible with Python 2.3, see PEP 291.? > > Looking at , it is not > > clear why we have to maintain this compatibility. My best guess is > > that there may be an external ctypes package that people want(ed) to > > keep compatible with 2.3, and also keep synchronized with 2.7. > > > > > > That's correct and the maintainer is/was Thomas Heller who I have cc'ed > > to see if he's okay with lifting the restriction. > > For me it is totally ok to lift this restriction. > Great! I'll also update PEP 291. -------------- next part -------------- An HTML attachment was scrubbed... URL: From theller at ctypes.org Wed May 11 12:07:30 2016 From: theller at ctypes.org (Thomas Heller) Date: Wed, 11 May 2016 18:07:30 +0200 Subject: [Python-Dev] ctypes compatibility with 2.3 In-Reply-To: References: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org> Message-ID: <3283bb48-531a-0795-babd-df203115df58@ctypes.org> Am 11.05.2016 um 18:04 schrieb Brett Cannon: > > > On Wed, 11 May 2016 at 04:35 Thomas Heller > wrote: > > Am 10.05.2016 um 19:39 schrieb Brett Cannon: > > > > > > On Tue, 10 May 2016 at 01:18 Martin Panter > > >> > wrote: > > > > I am working on , to fix shell > > injection problems with ctypes.util.find_library(). The > proposal for > > Python 3 is to change os.popen(shell-script) calls to use > > subprocess.Popen(). > > > > However the Python 2.7 version of the module has a comment > which says > > ?This file should be kept compatible with Python 2.3, see PEP > 291.? > > Looking at , it is not > > clear why we have to maintain this compatibility. My best guess is > > that there may be an external ctypes package that people > want(ed) to > > keep compatible with 2.3, and also keep synchronized with 2.7. > > > > > > That's correct and the maintainer is/was Thomas Heller who I have > cc'ed > > to see if he's okay with lifting the restriction. > > For me it is totally ok to lift this restriction. > > > Great! I'll also update PEP 291. Cool. While you're at it, the compatibility restriction for modulefinder could also be lifted. From meadori at gmail.com Wed May 11 12:19:09 2016 From: meadori at gmail.com (Meador Inge) Date: Wed, 11 May 2016 11:19:09 -0500 Subject: [Python-Dev] ctypes compatibility with 2.3 In-Reply-To: <3283bb48-531a-0795-babd-df203115df58@ctypes.org> References: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org> <3283bb48-531a-0795-babd-df203115df58@ctypes.org> Message-ID: On Wed, May 11, 2016 at 11:07 AM, Thomas Heller wrote: Cool. While you're at it, the compatibility restriction for modulefinder > could also be lifted. +1 The question of modulefinder actually came up recently*: http://bugs.python.org/issue26881 -- Meador * Posting here for reference. Thomas already knows this as he is on the issue26881 watch list :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 11 12:28:47 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 16:28:47 +0000 Subject: [Python-Dev] ctypes compatibility with 2.3 In-Reply-To: <3283bb48-531a-0795-babd-df203115df58@ctypes.org> References: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org> <3283bb48-531a-0795-babd-df203115df58@ctypes.org> Message-ID: On Wed, 11 May 2016 at 09:07 Thomas Heller wrote: > Am 11.05.2016 um 18:04 schrieb Brett Cannon: > > > > > > On Wed, 11 May 2016 at 04:35 Thomas Heller > > wrote: > > > > Am 10.05.2016 um 19:39 schrieb Brett Cannon: > > > > > > > > > On Tue, 10 May 2016 at 01:18 Martin Panter > > > > >> > > wrote: > > > > > > I am working on , to fix > shell > > > injection problems with ctypes.util.find_library(). The > > proposal for > > > Python 3 is to change os.popen(shell-script) calls to use > > > subprocess.Popen(). > > > > > > However the Python 2.7 version of the module has a comment > > which says > > > ?This file should be kept compatible with Python 2.3, see PEP > > 291.? > > > Looking at , it is > not > > > clear why we have to maintain this compatibility. My best > guess is > > > that there may be an external ctypes package that people > > want(ed) to > > > keep compatible with 2.3, and also keep synchronized with 2.7. > > > > > > > > > That's correct and the maintainer is/was Thomas Heller who I have > > cc'ed > > > to see if he's okay with lifting the restriction. > > > > For me it is totally ok to lift this restriction. > > > > > > Great! I'll also update PEP 291. > > Cool. While you're at it, the compatibility restriction for > modulefinder could also be lifted. > Will do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 11 12:34:10 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 16:34:10 +0000 Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3) In-Reply-To: References: Message-ID: Is there anything holding up PEP 515 at this point in terms of acceptance or implementation? On Sat, 19 Mar 2016 at 11:56 Guido van Rossum wrote: > All that sounds fine! > > On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah wrote: > > Guido van Rossum python.org> writes: > >> So should the preprocessing step just be s.replace('_', ''), or should > >> it reject underscores that don't follow the rules from the PEP > >> (perhaps augmented so they follow the spirit of the PEP and the letter > >> of the IBM spec)? > >> > >> Honestly I think it's also fine if specifying this exactly is left out > >> of the PEP, and handled by whoever adds this to Decimal. Having a PEP > >> to work from for the language spec and core builtins (int(), float() > >> complex()) is more important. > > > > I'd keep it simple for Decimal: Remove left and right whitespace (we're > > already doing this), then remove underscores from the remaining string > > (which must not contain any further whitespace), then use the IBM > grammar. > > > > > > We could add a clause to the PEP that only those strings that follow > > the spirit of the PEP are guaranteed to be accepted in the future. > > > > > > One reason for keeping it simple is that I would not like to slow down > > string conversion, but thinking about two grammars is also a problem -- > > part of the string conversion in libmpdec is modeled in ACL2, which > > would be invalidated or at least complicated with two grammars. > > > > > > > > Stefan Krah > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 11 12:43:24 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 16:43:24 +0000 Subject: [Python-Dev] file system path protocol PEP Message-ID: **deep, calming breath** Here is the PEP for __fspath__(). The draft lives at https://github.com/brettcannon/path-pep so feel free to send me PRs for spelling mistakes, grammatical errors, etc. ----- PEP: NNN Title: Adding a file system path protocol Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-May-2016 Post-History: 11-May-2016 Abstract ======== This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only ``str`` and/or ``bytes`` file system paths are accepted. The goal is to allow users to use the representation of a file system path that's easiest for them now as they migrate towards using path objects in the future. Rationale ========= Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path. This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not. To help elevate the representation of file system paths from their representation as strings and bytes to a more appropriate object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone. One issue in converting path objects to strings comes from the fact that only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file. Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library had become an issue. A solution that allowed any path-representing object to declare that is was a path and a way to extract a low-level representation that all path objects could support was desired. This PEP then proposes to introduce a new protocol to be followed by objects which represent file system paths. Providing a protocol allows for clear signalling of what objects represent file system paths as well as a way to extract a lower-level representation that can be used with older APIs which only support strings or bytes. Discussions regarding path objects that led to this PEP can be found in multiple threads on the python-ideas mailing list archive [#python-ideas-archive]_ for the months of March and April 2016 and on the python-dev mailing list archives [#python-dev-archive]_ during April 2016. Proposal ======== This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation. The other part is changes to Python's standard library to support the new protocol. These changes will also have the pathlib module drop its provisional status. Protocol -------- The following abstract base class defines the protocol for an object to be considered a path object:: import abc import typing as t class PathLike(abc.ABC): """Abstract base class for implementing the file system path protocol.""" @abc.abstractmethod def __fspath__(self) -> t.Union[str, bytes]: """Return the file system path representation of the object.""" raise NotImplementedError Objects representing file system paths will implement the ``__fspath__()`` method which will return the ``str`` or ``bytes`` representation of the path. The ``str`` representation is the preferred low-level path representation as it is human-readable and what people historically represent paths as. Standard library changes ------------------------ It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary). The modules mentioned below, though, deserve specific details as they have either fundamental changes that empower the ability to use path objects, or entail additions/removal of APIs. builtins '''''''' ``open()`` [#builtins-open]_ will be updated to accept path objects as well as continue to accept ``str`` and ``bytes``. os ''' The ``fspath()`` function will be added with the following semantics (although this is still an open issue: `Should os.fspath() return bytes?`_):: import typing as t def fspath(path: t.Union[PathLike, str]) -> str: """Return the string representation of the path. If a string is passed in then it is returned unchanged. """ if hasattr(path, '__fspath__'): path = path.__fspath__() if not isinstance(path, str): type_name = type(path).__name__ raise TypeError("expected a str or path object, not " + type_name) return path The ``os.fsencode()`` [#os-fsencode]_ and ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept path objects. As both functions coerce their arguments to ``bytes`` and ``str``, respectively, they will be updated to call ``__fspath__()`` if present to convert the path object to a ``str`` or ``bytes`` representation, and then perform their appropriate coercion operations as if the return value from ``__fspath__()`` had been the original argument to the coercion function in question. The addition of ``os.fspath()``, the updates to ``os.fsencode()``/``os.fsdecode()``, and the current semantics of ``pathlib.PurePath`` provide the semantics necessary to get the path representation one prefers. For a path object, ``pathlib.PurePath``/``Path`` can be used. If ``str`` is desired and a ``bytes`` object is considered an error, then ``os.fspath()`` can be used. If a ``str`` is desired and the encoding of ``bytes`` should be assumed to be the default file system encoding, then ``os.fsdecode()`` should be used. Finally, if a ``bytes`` representation is desired and any strings should be encoded using the default file system encoding then ``os.fsencode()`` is used. This PEP recommends using path objects when possible and falling back to string paths as necessary. Therefore, no function is provided for the case of wanting a bytes representation but without any automatic encoding to help discourage the use of multiple bytes encodings on a single file system. If it is necessary to deal with an existing file system directory with entries in a non-default encoding, this can be done with low-level functions using ``str`` and the PEP 383 ``surrogateescape`` error handler, or by using ``bytes`` directly. Another way to view this is as a hierarchy of file system path representations (highest- to lowest-level): path -> str -> bytes. The functions and classes under discussion can all accept objects on the same level of the hierarchy, but they vary in whether they promote or demote objects to another level. The ``pathlib.PurePath`` class can promote a ``str`` to a path object. The ``os.fspath()`` function can demote a path object to a string, but only if ``__fspath__()`` returns a string. The ``os.fsdecode()`` function will demote a path object to a string or promote a ``bytes`` object to a ``str``. The ``os.fsencode()`` function will demote a path or string object to ``bytes``. There is no function that provides a way to demote a path object directly to ``bytes`` and not allow demoting strings. The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` method. It will return the same value as currently found on the ``path`` attribute of ``DirEntry`` instances. os.path ''''''' The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function. During the discussions leading up to this PEP it was suggested that ``os.path`` not be updated using an "explicit is better than implicit" argument. The thinking was that since ``__fspath__()`` is polymorphic itself it may be better to have code working with ``os.path`` extract the path representation from path objects explicitly. There is also the consideration that adding support this deep into the low-level OS APIs will lead to code magically supporting path objects without requiring any documentation updated, leading to potential complaints when it doesn't work, unbeknownst to the project author. But it is the view of the authors that "practicality beats purity" in this instance. To help facilitate the transition to supporting path objects, it is better to make the transition as easy as possible than to worry about unexpected/undocumented duck typing support for projects. pathlib ''''''' The ``PathLike`` ABC as discussed in the Protocol_ section will be added to the pathlib module [#pathlib]_. The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept path objects. Both ``PurePath`` and ``Path`` will continue to not accept ``bytes`` path representations, and so if ``__fspath__()`` returns ``bytes`` it will raise an exception. The ``path`` attribute will be removed as this PEP makes its redundant (it has not been included in any released version of Python and so is not a backwards-compatibility concern). C API ''''' The C API will gain an equivalent function to ``os.fspath()`` that also allows bytes objects through:: /* Return the file system path of the object. If the object is str or bytes, then allow it to pass through with an incremented refcount. All other types raise a TypeError. */ PyObject * PyOS_RawFSPath(PyObject *path) { if (PyObject_HasAttrString(path, "__fspath__")) { path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL); if (path == NULL) { return NULL; } } else { Py_INCREF(path); } if (!PyUnicode_Check(path) && !PyBytes_Check(path)) { Py_DECREF(path); return PyErr_Format(PyExc_TypeError, "expected a string, bytes, or path object, not %S", path->ob_type); } return path; } Backwards compatibility ======================= There are no explicit backwards-compatibility concerns. Unless an object incidentally already defines a ``__fspath__()`` method there is not reason to expect pre-existing code to break or expect to have their semantics implicitly changed. Libraries wishing to support path objects and a version of Python prior to Python 3.6 can use the idiom of ``path.__fspath__() if hasattr(path, '__fspath__') else path``. Open Issues =========== Should os.fspath() return bytes? -------------------------------- Some have argued that ``os.fspath()`` should be configurable so that the user can specify what types are acceptable (e.g. an argument to say that bytes are acceptable instead of strings, or both types). Others have suggested that ``os.fspath()`` match the proposed semantics of ``PyOS_RawFSPath()``. Both camps argue that use of ``os.fspath()`` will only be for a transitionary period while more libraries gain acceptance of path objects, and so being more flexible in what ``os.fspath()`` works with will help with the transition. The opponents to this -- which support the currently proposed sematnics -- worry that being so flexible with accepting bytes will lead to people not properly considering the ramifications of working with bytes, especially if bytes are transparently appearing in their code due to ``os.fspath()``. The name and location of the protocol's ABC ------------------------------------------- The name of the ABC being proposed to represent the protocol has not been discussed very much, not which module it should exist in. Names other than ``PathLike`` which are viable are ``PathABC`` and ``FSPathABC``. The name can't be ``Path`` if the ABC is put into the pathlib module. Type hint for path-like objects ------------------------------- Creating a proper type hint for APIs that accept path objects as well as strings and bytes will probably be needed. It could be as simple as defining ``typing.Path``/``typing.FSPath`` to correspond to the ABC and then having ``typing.PathLike = typing.Union[typing.Path, str, bytes]``. The type hint could also potentially be made to be generic to accept the specific low-level representation, e.g. ``typing.PathLike[str]``. In the end the type hinting solution should be properly discussed with the right type hinting experts if this is the best approach. Rejected Ideas ============== Other names for the protocol's function --------------------------------------- Various names were proposed during discussions leading to this PEP, including ``__path__``, ``__pathname__``, and ``__fspathname__``. In the end people seemed to gravitate towards ``__fspath__`` for being unambiguous without unnecessarily long. Separate str/bytes methods -------------------------- At one point it was suggested that ``__fspath__()`` only return strings and another method named ``__fspathb__()`` be introduced to return bytes. The thinking that by making ``__fspath__()`` not be polymorphic it could make dealing with the potential string or bytes representations easier. But the general consensus was that returning bytes will more than likely be rare and that the various functions in the os module are the better abstraction to be promoting over direct calls to ``__fspath__()``. Providing a path attribute -------------------------- To help deal with the issue of ``pathlib.PurePath`` not inheriting from ``str``, originally it was proposed to introduce a ``path`` attribute to mirror what ``os.DirEntry`` provides. In the end, though, it was determined that a protocol would provide the same result while not directly exposing an API that most people will never need to interact with directly. Have ``__fspath__()`` only return strings ------------------------------------------ Much of the discussion that led to this PEP revolved around whether ``__fspath__()`` should be polymorphic and return ``bytes`` as well as ``str`` instead of only ``str``. The general sentiment for this view was that ``bytes`` are difficult to work with due to their inherent lack of information about their encoding, and PEP 383 makes it possible to represent all file system paths using ``str`` with the ``surrogateescape`` handler. Thus it would be better to forcibly promote the use of ``str`` as the low-level path representation for high-level path objects. In the end it was decided that using ``bytes`` to represent paths is simply not going to go away and thus they should be supported to some degree. For those not wanting the hassle of working with ``bytes``, ``os.fspath()`` is provided. A generic string encoding mechanism ----------------------------------- At one point there was discussion of developing a generic mechanism to extract a string representation of an object that had semantic meaning (``__str__()`` does not necessarily return anything of semantic significance beyond what may be helpful for debugging). In the end it was deemed to lack a motivating need beyond the one this PEP is trying to solve in a specific fashion. Acknowledgements ================ Thanks to everyone who participated in the various discussions related to this PEP that spanned both python-ideas and python-dev. Special thanks to Koos Zevenhoven and Stephen Turnbull for direct feedback on early drafts of this PEP. References ========== .. [#python-ideas-archive] The python-ideas mailing list archive (https://mail.python.org/pipermail/python-ideas/) .. [#python-dev-archive] The python-dev mailing list archive (https://mail.python.org/pipermail/python-dev/) .. [#libc-open] ``open()`` documention for the C standard library ( http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html ) .. [#pathlib] The ``pathlib`` module (https://docs.python.org/3/library/pathlib.html#module-pathlib) .. [#builtins-open] The ``builtins.open()`` function (https://docs.python.org/3/library/functions.html#open) .. [#os-fsencode] The ``os.fsencode()`` function (https://docs.python.org/3/library/os.html#os.fsencode) .. [#os-fsdecode] The ``os.fsdecode()`` function (https://docs.python.org/3/library/os.html#os.fsdecode) .. [#os-direntry] The ``os.DirEntry`` class (https://docs.python.org/3/library/os.html#os.DirEntry) .. [#os-path] The ``os.path`` module (https://docs.python.org/3/library/os.path.html#module-os.path) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed May 11 12:46:52 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 11 May 2016 09:46:52 -0700 Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3) In-Reply-To: References: Message-ID: If the authors are happy I'll accept it right away. (I vaguely recall there's another PEP that's ready for pronouncement -- but which one?) On Wed, May 11, 2016 at 9:34 AM, Brett Cannon wrote: > Is there anything holding up PEP 515 at this point in terms of acceptance > or implementation? > > On Sat, 19 Mar 2016 at 11:56 Guido van Rossum wrote: > >> All that sounds fine! >> >> On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah >> wrote: >> > Guido van Rossum python.org> writes: >> >> So should the preprocessing step just be s.replace('_', ''), or should >> >> it reject underscores that don't follow the rules from the PEP >> >> (perhaps augmented so they follow the spirit of the PEP and the letter >> >> of the IBM spec)? >> >> >> >> Honestly I think it's also fine if specifying this exactly is left out >> >> of the PEP, and handled by whoever adds this to Decimal. Having a PEP >> >> to work from for the language spec and core builtins (int(), float() >> >> complex()) is more important. >> > >> > I'd keep it simple for Decimal: Remove left and right whitespace (we're >> > already doing this), then remove underscores from the remaining string >> > (which must not contain any further whitespace), then use the IBM >> grammar. >> > >> > >> > We could add a clause to the PEP that only those strings that follow >> > the spirit of the PEP are guaranteed to be accepted in the future. >> > >> > >> > One reason for keeping it simple is that I would not like to slow down >> > string conversion, but thinking about two grammars is also a problem -- >> > part of the string conversion in libmpdec is modeled in ACL2, which >> > would be invalidated or at least complicated with two grammars. >> > >> > >> > >> > Stefan Krah >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 11 13:28:51 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 17:28:51 +0000 Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3) In-Reply-To: References: Message-ID: On Wed, 11 May 2016 at 09:47 Guido van Rossum wrote: > If the authors are happy I'll accept it right away. > > (I vaguely recall there's another PEP that's ready for pronouncement -- > but which one?) > PEP 509 is the only one I can think of. -Brett > > On Wed, May 11, 2016 at 9:34 AM, Brett Cannon wrote: > >> Is there anything holding up PEP 515 at this point in terms of acceptance >> or implementation? >> >> On Sat, 19 Mar 2016 at 11:56 Guido van Rossum wrote: >> >>> All that sounds fine! >>> >>> On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah >>> wrote: >>> > Guido van Rossum python.org> writes: >>> >> So should the preprocessing step just be s.replace('_', ''), or should >>> >> it reject underscores that don't follow the rules from the PEP >>> >> (perhaps augmented so they follow the spirit of the PEP and the letter >>> >> of the IBM spec)? >>> >> >>> >> Honestly I think it's also fine if specifying this exactly is left out >>> >> of the PEP, and handled by whoever adds this to Decimal. Having a PEP >>> >> to work from for the language spec and core builtins (int(), float() >>> >> complex()) is more important. >>> > >>> > I'd keep it simple for Decimal: Remove left and right whitespace (we're >>> > already doing this), then remove underscores from the remaining string >>> > (which must not contain any further whitespace), then use the IBM >>> grammar. >>> > >>> > >>> > We could add a clause to the PEP that only those strings that follow >>> > the spirit of the PEP are guaranteed to be accepted in the future. >>> > >>> > >>> > One reason for keeping it simple is that I would not like to slow down >>> > string conversion, but thinking about two grammars is also a problem -- >>> > part of the string conversion in libmpdec is modeled in ACL2, which >>> > would be invalidated or at least complicated with two grammars. >>> > >>> > >>> > >>> > Stefan Krah >>> > >>> > _______________________________________________ >>> > Python-Dev mailing list >>> > Python-Dev at python.org >>> > https://mail.python.org/mailman/listinfo/python-dev >>> > Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/brett%40python.org >>> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed May 11 13:36:46 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 11 May 2016 20:36:46 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: **another deep, calming breath** On Wed, May 11, 2016 at 7:43 PM, Brett Cannon wrote: > Open Issues > =========== > > Should os.fspath() return bytes? > -------------------------------- > In most cases, it of course should not. The section (or the title) do not represent my view on the topic, but bytes paths are not going away any time soon, so this requires considerations. Below is a copy paste of my comment in the discussion on my pull request ( https://github.com/brettcannon/path-pep/pull/2 ). It is about whether os.fspath should look like def fspath(path, *, type_constraint=str): ... This was already discussed before, receiving positive reactions. It would be an extension to what is in the draft Brett just posted. Note that the return type, however, does not depend on `type_constraint`. It fully depends on `path`. The constraint only determines when an exception is raised instead of returning a value. When using str-based paths normally, one would do os.fspath(path) Calling it with os.fspath(path, type_constraint=(str, bytes)) would turn off rejection of bytes paths (and be consistent with the rest of `os` and `os.path` where functions accept both types). But the default value for the keyword argument would be a good reminder that str-based paths should be used by default. This str-constraint is also present in the current drafted version, but it is not optional or visible in the signature. (See the diffs of my pull request for an example implementation of this.) So below is the copy-paste from the pull request discussion that I promised: """ Nobody wants to promote using `bytes` for paths without a proper reason. However, `os.fspath(path, type_contraint=str)` is already a compromise from the current `os.*` convention (of symmetrically supporting both `str` and `bytes`) towards enforcing `str`-based paths. As you know, the reason for using `os.fspath` is to switch to a lower-level representation of paths as strings. When you use it, you are already deciding that you are lower-level and want to examine or manipulate path strings manually. I think the `type_constraint=str` keyword-only argument (with the default) is a good way to remind the 'user' that `str` is the way to go unless you know what you are doing and, to `bytes` users, that `os.fspath` (by default) rejects bytes. Looking at the discussions on python-dev, one notices that the active people in the discussions were mostly in favor of exposing the bytes-supporting version in one way or the other. Some were even against bytes-rejecting versions. Most people (Python programmers), especially in the long term, should not be using `os.fspath` a lot. If they want to print a path, they should simply do so: `print(path_obj)`, or `print(f"The path is: {path_obj}")`. However, there will always be people that for whatever reason want to convert a pathlib object into a string, and without even consulting the docs, they may write `str(path_obj)`. And that's not the end of the world. It's not any worse than `str(integer)` for instance. After all, if the main point is to convert something into a generic string, the function should probably be called `str`, not `fspath`. But if the idea is to convert any path object into something that `os` and similar code internally understand, then `os.fspath` sounds fine to me. So I would not go as far as 'never use `str(path)`'. What we don't want is things like `open(str(path), 'w')`. I know you agree, because you wrote that yourself. But once this protocol is available, I see no reason why people would voluntarily wrap their path objects in `str(...)`, so I don't see that as a problem in the future (thanks to this PEP). So, especially in the long term, I don't expect `fspath` to be an everyday tool for people. Even in shorter term, people don't usually need it, because the stdlib will already support pathlib objects. Since it is not an everyday tool, I think it's ok to have that extra keyword-only argument. Even if it were an everyday tool, I can see the keyword-only argument in the signature as useful thing in a documentation sense. [As a side note, I could imagine someone convincing me that `fspath` should *always* accept both `str`- and `bytes`-based paths, because that would simplify it and align it with the rest of `os`. This would not be any worse than the status quo in terms of early failures. That would, however, still not detect accidental use of bytes paths (when forgotten to decode from the encoding it was received in).] """ -- Koos From guido at python.org Wed May 11 13:49:10 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 11 May 2016 10:49:10 -0700 Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3) In-Reply-To: References: Message-ID: On Wed, May 11, 2016 at 10:28 AM, Brett Cannon wrote: > > > On Wed, 11 May 2016 at 09:47 Guido van Rossum wrote: > >> If the authors are happy I'll accept it right away. >> >> (I vaguely recall there's another PEP that's ready for pronouncement -- >> but which one?) >> > > PEP 509 is the only one I can think of. > That's in limbo pending conclusive proof (through benchmarks) that at least one of Yury's patches that depends on it makes a big enough difference. Which IIUC itself is in limbo pending the wordcode changes (we're doing that right?). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dimaqq at gmail.com Wed May 11 13:56:07 2016 From: dimaqq at gmail.com (Dima Tisnek) Date: Wed, 11 May 2016 19:56:07 +0200 Subject: [Python-Dev] Slow downloads from python.org Message-ID: Sorry, this is probably wrong place to ask, but is it only me? I can't get more than 40KB/s downloading from python.org From brett at python.org Wed May 11 14:16:27 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 18:16:27 +0000 Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3) In-Reply-To: References: Message-ID: On Wed, 11 May 2016 at 10:49 Guido van Rossum wrote: > On Wed, May 11, 2016 at 10:28 AM, Brett Cannon wrote: > >> >> >> On Wed, 11 May 2016 at 09:47 Guido van Rossum wrote: >> >>> If the authors are happy I'll accept it right away. >>> >>> (I vaguely recall there's another PEP that's ready for pronouncement -- >>> but which one?) >>> >> >> PEP 509 is the only one I can think of. >> > > That's in limbo pending conclusive proof (through benchmarks) that at > least one of Yury's patches that depends on it makes a big enough > difference. > > Which IIUC itself is in limbo pending the wordcode changes (we're doing > that right?). > Yes. Last I checked the author of the patch was waiting on a review from Serhiy. -Brett > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 11 14:17:29 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 18:17:29 +0000 Subject: [Python-Dev] Slow downloads from python.org In-Reply-To: References: Message-ID: On Wed, 11 May 2016 at 10:56 Dima Tisnek wrote: > Sorry, this is probably wrong place to ask, but is it only me? > I can't get more than 40KB/s downloading from python.org It's just you or the problem has passed; just downloaded much faster than 40KB/s. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 11 16:04:04 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 20:04:04 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: A quick comment about sending me fixes. While I do appreciate them, sending them as a pull request is much easier for me as (a) I don't have to hunt the changes down in the text, and (b) you will see the fixes others have done already to the PEP and I then don't have to figure out what changes have not already been fixed. And honestly, reading the PEP in its rendered format on GitHub is easier IMO than in the text format unless you have something specific to respond to (and even if you do, you can copy and paste the relevant bits into an email reply). On Wed, 11 May 2016 at 09:43 Brett Cannon wrote: > **deep, calming breath** > > Here is the PEP for __fspath__(). The draft lives at > https://github.com/brettcannon/path-pep so feel free to send me PRs for > spelling mistakes, grammatical errors, etc. > > ----- > > PEP: NNN > Title: Adding a file system path protocol > Version: $Revision$ > Last-Modified: $Date$ > Author: Brett Cannon > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 11-May-2016 > Post-History: 11-May-2016 > > > Abstract > ======== > > This PEP proposes a protocol for classes which represent a file system > path to be able to provide a ``str`` or ``bytes`` representation. > Changes to Python's standard library are also proposed to utilize this > protocol where appropriate to facilitate the use of path objects where > historically only ``str`` and/or ``bytes`` file system paths are > accepted. The goal is to allow users to use the representation of a > file system path that's easiest for them now as they migrate towards > using path objects in the future. > > > Rationale > ========= > > Historically in Python, file system paths have been represented as > strings or bytes. This choice of representation has stemmed from C's > own decision to represent file system paths as > ``const char *`` [#libc-open]_. While that is a totally serviceable > format to use for file system paths, it's not necessarily optimal. At > issue is the fact that while all file system paths can be represented > as strings or bytes, not all strings or bytes represent a file system > path. This can lead to issues where any e.g. string duck-types to a > file system path whether it actually represents a path or not. > > To help elevate the representation of file system paths from their > representation as strings and bytes to a more appropriate object > representation, the pathlib module [#pathlib]_ was provisionally > introduced in Python 3.4 through PEP 428. While considered by some as > an improvement over strings and bytes for file system paths, it has > suffered from a lack of adoption. Typically the key issue listed > for the low adoption rate has been the lack of support in the standard > library. This lack of support required users of pathlib to manually > convert path objects to strings by calling ``str(path)`` which many > found error-prone. > > One issue in converting path objects to strings comes from > the fact that only generic way to get a string representation of the > path was to pass the object to ``str()``. This can pose a > problem when done blindly as nearly all Python objects have some > string representation whether they are a path or not, e.g. > ``str(None)`` will give a result that > ``builtins.open()`` [#builtins-open]_ will happily use to create a new > file. > > Exacerbating this whole situation is the > ``DirEntry`` object [#os-direntry]_. While path objects have a > representation that can be extracted using ``str()``, ``DirEntry`` > objects expose a ``path`` attribute instead. Having no common > interface between path objects, ``DirEntry``, and any other > third-party path library had become an issue. A solution that allowed > any path-representing object to declare that is was a path and a way > to extract a low-level representation that all path objects could > support was desired. > > This PEP then proposes to introduce a new protocol to be followed by > objects which represent file system paths. Providing a protocol allows > for clear signalling of what objects represent file system paths as > well as a way to extract a lower-level representation that can be used > with older APIs which only support strings or bytes. > > Discussions regarding path objects that led to this PEP can be found > in multiple threads on the python-ideas mailing list archive > [#python-ideas-archive]_ for the months of March and April 2016 and on > the python-dev mailing list archives [#python-dev-archive]_ during > April 2016. > > > Proposal > ======== > > This proposal is split into two parts. One part is the proposal of a > protocol for objects to declare and provide support for exposing a > file system path representation. The other part is changes to Python's > standard library to support the new protocol. These changes will also > have the pathlib module drop its provisional status. > > > Protocol > -------- > > The following abstract base class defines the protocol for an object > to be considered a path object:: > > import abc > import typing as t > > > class PathLike(abc.ABC): > > """Abstract base class for implementing the file system path > protocol.""" > > @abc.abstractmethod > def __fspath__(self) -> t.Union[str, bytes]: > """Return the file system path representation of the object.""" > raise NotImplementedError > > > Objects representing file system paths will implement the > ``__fspath__()`` method which will return the ``str`` or ``bytes`` > representation of the path. The ``str`` representation is the > preferred low-level path representation as it is human-readable and > what people historically represent paths as. > > > Standard library changes > ------------------------ > > It is expected that most APIs in Python's standard library that > currently accept a file system path will be updated appropriately to > accept path objects (whether that requires code or simply an update > to documentation will vary). The modules mentioned below, though, > deserve specific details as they have either fundamental changes that > empower the ability to use path objects, or entail additions/removal > of APIs. > > > builtins > '''''''' > > ``open()`` [#builtins-open]_ will be updated to accept path objects as > well as continue to accept ``str`` and ``bytes``. > > > os > ''' > > The ``fspath()`` function will be added with the following semantics > (although this is still an open issue: > `Should os.fspath() return bytes?`_):: > > import typing as t > > > def fspath(path: t.Union[PathLike, str]) -> str: > """Return the string representation of the path. > > If a string is passed in then it is returned unchanged. > """ > if hasattr(path, '__fspath__'): > path = path.__fspath__() > if not isinstance(path, str): > type_name = type(path).__name__ > raise TypeError("expected a str or path object, not " + > type_name) > return path > > The ``os.fsencode()`` [#os-fsencode]_ and > ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept > path objects. As both functions coerce their arguments to > ``bytes`` and ``str``, respectively, they will be updated to call > ``__fspath__()`` if present to convert the path object to a ``str`` or > ``bytes`` representation, and then perform their appropriate > coercion operations as if the return value from ``__fspath__()`` had > been the original argument to the coercion function in question. > > The addition of ``os.fspath()``, the updates to > ``os.fsencode()``/``os.fsdecode()``, and the current semantics of > ``pathlib.PurePath`` provide the semantics necessary to > get the path representation one prefers. For a path object, > ``pathlib.PurePath``/``Path`` can be used. If ``str`` is desired and > a ``bytes`` object is considered an error, then ``os.fspath()`` can be > used. If a ``str`` is desired and the encoding of ``bytes`` should be > assumed to be the default file system encoding, then ``os.fsdecode()`` > should be used. Finally, if a ``bytes`` representation is desired and > any strings should be encoded using the default file system encoding > then ``os.fsencode()`` is used. > > This PEP recommends using path objects when possible and falling back > to string paths as necessary. Therefore, no function is provided for > the case of wanting a bytes representation but without any automatic > encoding to help discourage the use of multiple bytes encodings on a > single file system. If it is necessary to deal with an existing file > system directory with entries in a non-default encoding, this can be > done with low-level functions using ``str`` and the PEP 383 > ``surrogateescape`` error handler, or by using ``bytes`` directly. > > Another way to view this is as a hierarchy of file system path > representations (highest- to lowest-level): path -> str -> bytes. The > functions and classes under discussion can all accept objects on the > same level of the hierarchy, but they vary in whether they promote or > demote objects to another level. The ``pathlib.PurePath`` class can > promote a ``str`` to a path object. The ``os.fspath()`` function can > demote a path object to a string, but only if ``__fspath__()`` returns > a string. The ``os.fsdecode()`` function will demote a path object to > a string or promote a ``bytes`` object to a ``str``. The > ``os.fsencode()`` function will demote a path or string object to > ``bytes``. There is no function that provides a way to demote a path > object directly to ``bytes`` and not allow demoting strings. > > The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` > method. It will return the same value as currently found on the > ``path`` attribute of ``DirEntry`` instances. > > > os.path > ''''''' > > The various path-manipulation functions of ``os.path`` [#os-path]_ > will be updated to accept path objects. For polymorphic functions that > accept both bytes and strings, they will be updated to simply use > code very much similar to > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This > will allow for their pre-existing type-checking code to continue to > function. > > During the discussions leading up to this PEP it was suggested that > ``os.path`` not be updated using an "explicit is better than implicit" > argument. The thinking was that since ``__fspath__()`` is polymorphic > itself it may be better to have code working with ``os.path`` extract > the path representation from path objects explicitly. There is also > the consideration that adding support this deep into the low-level OS > APIs will lead to code magically supporting path objects without > requiring any documentation updated, leading to potential complaints > when it doesn't work, unbeknownst to the project author. > > But it is the view of the authors that "practicality beats purity" in > this instance. To help facilitate the transition to supporting path > objects, it is better to make the transition as easy as possible than > to worry about unexpected/undocumented duck typing support for > projects. > > > pathlib > ''''''' > > The ``PathLike`` ABC as discussed in the Protocol_ section will be > added to the pathlib module [#pathlib]_. The constructor for > ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept > path objects. Both ``PurePath`` and ``Path`` will continue to not > accept ``bytes`` path representations, and so if ``__fspath__()`` > returns ``bytes`` it will raise an exception. > > The ``path`` attribute will be removed as this PEP makes its > redundant (it has not been included in any released version of Python > and so is not a backwards-compatibility concern). > > > C API > ''''' > > The C API will gain an equivalent function to ``os.fspath()`` that > also allows bytes objects through:: > > /* > Return the file system path of the object. > > If the object is str or bytes, then allow it to pass through with > an incremented refcount. All other types raise a TypeError. > */ > PyObject * > PyOS_RawFSPath(PyObject *path) > { > if (PyObject_HasAttrString(path, "__fspath__")) { > path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL); > if (path == NULL) { > return NULL; > } > } > else { > Py_INCREF(path); > } > > if (!PyUnicode_Check(path) && !PyBytes_Check(path)) { > Py_DECREF(path); > return PyErr_Format(PyExc_TypeError, > "expected a string, bytes, or path object, > not %S", > path->ob_type); > } > > return path; > } > > > Backwards compatibility > ======================= > > There are no explicit backwards-compatibility concerns. Unless an > object incidentally already defines a ``__fspath__()`` method there is > not reason to expect pre-existing code to break or expect to have > their semantics implicitly changed. > > Libraries wishing to support path objects and a version of Python > prior to Python 3.6 can use the idiom of > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. > > > Open Issues > =========== > > Should os.fspath() return bytes? > -------------------------------- > > Some have argued that ``os.fspath()`` should be configurable so that > the user can specify what types are acceptable (e.g. an argument to > say that bytes are acceptable instead of strings, or both types). > Others have suggested that ``os.fspath()`` match the proposed > semantics of ``PyOS_RawFSPath()``. Both camps argue that use of > ``os.fspath()`` will only be for a transitionary period while more > libraries gain acceptance of path objects, and so being more flexible > in what ``os.fspath()`` works with will help with the transition. The > opponents to this -- which support the currently proposed sematnics -- > worry that being so flexible with accepting bytes will lead to people > not properly considering the ramifications of working with bytes, > especially if bytes are transparently appearing in their code due to > ``os.fspath()``. > > > The name and location of the protocol's ABC > ------------------------------------------- > > The name of the ABC being proposed to represent the protocol has not > been discussed very much, not which module it should exist in. > Names other than ``PathLike`` which are viable are ``PathABC`` > and ``FSPathABC``. The name can't be ``Path`` if the ABC is put into > the pathlib module. > > > Type hint for path-like objects > ------------------------------- > > Creating a proper type hint for APIs that accept path objects as well > as strings and bytes will probably be needed. It could be as simple > as defining ``typing.Path``/``typing.FSPath`` to correspond to the ABC > and then having > ``typing.PathLike = typing.Union[typing.Path, str, bytes]``. The type > hint could also potentially be made to be generic to accept the > specific low-level representation, e.g. ``typing.PathLike[str]``. > > In the end the type hinting solution should be properly discussed > with the right type hinting experts if this is the best approach. > > > Rejected Ideas > ============== > > Other names for the protocol's function > --------------------------------------- > > Various names were proposed during discussions leading to this PEP, > including ``__path__``, ``__pathname__``, and ``__fspathname__``. In > the end people seemed to gravitate towards ``__fspath__`` for being > unambiguous without unnecessarily long. > > > Separate str/bytes methods > -------------------------- > > At one point it was suggested that ``__fspath__()`` only return > strings and another method named ``__fspathb__()`` be introduced to > return bytes. The thinking that by making ``__fspath__()`` not be > polymorphic it could make dealing with the potential string or bytes > representations easier. But the general consensus was that returning > bytes will more than likely be rare and that the various functions in > the os module are the better abstraction to be promoting over direct > calls to ``__fspath__()``. > > > Providing a path attribute > -------------------------- > > To help deal with the issue of ``pathlib.PurePath`` not inheriting > from ``str``, originally it was proposed to introduce a ``path`` > attribute to mirror what ``os.DirEntry`` provides. In the end, though, > it was determined that a protocol would provide the same result while > not directly exposing an API that most people will never need to > interact with directly. > > > Have ``__fspath__()`` only return strings > ------------------------------------------ > > Much of the discussion that led to this PEP revolved around whether > ``__fspath__()`` should be polymorphic and return ``bytes`` as well as > ``str`` instead of only ``str``. The general sentiment for this view > was that ``bytes`` are difficult to work with due to their > inherent lack of information about their encoding, and PEP 383 makes > it possible to represent all file system paths using ``str`` with the > ``surrogateescape`` handler. Thus it would be better to forcibly > promote the use of ``str`` as the low-level path representation for > high-level path objects. > > In the end it was decided that using ``bytes`` to represent paths is > simply not going to go away and thus they should be supported to some > degree. For those not wanting the hassle of working with ``bytes``, > ``os.fspath()`` is provided. > > > A generic string encoding mechanism > ----------------------------------- > > At one point there was discussion of developing a generic mechanism to > extract a string representation of an object that had semantic meaning > (``__str__()`` does not necessarily return anything of semantic > significance beyond what may be helpful for debugging). In the end it > was deemed to lack a motivating need beyond the one this PEP is > trying to solve in a specific fashion. > > > Acknowledgements > ================ > > Thanks to everyone who participated in the various discussions related > to this PEP that spanned both python-ideas and python-dev. Special > thanks to Koos Zevenhoven and Stephen Turnbull for direct feedback on > early drafts of this PEP. > > > References > ========== > > .. [#python-ideas-archive] The python-ideas mailing list archive > (https://mail.python.org/pipermail/python-ideas/) > > .. [#python-dev-archive] The python-dev mailing list archive > (https://mail.python.org/pipermail/python-dev/) > > .. [#libc-open] ``open()`` documention for the C standard library > ( > http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html > ) > > .. [#pathlib] The ``pathlib`` module > (https://docs.python.org/3/library/pathlib.html#module-pathlib) > > .. [#builtins-open] The ``builtins.open()`` function > (https://docs.python.org/3/library/functions.html#open) > > .. [#os-fsencode] The ``os.fsencode()`` function > (https://docs.python.org/3/library/os.html#os.fsencode) > > .. [#os-fsdecode] The ``os.fsdecode()`` function > (https://docs.python.org/3/library/os.html#os.fsdecode) > > .. [#os-direntry] The ``os.DirEntry`` class > (https://docs.python.org/3/library/os.html#os.DirEntry) > > .. [#os-path] The ``os.path`` module > (https://docs.python.org/3/library/os.path.html#module-os.path) > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed May 11 16:44:36 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 May 2016 23:44:36 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On 11.05.16 19:43, Brett Cannon wrote: > os.path > ''''''' > > The various path-manipulation functions of ``os.path`` [#os-path]_ > will be updated to accept path objects. For polymorphic functions that > accept both bytes and strings, they will be updated to simply use > code very much similar to > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This > will allow for their pre-existing type-checking code to continue to > function. I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect. I suggest first implement other changes and then look whether it is worth to add support of path protocol in os.path functions. From ethan at stoneleaf.us Wed May 11 16:51:31 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 May 2016 13:51:31 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: <57339B53.3010506@stoneleaf.us> On 05/11/2016 01:44 PM, Serhiy Storchaka wrote: >> os.path >> ''''''' >> >> The various path-manipulation functions of ``os.path`` [#os-path]_ >> will be updated to accept path objects. For polymorphic functions that >> accept both bytes and strings, they will be updated to simply use >> code very much similar to >> ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This >> will allow for their pre-existing type-checking code to continue to >> function. > > I afraid that this will hit a performance. Some os.path functions are > used in tight loops, they are hard optimized, and adding support of path > protocol can have visible negative effect. Do you have an example of os.path functions being used in a tight loop? -- ~Ethan~ From k7hoven at gmail.com Wed May 11 16:57:37 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 11 May 2016 23:57:37 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On Wed, May 11, 2016 at 11:04 PM, Brett Cannon wrote: > A quick comment about sending me fixes. While I do appreciate them, sending > them as a pull request is much easier for me as (a) I don't have to hunt the > changes down in the text, and (b) you will see the fixes others have done > already to the PEP and I then don't have to figure out what changes have not > already been fixed. And honestly, reading the PEP in its rendered format on > GitHub is easier IMO than in the text format unless you have something > specific to respond to (and even if you do, you can copy and paste the > relevant bits into an email reply). > Personally, I find it more important to settle the open issues first, so I will be looking for typos at a later stage. Besides, more typos may be added in the process. I previously left some typos uncorrected in attempt to keep the diffs of my commits cleaner. By the way, to see what additions and wordings I have suggested, the right way is to look at my PR(s). I might send more PRs to clarify open issues etc., but this will largely be based on my previous commits. -- Koos From ethan at stoneleaf.us Wed May 11 17:15:37 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 May 2016 14:15:37 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <57339B53.3010506@stoneleaf.us> References: <57339B53.3010506@stoneleaf.us> Message-ID: <5733A0F9.7070704@stoneleaf.us> On 05/11/2016 01:51 PM, Ethan Furman wrote: > On 05/11/2016 01:44 PM, Serhiy Storchaka wrote: >>> os.path >>> ''''''' >>> >>> The various path-manipulation functions of ``os.path`` [#os-path]_ >>> will be updated to accept path objects. For polymorphic functions that >>> accept both bytes and strings, they will be updated to simply use >>> code very much similar to >>> ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This >>> will allow for their pre-existing type-checking code to continue to >>> function. >> >> I afraid that this will hit a performance. Some os.path functions are >> used in tight loops, they are hard optimized, and adding support of path >> protocol can have visible negative effect. > > Do you have an example of os.path functions being used in a tight loop? Also, the C code for fspath can check types first and take the fast path if bytes/str are passed in, only falling back to the __fspath__ protocol if something else was passed in -- which should make any performance hits negligible. -- ~Ethan~ From k7hoven at gmail.com Wed May 11 17:22:22 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 00:22:22 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5733A0F9.7070704@stoneleaf.us> References: <57339B53.3010506@stoneleaf.us> <5733A0F9.7070704@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 12:15 AM, Ethan Furman wrote: > On 05/11/2016 01:51 PM, Ethan Furman wrote: >> >> On 05/11/2016 01:44 PM, Serhiy Storchaka wrote: > > >>>> os.path >>>> ''''''' >>>> >>>> The various path-manipulation functions of ``os.path`` [#os-path]_ >>>> will be updated to accept path objects. For polymorphic functions that >>>> accept both bytes and strings, they will be updated to simply use >>>> code very much similar to >>>> ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This >>>> will allow for their pre-existing type-checking code to continue to >>>> function. >>> >>> >>> I afraid that this will hit a performance. Some os.path functions are >>> used in tight loops, they are hard optimized, and adding support of path >>> protocol can have visible negative effect. >> >> >> Do you have an example of os.path functions being used in a tight loop? > I'd be interested in this too. > > Also, the C code for fspath can check types first and take the fast path if > bytes/str are passed in, only falling back to the __fspath__ protocol if > something else was passed in -- which should make any performance hits > negligible. My suggestion for the *python version* already does this too (again, see my PR). This is a win-win, as it also improves error messages (as suggested by Nick in the earlier discussions). -- Koos > > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com From Nikolaus at rath.org Wed May 11 17:28:31 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 11 May 2016 14:28:31 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: (Brett Cannon's message of "Wed, 11 May 2016 16:43:24 +0000") References: Message-ID: <87shxo1dz4.fsf@thinkpad.rath.org> On May 11 2016, Brett Cannon wrote: > This PEP proposes a protocol for classes which represent a file system > path to be able to provide a ``str`` or ``bytes`` representation. [...] As I said before, to me this seems like a lot of effort for a very specific use-case. So let me put forward two hypothetical scenarios to better understand your position: - A new module for URL handling is added to the standard library (or urllib is suitably extended). There is a proposal to add a new protocol that allows classes to provide a ``str`` or ``bytes`` representation of URLs. - A new (third-party) library for natural language processing arises that exposes a specific class for representing audio data. Existing language processing code just uses bytes objects. To ease transition and interoperability, it is proposed to add a new protocol for classes that represend audio data to provide a bytes representation. Do you think you would you be in favor of adding these protocols to the stdlib/languange reference as well? If not, what's the crucial difference to file system paths? Thanks, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From k7hoven at gmail.com Wed May 11 17:42:08 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 00:42:08 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <87shxo1dz4.fsf@thinkpad.rath.org> References: <87shxo1dz4.fsf@thinkpad.rath.org> Message-ID: On Thu, May 12, 2016 at 12:28 AM, Nikolaus Rath wrote: > On May 11 2016, Brett Cannon wrote: >> This PEP proposes a protocol for classes which represent a file system >> path to be able to provide a ``str`` or ``bytes`` representation. > [...] > > As I said before, to me this seems like a lot of effort for a very > specific use-case. So let me put forward two hypothetical scenarios to > better understand your position: I think you are touching important points. > - A new module for URL handling is added to the standard library (or > urllib is suitably extended). There is a proposal to add a new > protocol that allows classes to provide a ``str`` or ``bytes`` > representation of URLs. This reminds me of the thread I recently started on python-ideas about extending the concept of paths to URLs. I don't know if you are referring to something like that or not. Anyway, it would be important to know whether the str or bytes representation is to be used a file system path or an URL, so that would need to be a separate protocol. But everyone would have the experience from these discussions, so hopefully less discussion then :). (By the way, this is one reason to have 'fs' in the name of the __fspath__ method, although I have not mentioned it before.) > > - A new (third-party) library for natural language processing arises > that exposes a specific class for representing audio data. Existing > language processing code just uses bytes objects. To ease transition > and interoperability, it is proposed to add a new protocol for classes > that represend audio data to provide a bytes representation. > > Do you think you would you be in favor of adding these protocols to > the stdlib/languange reference as well? If not, what's the crucial > difference to file system paths? > File system paths are very fundamental, and will probably be used in context of your natural language example too. -- Koos > > Thanks, > -Nikolaus From ethan at stoneleaf.us Wed May 11 17:52:54 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 May 2016 14:52:54 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <87shxo1dz4.fsf@thinkpad.rath.org> References: <87shxo1dz4.fsf@thinkpad.rath.org> Message-ID: <5733A9B6.1000707@stoneleaf.us> On 05/11/2016 02:28 PM, Nikolaus Rath wrote: > On May 11 2016, Brett Cannon wrote: >> This PEP proposes a protocol for classes which represent a file system >> path to be able to provide a ``str`` or ``bytes`` representation. > [...] > > As I said before, to me this seems like a lot of effort for a very > specific use-case. So let me put forward two hypothetical scenarios to > better understand your position: > > - A new module for URL handling is added to the standard library (or > urllib is suitably extended). There is a proposal to add a new > protocol that allows classes to provide a ``str`` or ``bytes`` > representation of URLs. > > - A new (third-party) library for natural language processing arises > that exposes a specific class for representing audio data. Existing > language processing code just uses bytes objects. To ease transition > and interoperability, it is proposed to add a new protocol for classes > that represend audio data to provide a bytes representation. > > Do you think you would you be in favor of adding these protocols to > the stdlib/languange reference as well? If not, what's the crucial > difference to file system paths? I think a crucial reason for this work is to unify the stdlib: we currently have four (?) different things that can be or represent a file-system path: - str - bytes - DirEntry - Path Half of those objects don't work well with the rest of the standard library. As for your second example, the protocol already exists: it's called __bytes__. ;) -- ~Ethan~ From brett at python.org Wed May 11 17:57:59 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 21:57:59 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <87shxo1dz4.fsf@thinkpad.rath.org> References: <87shxo1dz4.fsf@thinkpad.rath.org> Message-ID: On Wed, 11 May 2016 at 14:29 Nikolaus Rath wrote: > On May 11 2016, Brett Cannon wrote: > > This PEP proposes a protocol for classes which represent a file system > > path to be able to provide a ``str`` or ``bytes`` representation. > [...] > > As I said before, to me this seems like a lot of effort for a very > specific use-case. So let me put forward two hypothetical scenarios to > better understand your position: > > - A new module for URL handling is added to the standard library (or > urllib is suitably extended). There is a proposal to add a new > protocol that allows classes to provide a ``str`` or ``bytes`` > representation of URLs. > > - A new (third-party) library for natural language processing arises > that exposes a specific class for representing audio data. Existing > language processing code just uses bytes objects. To ease transition > and interoperability, it is proposed to add a new protocol for classes > that represend audio data to provide a bytes representation. > > Do you think you would you be in favor of adding these protocols to > the stdlib/languange reference as well? Maybe for URLs, not for audio data (at least not in the stdlib; community can do what they want). > If not, what's the crucial > difference to file system paths? > Nearly everyone uses file system paths on a regular basis, less so than URLs but still a good amount of people. Very few people work with audio data. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 11 18:13:52 2016 From: brett at python.org (Brett Cannon) Date: Wed, 11 May 2016 22:13:52 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On Wed, 11 May 2016 at 13:45 Serhiy Storchaka wrote: > On 11.05.16 19:43, Brett Cannon wrote: > > os.path > > ''''''' > > > > The various path-manipulation functions of ``os.path`` [#os-path]_ > > will be updated to accept path objects. For polymorphic functions that > > accept both bytes and strings, they will be updated to simply use > > code very much similar to > > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This > > will allow for their pre-existing type-checking code to continue to > > function. > > I afraid that this will hit a performance. Some os.path functions are > used in tight loops, they are hard optimized, and adding support of path > protocol can have visible negative effect. > As others have asked, what specific examples do you have that os.path is used in a tight loop w/o any I/O that would overwhelm the performance? > > I suggest first implement other changes and then look whether it is > worth to add support of path protocol in os.path functions. > I see this whole discussion breaking down into a few groups which changes what gets done upfront and what might be done farther down the line: 1. Maximum acceptance: do whatever we can to make all representation of paths just work, which means making all places working with a path in the stdlib accept path objects, str, and bytes. 2. Safely use path objects: __fspath__() is there to signal an object is a file system path and to get back a lower-level representation so people stop calling str() on everything, providing some interface signaling that someone doesn't misuse an object as a path and only changing path consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the stdlib. 3. It ain't worth it: those that would rather just skip all of this and drop pathlib from the stdlib. Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to compromise somewhat and find a middle ground in the PEP with the level of changes in the stdlib but being more restrictive with os.fspath(). If I were doing a pure group #2 PEP I would drop os.path changes and make os.fspath() do what Ethan and Koos have suggested and simply pass through without checks whatever path.__fspath__() returned if the argument wasn't str or bytes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed May 11 19:05:32 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 May 2016 16:05:32 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: <5733BABC.3040903@stoneleaf.us> On 05/11/2016 03:13 PM, Brett Cannon wrote: > If [...] I would drop os.path changes and make os.fspath() do what > Ethan and Koos have suggested and simply pass through without checks > whatever path.__fspath__() returned if the argument wasn't str or bytes. Not to derail the conversation too much, as I know we're all getting burned out on the topic, but that last bit is not accurate: my druthers are to have __fspath__ be able to return str /or/ bytes, and if anything else comes from the object in question an exception must be raised. Maybe a word got lost between your thoughts and your fingers -- happens to me all the time. :) -- ~Ethan~ From k7hoven at gmail.com Wed May 11 19:08:03 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 02:08:03 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On Thu, May 12, 2016 at 1:13 AM, Brett Cannon wrote: > > > On Wed, 11 May 2016 at 13:45 Serhiy Storchaka wrote: >> >> On 11.05.16 19:43, Brett Cannon wrote: >> > os.path >> > ''''''' >> > >> > The various path-manipulation functions of ``os.path`` [#os-path]_ >> > will be updated to accept path objects. For polymorphic functions that >> > accept both bytes and strings, they will be updated to simply use >> > code very much similar to >> > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This >> > will allow for their pre-existing type-checking code to continue to >> > function. >> >> I afraid that this will hit a performance. Some os.path functions are >> used in tight loops, they are hard optimized, and adding support of path >> protocol can have visible negative effect. > > > As others have asked, what specific examples do you have that os.path is > used in a tight loop w/o any I/O that would overwhelm the performance? > >> >> >> I suggest first implement other changes and then look whether it is >> worth to add support of path protocol in os.path functions. > > > I see this whole discussion breaking down into a few groups which changes > what gets done upfront and what might be done farther down the line: > > 1. Maximum acceptance: do whatever we can to make all representation of paths > just work, which means making all places working with a path in the stdlib > accept path objects, str, and bytes. Since you are putting me in this camp, there is at least one thing you are wrong about. I don't want all places that work with a path to accept bytes. Only those that already do so, including os/os.path. And yes, I think the stdlib should show a good example in accepting path types (especially those provided in the stdlib itself). Whether Ethan is fully in camp 1, I don't know. Not that I think he would be any closer to the other camps, though. > 2. Safely use path objects: __fspath__() is there to signal an object is a file > system path and to get back a lower-level representation so people stop > calling str() on everything, providing some interface signaling that someone > doesn't misuse an object as a path and only changing path consumptions APIs > -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the > stdlib. > > 3. It ain't worth it: those that would rather just skip all of this and drop > pathlib from the stdlib. > > Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to > compromise somewhat and find a middle ground in the PEP with the level of > changes in the stdlib but being more restrictive with os.fspath(). If I were > doing a pure group #2 PEP I would drop os.path changes and make os.fspath() > do what Ethan and Koos have suggested and simply pass through without checks > whatever path.__fspath__() returned if the argument wasn't str or bytes. > Related to this, based on the earlier discussions, I had the impression that you were largely in the same camp as me. In fact, I thought you had politely left some things out of the PEP draft so I could fill them in. It turned out I was wrong about that, because you didn't merge them. -- Koos > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com > From k7hoven at gmail.com Wed May 11 19:49:30 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 02:49:30 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5733BABC.3040903@stoneleaf.us> References: <5733BABC.3040903@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 2:05 AM, Ethan Furman wrote: > On 05/11/2016 03:13 PM, Brett Cannon wrote: > >> If [...] I would drop os.path changes and make os.fspath() do what > >> Ethan and Koos have suggested and simply pass through without checks >> >> whatever path.__fspath__() returned if the argument wasn't str or bytes. > > > Not to derail the conversation too much, as I know we're all getting burned > out on the topic, but that last bit is not accurate: my druthers are to have > __fspath__ be able to return str /or/ bytes, and if anything else comes from > the object in question an exception must be raised. Maybe a word got lost > between your thoughts and your fingers -- happens to me all the time. :) Yes. This would also be equivalent to my fspath(path, type_constraint=(str,bytes)). And if the compromise I mentioned about the rejecting (by default or optionally) is lifted, the keyword argument would not be needed. I might be ok with throwing away the isinstance check on the return value of __fspath__() if it has significant impact on performance in realistic cases (with DirEntry most likely, I suppose), but I doubt it. -- Koos > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com From k7hoven at gmail.com Wed May 11 19:53:41 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 02:53:41 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <5733BABC.3040903@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven wrote: > On Thu, May 12, 2016 at 2:05 AM, Ethan Furman wrote: >> On 05/11/2016 03:13 PM, Brett Cannon wrote: >> >>> If [...] I would drop os.path changes and make os.fspath() do what >> >>> Ethan and Koos have suggested and simply pass through without checks >>> >>> whatever path.__fspath__() returned if the argument wasn't str or bytes. >> >> >> Not to derail the conversation too much, as I know we're all getting burned >> out on the topic, but that last bit is not accurate: my druthers are to have >> __fspath__ be able to return str /or/ bytes, and if anything else comes from >> the object in question an exception must be raised. Maybe a word got lost >> between your thoughts and your fingers -- happens to me all the time. :) > > Yes. This would also be equivalent to my fspath(path, > type_constraint=(str,bytes)). And if the compromise I mentioned about > the rejecting (by default or optionally) is lifted, the keyword the rejecting -> rejecting bytes Good night. -- Koos > argument would not be needed. I might be ok with throwing away the > isinstance check on the return value of __fspath__() if it has > significant impact on performance in realistic cases (with DirEntry > most likely, I suppose), but I doubt it. > > -- Koos > >> >> -- >> ~Ethan~ >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com From k7hoven at gmail.com Wed May 11 20:00:42 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 03:00:42 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <5733BABC.3040903@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 2:53 AM, Koos Zevenhoven wrote: > On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven wrote: >> On Thu, May 12, 2016 at 2:05 AM, Ethan Furman wrote: >>> On 05/11/2016 03:13 PM, Brett Cannon wrote: >>> >>>> If [...] I would drop os.path changes and make os.fspath() do what >>> >>>> Ethan and Koos have suggested and simply pass through without checks >>>> >>>> whatever path.__fspath__() returned if the argument wasn't str or bytes. >>> >>> >>> Not to derail the conversation too much, as I know we're all getting burned >>> out on the topic, but that last bit is not accurate: my druthers are to have >>> __fspath__ be able to return str /or/ bytes, and if anything else comes from >>> the object in question an exception must be raised. Maybe a word got lost >>> between your thoughts and your fingers -- happens to me all the time. :) >> >> Yes. This would also be equivalent to my fspath(path, >> type_constraint=(str,bytes)). And if the compromise I mentioned about >> the rejecting (by default or optionally) is lifted, the keyword > > the rejecting -> rejecting bytes > >> argument would not be needed. I might be ok with throwing away the >> isinstance check on the return value of __fspath__() if it has >> significant impact on performance in realistic cases (with DirEntry >> most likely, I suppose), but I doubt it. >> >> -- Koos >> I will send a pull request about this tomorrow. -- Koos >>> >>> -- >>> ~Ethan~ >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com From arthur.darcet+list at m4x.org Wed May 11 20:13:32 2016 From: arthur.darcet+list at m4x.org (Arthur Darcet) Date: Thu, 12 May 2016 02:13:32 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <57339B53.3010506@stoneleaf.us> References: <57339B53.3010506@stoneleaf.us> Message-ID: On 11 May 2016 at 22:51, Ethan Furman wrote: > On 05/11/2016 01:44 PM, Serhiy Storchaka wrote: > > os.path >>> ''''''' >>> >>> The various path-manipulation functions of ``os.path`` [#os-path]_ >>> will be updated to accept path objects. For polymorphic functions that >>> accept both bytes and strings, they will be updated to simply use >>> code very much similar to >>> ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This >>> will allow for their pre-existing type-checking code to continue to >>> function. >>> >> >> I afraid that this will hit a performance. Some os.path functions are >> used in tight loops, they are hard optimized, and adding support of path >> protocol can have visible negative effect. >> > > Do you have an example of os.path functions being used in a tight loop? > os.path.getmtime could be used in a tight loop, to sync directories with a lot of files for instance. % python3 -m timeit -s "import os.path; p = 'out'" "hasattr(p, '__fspath__'), os.path.getmtime(p)" 100000 loops, best of 3: 2.67 usec per loop % python3 -m timeit -s "import os.path; p = 'out'" "isinstance(p, (str, bytes)), os.path.getmtime(p)" 100000 loops, best of 3: 2.45 usec per loop % python3 -m timeit -s "import os.path; p = 'out'" "os.path.getmtime(p)" 100000 loops, best of 3: 2.02 usec per loop a 25% markup is a lot imo. a isinstance check prior to the hasattr might be a way to mitigate this a bit (but it doesn't help much) Granted, this example could be optimised by calling os.stat directly, which is not in os.path, but still, worth considering -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed May 11 20:40:22 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 May 2016 17:40:22 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57339B53.3010506@stoneleaf.us> Message-ID: <5733D0F6.9030007@stoneleaf.us> On 05/11/2016 05:13 PM, Arthur Darcet wrote: > os.path.getmtime could be used in a tight loop, to sync directories with > a lot of files for instance. > > % python3 -m timeit -s "import os.path; p = 'out'" "hasattr(p, > '__fspath__'), os.path.getmtime(p)" > 100000 loops, best of 3: 2.67 usec per loop > % python3 -m timeit -s "import os.path; p = 'out'" "isinstance(p, (str, > bytes)), os.path.getmtime(p)" > 100000 loops, best of 3: 2.45 usec per loop > % python3 -m timeit -s "import os.path; p = 'out'" "os.path.getmtime(p)" > 100000 loops, best of 3: 2.02 usec per loop > > a 25% markup is a lot imo. I don't think those results are very informative, since os.path.getmtime() accepts both str and bytes, so it must already have the str/bytes check. -- ~Ethan~ From brett at python.org Wed May 11 21:01:25 2016 From: brett at python.org (Brett Cannon) Date: Thu, 12 May 2016 01:01:25 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On Wed, 11 May 2016 at 16:08 Koos Zevenhoven wrote: > On Thu, May 12, 2016 at 1:13 AM, Brett Cannon wrote: > > > > > > On Wed, 11 May 2016 at 13:45 Serhiy Storchaka > wrote: > >> > >> On 11.05.16 19:43, Brett Cannon wrote: > >> > os.path > >> > ''''''' > >> > > >> > The various path-manipulation functions of ``os.path`` [#os-path]_ > >> > will be updated to accept path objects. For polymorphic functions that > >> > accept both bytes and strings, they will be updated to simply use > >> > code very much similar to > >> > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This > >> > will allow for their pre-existing type-checking code to continue to > >> > function. > >> > >> I afraid that this will hit a performance. Some os.path functions are > >> used in tight loops, they are hard optimized, and adding support of path > >> protocol can have visible negative effect. > > > > > > As others have asked, what specific examples do you have that os.path is > > used in a tight loop w/o any I/O that would overwhelm the performance? > > > >> > >> > >> I suggest first implement other changes and then look whether it is > >> worth to add support of path protocol in os.path functions. > > > > > > I see this whole discussion breaking down into a few groups which changes > > what gets done upfront and what might be done farther down the line: > > > > 1. Maximum acceptance: do whatever we can to make all representation of > paths > > just work, which means making all places working with a path in the > stdlib > > accept path objects, str, and bytes. > > Since you are putting me in this camp, there is at least one thing you > are wrong about. I don't want all places that work with a path to > accept bytes. Only those that already do so, including os/os.path. And > yes, I think the stdlib should show a good example in accepting path > types (especially those provided in the stdlib itself). > That's actually what I meant. I'm not advocating widening the APIs that accept bytes at all. -Brett > > Whether Ethan is fully in camp 1, I don't know. Not that I think he > would be any closer to the other camps, though. > > > 2. Safely use path objects: __fspath__() is there to signal an object is > a file > > system path and to get back a lower-level representation so people stop > > calling str() on everything, providing some interface signaling that > someone > > doesn't misuse an object as a path and only changing path consumptions > APIs > > -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in > the > > stdlib. > > > > 3. It ain't worth it: those that would rather just skip all of this and > drop > > pathlib from the stdlib. > > > > Ethan and Koos are in group #1 and I'm personally in group #2 but I > tried to > > compromise somewhat and find a middle ground in the PEP with the level of > > changes in the stdlib but being more restrictive with os.fspath(). If I > were > > doing a pure group #2 PEP I would drop os.path changes and make > os.fspath() > > do what Ethan and Koos have suggested and simply pass through without > checks > > whatever path.__fspath__() returned if the argument wasn't str or bytes. > > > > Related to this, based on the earlier discussions, I had the > impression that you were largely in the same camp as me. In fact, I > thought you had politely left some things out of the PEP draft so I > could fill them in. It turned out I was wrong about that, because you > didn't merge them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 11 21:03:49 2016 From: brett at python.org (Brett Cannon) Date: Thu, 12 May 2016 01:03:49 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On Wed, 11 May 2016 at 15:13 Brett Cannon wrote: > On Wed, 11 May 2016 at 13:45 Serhiy Storchaka wrote: > >> On 11.05.16 19:43, Brett Cannon wrote: >> > os.path >> > ''''''' >> > >> > The various path-manipulation functions of ``os.path`` [#os-path]_ >> > will be updated to accept path objects. For polymorphic functions that >> > accept both bytes and strings, they will be updated to simply use >> > code very much similar to >> > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This >> > will allow for their pre-existing type-checking code to continue to >> > function. >> >> I afraid that this will hit a performance. Some os.path functions are >> used in tight loops, they are hard optimized, and adding support of path >> protocol can have visible negative effect. >> > > As others have asked, what specific examples do you have that os.path is > used in a tight loop w/o any I/O that would overwhelm the performance? > > >> >> I suggest first implement other changes and then look whether it is >> worth to add support of path protocol in os.path functions. >> > > I see this whole discussion breaking down into a few groups which changes > what gets done upfront and what might be done farther down the line: > > 1. Maximum acceptance: do whatever we can to make all representation > of paths just work, which means making all places working with a path in > the stdlib accept path objects, str, and bytes. > 2. Safely use path objects: __fspath__() is there to signal an object > is a file system path and to get back a lower-level representation so > people stop calling str() on everything, providing some interface signaling > that someone doesn't misuse an object as a path and only changing path > consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g. > os.path -- in the stdlib. > 3. It ain't worth it: those that would rather just skip all of this > and drop pathlib from the stdlib. > > Ethan and Koos are in group #1 and I'm personally in group #2 but I tried > to compromise somewhat and find a middle ground in the PEP with the level > of changes in the stdlib but being more restrictive with os.fspath(). If I > were doing a pure group #2 PEP I would drop os.path changes and make > os.fspath() do what Ethan and Koos have suggested and simply pass through > without checks whatever path.__fspath__() returned if the argument wasn't > str or bytes. > I should mention there's also a side group who really wanted to either minimize bytes usage in all of this or not have a polymorphic os.fspath(), hence why that function is defined the way it is around only returning strings in the PEP. IOW the PEP is already an attempt at compromise and if we can't find a consensus somehow I'll update the PEP to directly reflect my personal preferences and let others write their own competing PEPs if they prefer (but I think it's in everyone's best interests if that's a last resort). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 11 21:05:29 2016 From: brett at python.org (Brett Cannon) Date: Thu, 12 May 2016 01:05:29 +0000 Subject: [Python-Dev] ctypes compatibility with 2.3 In-Reply-To: References: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org> <3283bb48-531a-0795-babd-df203115df58@ctypes.org> Message-ID: On Wed, 11 May 2016 at 09:28 Brett Cannon wrote: > On Wed, 11 May 2016 at 09:07 Thomas Heller wrote: > >> Am 11.05.2016 um 18:04 schrieb Brett Cannon: >> > >> > >> > On Wed, 11 May 2016 at 04:35 Thomas Heller > > > wrote: >> > >> > Am 10.05.2016 um 19:39 schrieb Brett Cannon: >> > > >> > > >> > > On Tue, 10 May 2016 at 01:18 Martin Panter > > >> > > > >>> >> > wrote: >> > > >> > > I am working on , to fix >> shell >> > > injection problems with ctypes.util.find_library(). The >> > proposal for >> > > Python 3 is to change os.popen(shell-script) calls to use >> > > subprocess.Popen(). >> > > >> > > However the Python 2.7 version of the module has a comment >> > which says >> > > ?This file should be kept compatible with Python 2.3, see PEP >> > 291.? >> > > Looking at , it >> is not >> > > clear why we have to maintain this compatibility. My best >> guess is >> > > that there may be an external ctypes package that people >> > want(ed) to >> > > keep compatible with 2.3, and also keep synchronized with 2.7. >> > > >> > > >> > > That's correct and the maintainer is/was Thomas Heller who I have >> > cc'ed >> > > to see if he's okay with lifting the restriction. >> > >> > For me it is totally ok to lift this restriction. >> > >> > >> > Great! I'll also update PEP 291. >> >> Cool. While you're at it, the compatibility restriction for >> modulefinder could also be lifted. >> > > Will do. > PEP 291 no longer lists any restrictions on ctypes or modulefinder. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vadmium+py at gmail.com Wed May 11 23:50:53 2016 From: vadmium+py at gmail.com (Martin Panter) Date: Thu, 12 May 2016 03:50:53 +0000 Subject: [Python-Dev] ctypes compatibility with 2.3 In-Reply-To: References: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org> <3283bb48-531a-0795-babd-df203115df58@ctypes.org> Message-ID: On 12 May 2016 at 01:05, Brett Cannon wrote: > > > On Wed, 11 May 2016 at 09:28 Brett Cannon wrote: >> >> On Wed, 11 May 2016 at 09:07 Thomas Heller wrote: >>> >>> Am 11.05.2016 um 18:04 schrieb Brett Cannon: >>> > >>> > >>> > On Wed, 11 May 2016 at 04:35 Thomas Heller >> > > wrote: >>> > For me it is totally ok to lift this restriction. >>> > >>> > >>> > Great! I'll also update PEP 291. >>> >>> Cool. While you're at it, the compatibility restriction for >>> modulefinder could also be lifted. >> >> >> Will do. > > > PEP 291 no longer lists any restrictions on ctypes or modulefinder. Thanks everyone for your responses. I will look at removing the notices in the code when I get a chance. That would probably involve reverting https://hg.python.org/cpython/rev/381a72ab5fb8 And also the modulefinder.py comment There are also these commits that could be backported https://hg.python.org/cpython/rev/0980034adaa7 (ctypes) https://hg.python.org/cpython/diff/627db59031be/Lib/modulefinder.py but it might be safer just to leave the compatibility code there, perhaps with a clarifying comment. From storchaka at gmail.com Thu May 12 00:32:21 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 12 May 2016 07:32:21 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <57339B53.3010506@stoneleaf.us> References: <57339B53.3010506@stoneleaf.us> Message-ID: On 11.05.16 23:51, Ethan Furman wrote: > On 05/11/2016 01:44 PM, Serhiy Storchaka wrote: >> I afraid that this will hit a performance. Some os.path functions are >> used in tight loops, they are hard optimized, and adding support of path >> protocol can have visible negative effect. > > Do you have an example of os.path functions being used in a tight loop? posixpath.realpath(), os.walk(), glob.glob() calls split() and join() for every path component. dirname() and basename() are also often called. I doesn't count functions like islink() and isfile() since they just pass the argument to underlying stat function and don't need conversion. From storchaka at gmail.com Thu May 12 00:55:59 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 12 May 2016 07:55:59 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On 12.05.16 01:13, Brett Cannon wrote: > > > On Wed, 11 May 2016 at 13:45 Serhiy Storchaka > wrote: > > On 11.05.16 19:43, Brett Cannon wrote: > > os.path > > ''''''' > > > > The various path-manipulation functions of ``os.path`` [#os-path]_ > > will be updated to accept path objects. For polymorphic functions > that > > accept both bytes and strings, they will be updated to simply use > > code very much similar to > > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This > > will allow for their pre-existing type-checking code to continue to > > function. > > I afraid that this will hit a performance. Some os.path functions are > used in tight loops, they are hard optimized, and adding support of path > protocol can have visible negative effect. > > > As others have asked, what specific examples do you have that os.path is > used in a tight loop w/o any I/O that would overwhelm the performance? Most examples does some I/O (like os.lstat()): posixpath.realpath(), os.walk(), glob.glob(). But for example os.walk() was significantly boosted with using os.scandir(), it would be sad to make it slower again. os.path is used in number of files, sometimes in loops, sometimes indirectly. It is hard to find all examples. Such functions as glob.glob() calls split() and join() for every component, but they also use string or bytes operations with paths. So they need to convert argument to str or bytes before start iteration, and always call os.path functions only with str or bytes. Additional conversion in every os.path function is redundant. I suppose most other high-level functions that manipulates paths in a loop also should convert arguments once at the start and don't need the support of path protocol in os.path functions. > I see this whole discussion breaking down into a few groups which > changes what gets done upfront and what might be done farther down the line: > > 1. Maximum acceptance: do whatever we can to make all representation of > paths just work, which means making all places working with a path > in the stdlib accept path objects, str, and bytes. > 2. Safely use path objects: __fspath__() is there to signal an object > is a file system path and to get back a lower-level representation > so people stop calling str() on everything, providing some interface > signaling that someone doesn't misuse an object as a path and only > changing path consumptions APIs -- e.g. open() -- and not path > manipulation APIs -- e.g. os.path -- in the stdlib. > 3. It ain't worth it: those that would rather just skip all of this and > drop pathlib from the stdlib. > > Ethan and Koos are in group #1 and I'm personally in group #2 but I > tried to compromise somewhat and find a middle ground in the PEP with > the level of changes in the stdlib but being more restrictive with > os.fspath(). If I were doing a pure group #2 PEP I would drop os.path > changes and make os.fspath() do what Ethan and Koos have suggested and > simply pass through without checks whatever path.__fspath__() returned > if the argument wasn't str or bytes. I'm for adding conversions in C implemented path consuming APIs and may be in high-level path manipulation functions like os.walk(), but left low-level API of os.path, fnmatch and glob unchanged. From g.brandl at gmx.net Thu May 12 01:30:49 2016 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 12 May 2016 07:30:49 +0200 Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3) In-Reply-To: References: Message-ID: I'm happy with the latest version. Georg On 05/11/2016 06:46 PM, Guido van Rossum wrote: > If the authors are happy I'll accept it right away. > > (I vaguely recall there's another PEP that's ready for pronouncement -- but > which one?) > > On Wed, May 11, 2016 at 9:34 AM, Brett Cannon > wrote: > > Is there anything holding up PEP 515 at this point in terms of acceptance or > implementation? > > On Sat, 19 Mar 2016 at 11:56 Guido van Rossum > wrote: > > All that sounds fine! > > On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah > wrote: > > Guido van Rossum python.org > writes: > >> So should the preprocessing step just be s.replace('_', ''), or should > >> it reject underscores that don't follow the rules from the PEP > >> (perhaps augmented so they follow the spirit of the PEP and the letter > >> of the IBM spec)? > >> > >> Honestly I think it's also fine if specifying this exactly is left out > >> of the PEP, and handled by whoever adds this to Decimal. Having a PEP > >> to work from for the language spec and core builtins (int(), float() > >> complex()) is more important. > > > > I'd keep it simple for Decimal: Remove left and right whitespace (we're > > already doing this), then remove underscores from the remaining string > > (which must not contain any further whitespace), then use the IBM grammar. > > > > > > We could add a clause to the PEP that only those strings that follow > > the spirit of the PEP are guaranteed to be accepted in the future. > > > > > > One reason for keeping it simple is that I would not like to slow down > > string conversion, but thinking about two grammars is also a problem -- > > part of the string conversion in libmpdec is modeled in ACL2, which > > would be invalidated or at least complicated with two grammars. > > > > > > > > Stefan Krah > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido ) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido ) > > From ethan at stoneleaf.us Thu May 12 03:54:14 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 May 2016 00:54:14 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: <573436A6.3000509@stoneleaf.us> On 05/11/2016 09:55 PM, Serhiy Storchaka wrote: > [...] But for example os.walk() was significantly > boosted with using os.scandir(), it would be sad to make it slower > again. scandir's speed improvement is due to not not throwing away data the OS was already giving us. > os.path is used in number of files, sometimes in loops, sometimes > indirectly. It is hard to find all examples. Currently, any of these functions that already take a string have to do a couple pointer comparisons to make sure they have a string; any of these functions that take both a string and a bytes have to do a couple pointer comparisons to make sure they have a string or a bytes; the only difference if this PEP is accepted is the fall-back path when those first checks fail. As an example: if os.walk is called with a Path, it converts the Path to a string (once!) and then uses that string to generate more strings and return strings. When os.walk calls os.path.join or os.path.split it will be with strings (or bytes), not with the original Path object. > Such functions as glob.glob() calls split() and join() for every > component, but they also use string or bytes operations with paths. So > they need to convert argument to str or bytes before start iteration, > and always call os.path functions only with str or bytes. Exactly. > Additional > conversion in every os.path function is redundant. And won't happen, since the fast-path checks will confirm that the argument is a string or bytes object. > I suppose most other > high-level functions that manipulates paths in a loop also should > convert arguments once at the start and don't need the support of path > protocol in os.path functions. Human's can call the os.path functions; therefore, the os.path functions need to support the __fspath__ protocol. > I'm for adding conversions in C implemented path consuming APIs and may > be in high-level path manipulation functions like os.walk(), but left > low-level API of os.path, fnmatch and glob unchanged. So instead of the easy to remember "Path doesn't work with the rest of the standard library" we'll have "Path works with some APIs, but not others -- guess you better look it up" ? That is not an improvement. -- ~Ethan~ From srkunze at mail.de Thu May 12 04:04:42 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 12 May 2016 10:04:42 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: <5734391A.702@mail.de> On 12.05.2016 00:13, Brett Cannon wrote: > I see this whole discussion breaking down into a few groups which > changes what gets done upfront and what might be done farther down the > line: > > 1. Maximum acceptance: do whatever we can to make all representation > of paths just work, which means making all places working with a > path in the stdlib accept path objects, str, and bytes. > 2. Safely use path objects: __fspath__() is there to signal an object > is a file system path and to get back a lower-level representation > so people stop calling str() on everything, providing some > interface signaling that someone doesn't misuse an object as a > path and only changing path consumptions APIs -- e.g. open() -- > and not path manipulation APIs -- e.g. os.path -- in the stdlib. > 3. It ain't worth it: those that would rather just skip all of this > and drop pathlib from the stdlib. > Sorry for being picky here. I think the last group needs to be split up: 3. It ain't worth it: those that would rather just skip all of this 4. drop pathlib from the stdlib. I put myself into camp 3, mostly because I don't consider the "wallet garden problem" a problem at all and I realized that our past issues with pathlib resulted from missing features in pathlib not in the rest of the stdlib. Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu May 12 04:14:38 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 12 May 2016 11:14:38 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <573436A6.3000509@stoneleaf.us> References: <573436A6.3000509@stoneleaf.us> Message-ID: On 12.05.16 10:54, Ethan Furman wrote: > Currently, any of these functions that already take a string have to do > a couple pointer comparisons to make sure they have a string; any of > these functions that take both a string and a bytes have to do a couple > pointer comparisons to make sure they have a string or a bytes; the > only difference if this PEP is accepted is the fall-back path when those > first checks fail. This is cheap in C, but os.path functions are implemented in Python. They have to make at least one function call (os.fspath(), hasattr() or isinstance()), not counting a bytecode for retrieving arguments, resolving attributes, comparing, jumps. Currently os.path functions use tricks to avoid overheads Yet one problem is that currently many os,path functions work with duck-typed strings (e.g. UserString). Using os.fspath() likely limit supported types to str, bytes and types that support the path protocol. From srkunze at mail.de Thu May 12 04:31:18 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 12 May 2016 10:31:18 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: <57343F56.6090609@mail.de> Thanks Brett for your hard work. My comments below: On 11.05.2016 18:43, Brett Cannon wrote: > Rationale > ========= > > Historically in Python, file system paths have been represented as > strings or bytes. This choice of representation has stemmed from C's > own decision to represent file system paths as > ``const char *`` [#libc-open]_. While that is a totally serviceable > format to use for file system paths, it's not necessarily optimal. At > issue is the fact that while all file system paths can be represented > as strings or bytes, not all strings or bytes represent a file system > path. I can remember this argument being made during the discussion. I am not sure if that 100% correct as soon as we talk about PurePaths. > This can lead to issues where any e.g. string duck-types to a > file system path whether it actually represents a path or not. > > To help elevate the representation of file system paths from their > representation as strings and bytes to a more appropriate object > representation, the pathlib module [#pathlib]_ was provisionally > introduced in Python 3.4 through PEP 428. While considered by some as > an improvement over strings and bytes for file system paths, it has > suffered from a lack of adoption. Typically the key issue listed > for the low adoption rate has been the lack of support in the standard > library. This lack of support required users of pathlib to manually > convert path objects to strings by calling ``str(path)`` which many > found error-prone. > > One issue in converting path objects to strings comes from > the fact that only generic way to get a string representation of the > path was to pass the object to ``str()``. This can pose a > problem when done blindly as nearly all Python objects have some > string representation whether they are a path or not, e.g. > ``str(None)`` will give a result that > ``builtins.open()`` [#builtins-open]_ will happily use to create a new > file. > > Exacerbating this whole situation is the > ``DirEntry`` object [#os-direntry]_. While path objects have a > representation that can be extracted using ``str()``, ``DirEntry`` > objects expose a ``path`` attribute instead. Having no common > interface between path objects, ``DirEntry``, and any other > third-party path library had become an issue. A solution that allowed > any path-representing object to declare that is was a path and a way > to extract a low-level representation that all path objects could > support was desired. I think the "Rationale" section ignores the fact the Path also supports the .path attribute now. Which indeed defines a common interface between path objects. > > [...] > > Proposal > ======== > > This proposal is split into two parts. One part is the proposal of a > protocol for objects to declare and provide support for exposing a > file system path representation. https://docs.python.org/3/whatsnew/changelog.html says: "Add ?path? attribute to pathlib.Path objects, returning the same as str(), to make it more similar to DirEntry. Library code can now write getattr(p, ?path?, p) to get the path as a string from a Path, a DirEntry, or a plain string. This is essentially a small one-off protocol." So, in order to promote the "small one-off protocol" to a more broader protocol, this PEP proposes a simple rename of .path to .__fspath__, is that correct? The only issue I see with it is that it requires another function (os.fspath) to extract the "low-level representation". .path seems far easier to me. > The other part is changes to Python's > standard library to support the new protocol. I think this could be another PEP unrelated to the first part. > These changes will also have the pathlib module drop its provisional > status. Not sure if that should be part of the PEP, maybe yes. > [...] The remainder of the PEP unfolds as a flawless implication of the rationale and the proposed idea. Unfortunately, I don't have anything to contribute to the open issues. All solutions have their pros and cons and everything that could be said has been said. I think you need to decide. Sven From k7hoven at gmail.com Thu May 12 05:52:44 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 12:52:44 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <573436A6.3000509@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 11:14 AM, Serhiy Storchaka wrote: > > This is cheap in C, but os.path functions are implemented in Python. They > have to make at least one function call (os.fspath(), hasattr() or > isinstance()), not counting a bytecode for retrieving arguments, resolving > attributes, comparing, jumps. Currently os.path functions use tricks to > avoid overheads > I suppose a C-implemented version of fspath *called from python* might be the fastest option at least in some cases. After all, a function call (isinstance or hasattr) is likely anyway, unless of course `try: path.__fspath__` is used. > Yet one problem is that currently many os,path functions work with > duck-typed strings (e.g. UserString). Using os.fspath() likely limit > supported types to str, bytes and types that support the path protocol. > Something like path = path.__fspath__() if hasattr(path, '__fspath__') else path as currently in the PEP, would not have this problem. However, I wonder whether such duck string paths actually exist (although it does remind me of my earlier experiments for solving the pathlib compatibility problem ;-). -- Koos P.S: I think it's great that you are concerned about the performance, and I find it important. However, this does feel like premature optimization to me at this point. We should first decide what functions should support the protocol, and after that, find the performance concerns and fix them. I have the feeling that the cases where this would be a performance bottleneck are quite rare, but if they are not, they may well be fixable. From k7hoven at gmail.com Thu May 12 06:12:58 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 13:12:58 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <57343F56.6090609@mail.de> References: <57343F56.6090609@mail.de> Message-ID: On Thu, May 12, 2016 at 11:31 AM, Sven R. Kunze wrote: > On 11.05.2016 18:43, Brett Cannon wrote: >> >> Rationale >> ========= >> >> Historically in Python, file system paths have been represented as >> strings or bytes. This choice of representation has stemmed from C's >> own decision to represent file system paths as >> ``const char *`` [#libc-open]_. While that is a totally serviceable >> format to use for file system paths, it's not necessarily optimal. At >> issue is the fact that while all file system paths can be represented >> as strings or bytes, not all strings or bytes represent a file system >> path. > > > I can remember this argument being made during the discussion. I am not sure > if that 100% correct as soon as we talk about PurePaths. > I had suggested an alternative wording for this (see my commit on the work on Rationale). >> Proposal >> ======== >> >> This proposal is split into two parts. One part is the proposal of a >> protocol for objects to declare and provide support for exposing a >> file system path representation. > > > https://docs.python.org/3/whatsnew/changelog.html says: > > "Add ?path? attribute to pathlib.Path objects, returning the same as str(), > to make it more similar to DirEntry. Library code can now write getattr(p, > ?path?, p) to get the path as a string from a Path, a DirEntry, or a plain > string. This is essentially a small one-off protocol." > > So, in order to promote the "small one-off protocol" to a more broader > protocol, this PEP proposes a simple rename of .path to .__fspath__, is that > correct? > Well, I have brought this up previously several times. Indeed I see this as a further development of that duck-typing compatiblity approach. However, while the .path attribute is prior art, it has not been in a release yet. > Unfortunately, I don't have anything to contribute to the open issues. All > solutions have their pros and cons and everything that could be said has > been said. I think you need to decide. > Surprising enough, there are new things being said all the time. But luckily there seem to be signs of convergence. -- Koos From ncoghlan at gmail.com Thu May 12 08:04:02 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 12 May 2016 22:04:02 +1000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On 12 May 2016 at 02:43, Brett Cannon wrote: > **deep, calming breath** > > Here is the PEP for __fspath__(). The draft lives at > https://github.com/brettcannon/path-pep so feel free to send me PRs for > spelling mistakes, grammatical errors, etc. Thanks for putting this together :) > C API > ''''' > > The C API will gain an equivalent function to ``os.fspath()`` that > also allows bytes objects through:: > > /* > Return the file system path of the object. > > If the object is str or bytes, then allow it to pass through with > an incremented refcount. All other types raise a TypeError. > */ > PyObject * > PyOS_RawFSPath(PyObject *path) > { > if (PyObject_HasAttrString(path, "__fspath__")) { > path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL); > if (path == NULL) { > return NULL; > } > } > else { > Py_INCREF(path); > } > > if (!PyUnicode_Check(path) && !PyBytes_Check(path)) { > Py_DECREF(path); > return PyErr_Format(PyExc_TypeError, > "expected a string, bytes, or path object, > not %S", > path->ob_type); > } > > return path; > } I'd still like to see this exposed to Python code as os._raw_fspath() (with the leading underscore just meaning "this probably isn't the API you want" rather than indicating a private or unstable API), and then fspath() defined as a wrapper around it which disallows bytes as output. However, I don't have a specific use case, and it would be straightforward to add later, so the overall PEP gets a +1 from me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From k7hoven at gmail.com Thu May 12 08:40:21 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 15:40:21 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On Thu, May 12, 2016 at 3:04 PM, Nick Coghlan wrote: > > I'd still like to see this exposed to Python code as os._raw_fspath() > (with the leading underscore just meaning "this probably isn't the API > you want" rather than indicating a private or unstable API), and then > fspath() defined as a wrapper around it which disallows bytes as > output. > I don't remember (should probably check) if you previously proposed implementing exactly that in C, but I indeed agree with what you write above, except that I don't like the "_raw_" prefix in the name. I would be willing to call that simply fspath though, since as mentioned before in this thread (I think by Brett and me), the reasons for rejecting bytes in fspath are really quite minor. From ncoghlan at gmail.com Thu May 12 09:20:25 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 12 May 2016 23:20:25 +1000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On 12 May 2016 at 22:40, Koos Zevenhoven wrote: > On Thu, May 12, 2016 at 3:04 PM, Nick Coghlan wrote: >> >> I'd still like to see this exposed to Python code as os._raw_fspath() >> (with the leading underscore just meaning "this probably isn't the API >> you want" rather than indicating a private or unstable API), and then >> fspath() defined as a wrapper around it which disallows bytes as >> output. > > I don't remember (should probably check) if you previously proposed > implementing exactly that in C, but I indeed agree with what you write > above, except that I don't like the "_raw_" prefix in the name. I > would be willing to call that simply fspath though, since as mentioned > before in this thread (I think by Brett and me), the reasons for > rejecting bytes in fspath are really quite minor. It's not unusual for me to encounter "POSIX oughtta be enough for anyone" folks that are not yet entirely convinced that bytes-are-not-text, so I'm actually in favour of making the default Python-level API str-only as a healthy nudge away from the "text-is-just-bytes-with-an-encoding!" school of thought. However, in terms of the three groups Brett articulated (maximum flexibility, encouraging cross-platform correctness, and forgetting the whole idea), I'm in both camps 1 & 2 - I work with POSIX enough that I'm entirely on board with the notion that if you're specifically modelling *POSIX* paths, then bytes-with-an-assumed-encoding is frequently a good enough representation, but also deal with other environments (like Windows, the JVM and the CLR) enough to know that that particular representation of filesystem paths breaks down the moment you expand your scope of interest beyond *nix platforms. Hence the suggestion of having os.fspath() be the group 2 guaranteed-to-only-emit-str API, with os._raw_fspath() as the lower level "I know I'm potentially being POSIX-centric here, but that's OK for my use case" group 1 interface. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From arigo at tunes.org Thu May 12 09:39:11 2016 From: arigo at tunes.org (Armin Rigo) Date: Thu, 12 May 2016 15:39:11 +0200 Subject: [Python-Dev] PyGC_Collect ignores state of `enabled` In-Reply-To: References: Message-ID: Hi Lukasz, On 10 May 2016 at 04:13, ?ukasz Langa wrote: > However, because of PyGC_Collect() called in Py_Finalize(), during > interpreter shutdown the collection is done anyway, Linux does CoW and the > memory usage spikes. Which is ironic on process shutdown. Try to call os._exit() to avoid doing all this work on shutdown (after you have checked that it is indeed not doing anything interesting). A bient?t, Armin. From k7hoven at gmail.com Thu May 12 09:41:26 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 16:41:26 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: On Thu, May 12, 2016 at 4:20 PM, Nick Coghlan wrote: > > It's not unusual for me to encounter "POSIX oughtta be enough for > anyone" folks that are not yet entirely convinced that > bytes-are-not-text, so I'm actually in favour of making the default > Python-level API str-only as a healthy nudge away from the > "text-is-just-bytes-with-an-encoding!" school of thought. > This was also how I convinced myself about the default str constraint. However, I'm afraid it would be a weak weapon against using bytes paths, since the people using bytes paths would not be likely to call it, regardless of whether it supports bytes or not. The nice thing is that pathlib is str-only and *that* will push people away from bytes paths. > However, in terms of the three groups Brett articulated (maximum > flexibility, encouraging cross-platform correctness, and forgetting > the whole idea), I'm in both camps 1 & 2 - I work with POSIX enough > that I'm entirely on board with the notion that if you're specifically > modelling *POSIX* paths, then bytes-with-an-assumed-encoding is > frequently a good enough representation, but also deal with other > environments (like Windows, the JVM and the CLR) enough to know that > that particular representation of filesystem paths breaks down the > moment you expand your scope of interest beyond *nix platforms. > I also agree with parts about Brett's "camp 2". -- Koos From dimaqq at gmail.com Thu May 12 10:45:34 2016 From: dimaqq at gmail.com (Dima Tisnek) Date: Thu, 12 May 2016 16:45:34 +0200 Subject: [Python-Dev] Slow downloads from python.org In-Reply-To: References: Message-ID: Gone now, must've been transient. I've no idea if it was python end, my end (tail?) or something slithery inbetween. On 11 May 2016 at 20:17, Brett Cannon wrote: > > > On Wed, 11 May 2016 at 10:56 Dima Tisnek wrote: >> >> Sorry, this is probably wrong place to ask, but is it only me? >> I can't get more than 40KB/s downloading from python.org > > > It's just you or the problem has passed; just downloaded much faster than > 40KB/s. From gvanrossum at gmail.com Thu May 12 11:11:37 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 12 May 2016 08:11:37 -0700 Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3) In-Reply-To: References: Message-ID: OK, then PEP 515 is now officially accepted! Congratulations. Start the implementation work! --Guido (mobile) On May 11, 2016 10:33 PM, "Georg Brandl" wrote: I'm happy with the latest version. Georg On 05/11/2016 06:46 PM, Guido van Rossum wrote: > If the authors are happy I'll accept it right away. > > (I vaguely recall there's another PEP that's ready for pronouncement -- but > which one?) > > On Wed, May 11, 2016 at 9:34 AM, Brett Cannon > wrote: > > Is there anything holding up PEP 515 at this point in terms of acceptance or > implementation? > > On Sat, 19 Mar 2016 at 11:56 Guido van Rossum > wrote: > > All that sounds fine! > > On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah > wrote: > > Guido van Rossum python.org > writes: > >> So should the preprocessing step just be s.replace('_', ''), or should > >> it reject underscores that don't follow the rules from the PEP > >> (perhaps augmented so they follow the spirit of the PEP and the letter > >> of the IBM spec)? > >> > >> Honestly I think it's also fine if specifying this exactly is left out > >> of the PEP, and handled by whoever adds this to Decimal. Having a PEP > >> to work from for the language spec and core builtins (int(), float() > >> complex()) is more important. > > > > I'd keep it simple for Decimal: Remove left and right whitespace (we're > > already doing this), then remove underscores from the remaining string > > (which must not contain any further whitespace), then use the IBM grammar. > > > > > > We could add a clause to the PEP that only those strings that follow > > the spirit of the PEP are guaranteed to be accepted in the future. > > > > > > One reason for keeping it simple is that I would not like to slow down > > string conversion, but thinking about two grammars is also a problem -- > > part of the string conversion in libmpdec is modeled in ACL2, which > > would be invalidated or at least complicated with two grammars. > > > > > > > > Stefan Krah > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido ) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido ) > > _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu May 12 11:42:36 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 May 2016 08:42:36 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <57343F56.6090609@mail.de> References: <57343F56.6090609@mail.de> Message-ID: <5734A46C.40306@stoneleaf.us> On 05/12/2016 01:31 AM, Sven R. Kunze wrote: > I think the "Rationale" section ignores the fact the Path also supports > the .path attribute now. Which indeed defines a common interface between > path objects. The version of Python that has Path.path has not been released yet. And even so, .path is not a "common interface" as neither str nor bytes have it, and they also are used as path objects. And even given all that, for smoother interoperability with the rest of the stdlib, or at least the os.* portion, those functions would still need to be upgraded to check for .path on the incoming arguments -- at which point we may as well make a protocol to properly support file system paths instead of relying on the rather generic attribute name of 'path'. -- ~Ethan~ From srkunze at mail.de Thu May 12 12:13:08 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 12 May 2016 18:13:08 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <87shxo1dz4.fsf@thinkpad.rath.org> Message-ID: <5734AB94.4010807@mail.de> On 11.05.2016 23:57, Brett Cannon wrote: > On Wed, 11 May 2016 at 14:29 Nikolaus Rath > wrote: > > On May 11 2016, Brett Cannon > wrote: > > This PEP proposes a protocol for classes which represent a file > system > > path to be able to provide a ``str`` or ``bytes`` representation. > [...] > > As I said before, to me this seems like a lot of effort for a very > specific use-case. > Exactly. Especially when considering what else can be done to improve the situation considerably. > So let me put forward two hypothetical scenarios to > better understand your position: > > - A new module for URL handling is added to the standard library (or > urllib is suitably extended). There is a proposal to add a new > protocol that allows classes to provide a ``str`` or ``bytes`` > representation of URLs. > > - A new (third-party) library for natural language processing arises > that exposes a specific class for representing audio data. Existing > language processing code just uses bytes objects. To ease transition > and interoperability, it is proposed to add a new protocol for > classes > that represend audio data to provide a bytes representation. > You can even add the timedelta-to-seconds protocol that somebody thought would be good idea: https://mail.python.org/pipermail/python-dev/2016-April/144018.html https://mail.python.org/pipermail/python-ideas/2016-May/040226.html The generalization is straight-forward and a result of this discussion. If it works and is a good idea for pathlib, then there's absolutely no reason not to do this for the datetime lib and other rich-object libs. Same goes the other way round. Question still is: is it a good idea? Maybe, it will become a successful pattern. Maybe not. > Do you think you would you be in favor of adding these protocols to > the stdlib/languange reference as well? > > > Maybe for URLs, not for audio data (at least not in the stdlib; > community can do what they want). > > If not, what's the crucial > difference to file system paths? > > > Nearly everyone uses file system paths on a regular basis, less so > than URLs but still a good amount of people. Very few people work with > audio data. Amount of usage should be taken into account of course. However, question remains if that suffices as a justification for the effort. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 12 12:24:04 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 12 May 2016 09:24:04 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5734A46C.40306@stoneleaf.us> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: I am glad this is finally happening. There's quite a bit of noise in the thread which I have to ignore. The two issues that I want to respond to are speed and whether os.fspath() can return bytes. - Speed: We should trust our ability to optimize the implementations where necessary. First the API issues need to be settled. - Bytes: I strongly believe that os.fspath() should be a thin wrapper around the __fspath__ protocol, like next() wraps the .__next__ protocol. It should not get into bytes vs. string politics. If your app really needs strings, call os.fsdecode(). So this is my version (unoptimized): def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...) Other than that I think the PEP is already in fine shape. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Thu May 12 12:26:14 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 12 May 2016 18:26:14 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5734A46C.40306@stoneleaf.us> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: <5734AEA6.5010308@mail.de> On 12.05.2016 17:42, Ethan Furman wrote: > On 05/12/2016 01:31 AM, Sven R. Kunze wrote: > >> I think the "Rationale" section ignores the fact the Path also supports >> the .path attribute now. Which indeed defines a common interface between >> path objects. > > The version of Python that has Path.path has not been released yet. > And even so, .path is not a "common interface" as neither str nor > bytes have it, and they also are used as path objects. str and bytes will receive the __fspath__ attribute when this PEP is accepted? > And even given all that, for smoother interoperability with the rest > of the stdlib, or at least the os.* portion, those functions would > still need to be upgraded to check for .path on the incoming arguments > -- at which point we may as well make a protocol to properly support > file system paths instead of relying on the rather generic attribute > name of 'path'. Just so, if you accept changing os.* as a necessary solution. If not, keeping .path would suffice and would be much simpler. Best, Sven From ethan at stoneleaf.us Thu May 12 12:56:39 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 May 2016 09:56:39 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5734AEA6.5010308@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <5734AEA6.5010308@mail.de> Message-ID: <5734B5C7.4060206@stoneleaf.us> On 05/12/2016 09:26 AM, Sven R. Kunze wrote: > On 12.05.2016 17:42, Ethan Furman wrote: >> On 05/12/2016 01:31 AM, Sven R. Kunze wrote: >>> I think the "Rationale" section ignores the fact the Path also supports >>> the .path attribute now. Which indeed defines a common interface between >>> path objects. >> >> The version of Python that has Path.path has not been released yet. >> And even so, .path is not a "common interface" as neither str nor >> bytes have it, and they also are used as path objects. > > str and bytes will receive the __fspath__ attribute when this PEP is > accepted? No, they won't. The __fspath__ protocol will reduce the rich path object down to a str/bytes object. One could argue that a .path attribute is similar, but consider: if you are handed a random object with a .path attribute, how certain can you be that it represents a file system path? Contrariwise, how certain can you be of an object that has __fspath__? At any rate, we seem to be down to the details of os.fspath() so I don't see any reason to discuss .path any further. -- ~Ethan~ From brett at python.org Thu May 12 13:18:45 2016 From: brett at python.org (Brett Cannon) Date: Thu, 12 May 2016 17:18:45 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: On Thu, 12 May 2016 at 09:25 Guido van Rossum wrote: > I am glad this is finally happening. There's quite a bit of noise in the > thread which I have to ignore. > Don't worry, I'm not ignoring it on your behalf. :) > The two issues that I want to respond to are speed and whether os.fspath() > can return bytes. > > - Speed: We should trust our ability to optimize the implementations where > necessary. First the API issues need to be settled. > Added a note to the PEP to say perf isn't a worry for os.path. > > - Bytes: I strongly believe that os.fspath() should be a thin wrapper > around the __fspath__ protocol, like next() wraps the .__next__ protocol. > It should not get into bytes vs. string politics. If your app really needs > strings, call os.fsdecode(). So this is my version (unoptimized): > > def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: > if isinstance(p, (str, bytes)): > return p > try: > return p.__fspath__ > except AttributeError: > raise TypeError(...) > > Other than that I think the PEP is already in fine shape. > Just to double-check, did you mean for __fspath__ to only be an attribute in your example, or did you leave off the `()` by accident? As of right now the PEP is proposing a method for the protocol to follow common practice of using methods and in case the representation is not always pre-computed and thus not necessarily giving the wrong impression that the attribute access is cheap. But admittedly an attribute was previously proposed and there wasn't a terribly strong argument against it beyond "we historically haven't done it that way", so I'm open to swapping to an attribute if that's your preference. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Thu May 12 13:21:45 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 12 May 2016 19:21:45 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5734B5C7.4060206@stoneleaf.us> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <5734AEA6.5010308@mail.de> <5734B5C7.4060206@stoneleaf.us> Message-ID: <5734BBA9.2050807@mail.de> On 12.05.2016 18:56, Ethan Furman wrote: > On 05/12/2016 09:26 AM, Sven R. Kunze wrote: >> str and bytes will receive the __fspath__ attribute when this PEP is >> accepted? > > No, they won't. The __fspath__ protocol will reduce the rich path > object down to a str/bytes object. Would this make the implementation of os.fspath simpler? After all, str and bytes are to some extend path-like objects. Best, Sven From ethan at stoneleaf.us Thu May 12 13:27:21 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 May 2016 10:27:21 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5734BBA9.2050807@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <5734AEA6.5010308@mail.de> <5734B5C7.4060206@stoneleaf.us> <5734BBA9.2050807@mail.de> Message-ID: <5734BCF9.6060305@stoneleaf.us> On 05/12/2016 10:21 AM, Sven R. Kunze wrote: > On 12.05.2016 18:56, Ethan Furman wrote: >> On 05/12/2016 09:26 AM, Sven R. Kunze wrote: >>> str and bytes will receive the __fspath__ attribute when this PEP is >>> accepted? >> >> No, they won't. The __fspath__ protocol will reduce the rich path >> object down to a str/bytes object. > > Would this make the implementation of os.fspath simpler? Maybe, but a bad idea for two reasons: 1) Reducing a str to the exact same str is silly; and, more importantly 2) not every str/bytes is a path -- ~Ethan~ From sjoerdjob at sjoerdjob.com Thu May 12 13:22:25 2016 From: sjoerdjob at sjoerdjob.com (Sjoerd Job Postmus) Date: Thu, 12 May 2016 19:22:25 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: Message-ID: <20160512172225.GA16846@sjoerdjob.com> I would like to make just 1 comment regarding the question of accepting (or not) bytes as output of `os.fspath`. The whole point of adding `os.fspath` is to make it easier to use Path objects. This is in an effort to gain greater adoption of pathlib in libraries. Now, this is an excellent idea. However, if it were to reject bytes, that would mean that when libraries start to use pathlib, it would suddenly become harder for people that actually need bytes-support to use pathlib. Now, the claim 'if you need bytes, you should not be using pathlib` is a reasonable one. But what if I need bytes *and* a specific library (say, image handling, or a web framework, or ...). It's not up to me if that library uses pathlib or plain old os.path.join. Is using surrogate-escapes enough for this case? I myself am not sure, (and also not affected), but it sounds to me that rejecting bytes is a wrong approach if there is no proper workaround (assuming the use-case of pathlib is somewhere deep in library code). From k7hoven at gmail.com Thu May 12 14:11:18 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 21:11:18 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 7:24 PM, Guido van Rossum wrote: > I am glad this is finally happening. There's quite a bit of noise in the > thread which I have to ignore. The two issues that I want to respond to are > speed and whether os.fspath() can return bytes. > > - Speed: We should trust our ability to optimize the implementations where > necessary. First the API issues need to be settled. > > - Bytes: I strongly believe that os.fspath() should be a thin wrapper around > the __fspath__ protocol, like next() wraps the .__next__ protocol. It should > not get into bytes vs. string politics. If your app really needs strings, > call os.fsdecode(). So this is my version (unoptimized): > :) Thank you for this. I can breathe now. Some questions remain: > def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: > if isinstance(p, (str, bytes)): > return p > try: > return p.__fspath__ > except AttributeError: > raise TypeError(...) > (I know Brett already posted this question, but somehow it did not show up in my mailbox before I had written this. I'm (re)posting because there is some stuff here that is not in Brett's email ) You might be suggesting that __fspath__ should be an attribute, not a method, or did you mean something like: def fspath(p): if isinstance(p, (str, bytes)): return p try: p.__fspath__ except AttributeError: raise TypeError(...) return p.__fspath__() IMO, either is fine, I suppose. As you know, it's mostly a question of whether __fspath__ will be a property or a method (on PurePath for instance). But if you meant the former, that would change also the ABC and the protocol description. -- Koos From k7hoven at gmail.com Thu May 12 14:32:35 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 12 May 2016 21:32:35 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <20160512172225.GA16846@sjoerdjob.com> References: <20160512172225.GA16846@sjoerdjob.com> Message-ID: On Thu, May 12, 2016 at 8:22 PM, Sjoerd Job Postmus wrote: > I would like to make just 1 comment regarding the question of accepting > (or not) bytes as output of `os.fspath`. > > The whole point of adding `os.fspath` is to make it easier to use Path > objects. This is in an effort to gain greater adoption of pathlib in > libraries. Now, this is an excellent idea. > > However, if it were to reject bytes, that would mean that when libraries > start to use pathlib, it would suddenly become harder for people that > actually need bytes-support to use pathlib. > > Now, the claim 'if you need bytes, you should not be using pathlib` is a > reasonable one. But what if I need bytes *and* a specific library (say, > image handling, or a web framework, or ...). It's not up to me if that > library uses pathlib or plain old os.path.join. > > Is using surrogate-escapes enough for this case? I myself am not sure, > (and also not affected), but it sounds to me that rejecting bytes is a > wrong approach if there is no proper workaround (assuming the use-case > of pathlib is somewhere deep in library code). > This is out of the scope of this PEP and probably a very insignificant issue (luckily, this is not the pathlib PEP). Surrogates will probably work and if not, on can "blaim" broken filenames ;). -- Koos From guido at python.org Thu May 12 14:35:40 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 12 May 2016 11:35:40 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 10:18 AM, Brett Cannon wrote: > > On Thu, 12 May 2016 at 09:25 Guido van Rossum wrote: > >> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: >> if isinstance(p, (str, bytes)): >> return p >> try: >> return p.__fspath__ >> except AttributeError: >> raise TypeError(...) >> > >> Other than that I think the PEP is already in fine shape. >> > > - Bytes: I strongly believe that os.fspath() should be a thin wrapper > around the __fspath__ protocol, like next() wraps the .__next__ protocol. > It should not get into bytes vs. string politics. If your app really needs > strings, call os.fsdecode(). So this is my version (unoptimized): > > Just to double-check, did you mean for __fspath__ to only be an attribute > in your example, or did you leave off the `()` by accident? As of right now > the PEP is proposing a method for the protocol to follow common practice of > using methods and in case the representation is not always pre-computed and > thus not necessarily giving the wrong impression that the attribute access > is cheap. But admittedly an attribute was previously proposed and there > wasn't a terribly strong argument against it beyond "we historically > haven't done it that way", so I'm open to swapping to an attribute if > that's your preference. > >> Whoops. Didn't mean to change that! Yes, __fspath__ should remain a method. You can breathe again. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu May 12 14:51:55 2016 From: brett at python.org (Brett Cannon) Date: Thu, 12 May 2016 18:51:55 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: On Thu, 12 May 2016 at 11:36 Guido van Rossum wrote: > On Thu, May 12, 2016 at 10:18 AM, Brett Cannon wrote: > >> On Thu, 12 May 2016 at 09:25 Guido van Rossum wrote: >> > def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: >>> if isinstance(p, (str, bytes)): >>> return p >>> try: >>> return p.__fspath__ >>> except AttributeError: >>> raise TypeError(...) >>> >> >>> Other than that I think the PEP is already in fine shape. >>> >> >> - Bytes: I strongly believe that os.fspath() should be a thin wrapper >> around the __fspath__ protocol, like next() wraps the .__next__ protocol. >> It should not get into bytes vs. string politics. If your app really needs >> strings, call os.fsdecode(). So this is my version (unoptimized): >> >> Just to double-check, did you mean for __fspath__ to only be an attribute >> in your example, or did you leave off the `()` by accident? As of right now >> the PEP is proposing a method for the protocol to follow common practice of >> using methods and in case the representation is not always pre-computed and >> thus not necessarily giving the wrong impression that the attribute access >> is cheap. But admittedly an attribute was previously proposed and there >> wasn't a terribly strong argument against it beyond "we historically >> haven't done it that way", so I'm open to swapping to an attribute if >> that's your preference. >> >>> > Whoops. Didn't mean to change that! Yes, __fspath__ should remain a > method. You can breathe again. :-) > That's a mechanical change so not exactly the most stressful aspect of this PEP. :) I'll add the attribute angle to the Rejected Ideas, though. Anyway, with your strong preference of how to tweak os.fspath() what specifically would you like to see discussed at this point? Assuming the os.fspath() -> bytes discussion is dealt with, the only open issues listed in the PEP are the naming and placement of the ABC and how to do type hints for all of this (thanks to the dichotomy of path objects using the protocol and path-like objects which is the union of path object, str, and bytes and the joy of trying to name all of this well). -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 12 15:02:55 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 12 May 2016 12:02:55 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 11:51 AM, Brett Cannon wrote: > > Anyway, with your strong preference of how to tweak os.fspath() what > specifically would you like to see discussed at this point? > Preferably nothing. :-) There's been too much discussion already. > Assuming the os.fspath() -> bytes discussion is dealt with, the only open > issues listed in the PEP are the naming and placement of the ABC and how to > do type hints for all of this (thanks to the dichotomy of path objects > using the protocol and path-like objects which is the union of path object, > str, and bytes and the joy of trying to name all of this well). > Name and placement: I think it belongs in os, and os.PathLike sounds like a fine name. (I'm surprised that os.DirEntry doesn't exist.) Typing: do you want it to be a generic class? If not, the types can be left out of the stdlib and only put in the stub (though you can show them in the PEP of course). If you want it to be generic we have more work to do. I'm out of time now but we can discuss that after 3pm today. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu May 12 15:18:12 2016 From: brett at python.org (Brett Cannon) Date: Thu, 12 May 2016 19:18:12 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: On Thu, 12 May 2016 at 12:03 Guido van Rossum wrote: > On Thu, May 12, 2016 at 11:51 AM, Brett Cannon wrote: > >> >> Anyway, with your strong preference of how to tweak os.fspath() what >> specifically would you like to see discussed at this point? >> > > Preferably nothing. :-) There's been too much discussion already. > Works for me. :) I'll update the PEP with the new semantics for os.fspath() and send out the updated version later today. > > >> Assuming the os.fspath() -> bytes discussion is dealt with, the only open >> issues listed in the PEP are the naming and placement of the ABC and how to >> do type hints for all of this (thanks to the dichotomy of path objects >> using the protocol and path-like objects which is the union of path object, >> str, and bytes and the joy of trying to name all of this well). >> > > Name and placement: I think it belongs in os, and os.PathLike sounds like > a fine name. (I'm surprised that os.DirEntry doesn't exist.) > SGTM. And os.DirEntry doesn't exist simply because it's posix.DirEntry since it's implemented entirely in C. We could add an alias but since it isn't constructed from scratch I don't think it's worth it. > > Typing: do you want it to be a generic class? If not, the types can be > left out of the stdlib and only put in the stub (though you can show them > in the PEP of course). > If we aren't going to restrict what os.fspath() returns then I don't see any need to make the type generic (I mean technically a generic version might be nice for e.g. the constructor of pathlib only taking strings, but it's probably overkill). I guess my real question is whether we want to create typing.PathLike to match os.PathLike? And it sounds like you don't want to bother with a potential second type that corresponds to Union[str, bytes, PathLike]. > > If you want it to be generic we have more work to do. I'm out of time now > but we can discuss that after 3pm today. > Sure thing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 12 15:20:53 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 12 May 2016 12:20:53 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: There's no need for typing.PathLike. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu May 12 15:30:45 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 May 2016 12:30:45 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <20160512172225.GA16846@sjoerdjob.com> References: <20160512172225.GA16846@sjoerdjob.com> Message-ID: <5734D9E5.6060701@stoneleaf.us> On 05/12/2016 10:22 AM, Sjoerd Job Postmus wrote: > However, if it were to reject bytes, that would mean that when libraries > start to use pathlib, it would suddenly become harder for people that > actually need bytes-support to use pathlib. pathlib is not about bytes support. While bytes are necessary in this digital world we live in, most things that look like text should be text, and that includes most paths. > > Now, the claim 'if you need bytes, you should not be using pathlib` is a > reasonable one. But what if I need bytes *and* a specific library (say, > image handling, or a web framework, or ...). It's not up to me if that > library uses pathlib or plain old os.path.join. If you need bytes support for your paths, there's at least one [1] that has that support. -- ~Ethan~ [1] Yeah, it's mine [2]. ;) I haven't checked if the other third-party libs do or not. [2] https://pypi.python.org/pypi/antipathy From brett at python.org Thu May 12 16:53:12 2016 From: brett at python.org (Brett Cannon) Date: Thu, 12 May 2016 20:53:12 +0000 Subject: [Python-Dev] File system path PEP, part 2 Message-ID: Second draft that takes Guido's comments into consideration. The biggest change is os.fspath() now returns whatever path.__fspath__() returns instead of restricting it to only str. Minor changes: - Renamed the C function to PyOS_FSPath() - Added an Implementation section with a TODO list - Bunch of things added to the Rejected Ideas section ---------- PEP: NNN Title: Adding a file system path protocol Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-May-2016 Post-History: 11-May-2016, 12-May-2016 Abstract ======== This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only ``str`` and/or ``bytes`` file system paths are accepted. The goal is to facilitate the migration of users towards rich path objects while providing an easy way to work with code expecting ``str`` or ``bytes``. Rationale ========= Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path. This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not. To help elevate the representation of file system paths from their representation as strings and bytes to a richer object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone. One issue in converting path objects to strings comes from the fact that the only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file. Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library has become an issue. A solution that allows any path-representing object to declare that it is a path and a way to extract a low-level representation that all path objects could support is desired. This PEP then proposes to introduce a new protocol to be followed by objects which represent file system paths. Providing a protocol allows for explicit signaling of what objects represent file system paths as well as a way to extract a lower-level representation that can be used with older APIs which only support strings or bytes. Discussions regarding path objects that led to this PEP can be found in multiple threads on the python-ideas mailing list archive [#python-ideas-archive]_ for the months of March and April 2016 and on the python-dev mailing list archives [#python-dev-archive]_ during April 2016. Proposal ======== This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation. The other part deals with changes to Python's standard library to support the new protocol. These changes will also lead to the pathlib module dropping its provisional status. Protocol -------- The following abstract base class defines the protocol for an object to be considered a path object:: import abc import typing as t class PathLike(abc.ABC): """Abstract base class for implementing the file system path protocol.""" @abc.abstractmethod def __fspath__(self) -> t.Union[str, bytes]: """Return the file system path representation of the object.""" raise NotImplementedError Objects representing file system paths will implement the ``__fspath__()`` method which will return the ``str`` or ``bytes`` representation of the path. The ``str`` representation is the preferred low-level path representation as it is human-readable and what people historically represent paths as. Standard library changes ------------------------ It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary). The modules mentioned below, though, deserve specific details as they have either fundamental changes that empower the ability to use path objects, or entail additions/removal of APIs. builtins '''''''' ``open()`` [#builtins-open]_ will be updated to accept path objects as well as continue to accept ``str`` and ``bytes``. os ''' The ``fspath()`` function will be added with the following semantics:: import typing as t def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]: """Return the string representation of the path. If str or bytes is passed in, it is returned unchanged. """ if isinstance(path, (str, bytes)): return path try: return path.__fspath__() except AttributeError: if hasattr(path, '__fspath__'): raise raise TypeError("expected str, bytes or path object, not " + type(path).__name__) The ``os.fsencode()`` [#os-fsencode]_ and ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept path objects. As both functions coerce their arguments to ``bytes`` and ``str``, respectively, they will be updated to call ``__fspath__()`` if present to convert the path object to a ``str`` or ``bytes`` representation, and then perform their appropriate coercion operations as if the return value from ``__fspath__()`` had been the original argument to the coercion function in question. The addition of ``os.fspath()``, the updates to ``os.fsencode()``/``os.fsdecode()``, and the current semantics of ``pathlib.PurePath`` provide the semantics necessary to get the path representation one prefers. For a path object, ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or ``bytes`` representation without any coersion, then ``os.fspath()`` can be used. If a ``str`` is desired and the encoding of ``bytes`` should be assumed to be the default file system encoding, then ``os.fsdecode()`` should be used. If a ``bytes`` representation is desired and any strings should be encoded using the default file system encoding, then ``os.fsencode()`` is used. This PEP recommends using path objects when possible and falling back to string paths as necessary and using ``bytes`` as a last resort. Another way to view this is as a hierarchy of file system path representations (highest- to lowest-level): path ? str ? bytes. The functions and classes under discussion can all accept objects on the same level of the hierarchy, but they vary in whether they promote or demote objects to another level. The ``pathlib.PurePath`` class can promote a ``str`` to a path object. The ``os.fspath()`` function can demote a path object to a ``str`` or ``bytes`` instance, depending on what ``__fspath__()`` returns. The ``os.fsdecode()`` function will demote a path object to a string or promote a ``bytes`` object to a ``str``. The ``os.fsencode()`` function will demote a path or string object to ``bytes``. There is no function that provides a way to demote a path object directly to ``bytes`` while bypassing string demotion. The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` method. It will return the same value as currently found on the ``path`` attribute of ``DirEntry`` instances. The Protocol_ ABC will be added to the ``os`` module under the name ``os.PathLike``. os.path ''''''' The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use ``os.fspath()``. During the discussions leading up to this PEP it was suggested that ``os.path`` not be updated using an "explicit is better than implicit" argument. The thinking was that since ``__fspath__()`` is polymorphic itself it may be better to have code working with ``os.path`` extract the path representation from path objects explicitly. There is also the consideration that adding support this deep into the low-level OS APIs will lead to code magically supporting path objects without requiring any documentation updated, leading to potential complaints when it doesn't work, unbeknownst to the project author. But it is the view of this PEP that "practicality beats purity" in this instance. To help facilitate the transition to supporting path objects, it is better to make the transition as easy as possible than to worry about unexpected/undocumented duck typing support for path objects by projects. There has also been the suggestion that ``os.path`` functions could be used in a tight loop and the overhead of checking or calling ``__fspath__()`` would be too costly. In this scenario only path-consuming APIs would be directly updated and path-manipulating APIs like the ones in ``os.path`` would go unmodified. This would require library authors to update their code to support path objects if they performed any path manipulations, but if the library code passed the path straight through then the library wouldn't need to be updated. It is the view of this PEP and Guido, though, that this is an unnecessary worry and that performance will still be acceptable. pathlib ''''''' The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path`` will continue to not accept ``bytes`` path representations, and so if ``__fspath__()`` returns ``bytes`` it will raise an exception. The ``path`` attribute will be removed as this PEP makes it redundant (it has not been included in any released version of Python and so is not a backwards-compatibility concern). C API ''''' The C API will gain an equivalent function to ``os.fspath()``:: /* Return the file system path of the object. If the object is str or bytes, then allow it to pass through with an incremented refcount. If the object defines __fspath__(), then return the result of that method. All other types raise a TypeError. */ PyObject * PyOS_FSPath(PyObject *path) { if (PyUnicode_Check(path) || PyBytes_Check(path)) { Py_INCREF(path); return path; } if (PyObject_HasAttrString(path, "__fspath__")) { return PyObject_CallMethodObjArgs(path, "__fspath__", NULL); } return PyErr_Format(PyExc_TypeError, "expected a str, bytes, or path object, not %S", path->ob_type); } Backwards compatibility ======================= There are no explicit backwards-compatibility concerns. Unless an object incidentally already defines a ``__fspath__()`` method there is no reason to expect the pre-existing code to break or expect to have its semantics implicitly changed. Libraries wishing to support path objects and a version of Python prior to Python 3.6 and the existence of ``os.fspath()`` can use the idiom of ``path.__fspath__() if hasattr(path, "__fspath__") else path``. Implementation ============== This is the task list for what this PEP proposes: #. Remove the ``path`` attribute from pathlib #. Remove the provisional status of pathlib #. Add ``os.PathLike`` #. Add ``os.fspath()`` #. Add ``PyOS_FSPath()`` #. Update ``os.fsencode()`` #. Update ``os.fsdecode()`` #. Update ``pathlib.PurePath`` and ``pathlib.Path`` #. Update ``builtins.open()`` #. Update ``os.DirEntry`` #. Update ``os.path`` Rejected Ideas ============== Other names for the protocol's method ------------------------------------- Various names were proposed during discussions leading to this PEP, including ``__path__``, ``__pathname__``, and ``__fspathname__``. In the end people seemed to gravitate towards ``__fspath__`` for being unambiguous without being unnecessarily long. Separate str/bytes methods -------------------------- At one point it was suggested that ``__fspath__()`` only return strings and another method named ``__fspathb__()`` be introduced to return bytes. The thinking is that by making ``__fspath__()`` not be polymorphic it could make dealing with the potential string or bytes representations easier. But the general consensus was that returning bytes will more than likely be rare and that the various functions in the os module are the better abstraction to promote over direct calls to ``__fspath__()``. Providing a ``path`` attribute ------------------------------ To help deal with the issue of ``pathlib.PurePath`` not inheriting from ``str``, originally it was proposed to introduce a ``path`` attribute to mirror what ``os.DirEntry`` provides. In the end, though, it was determined that a protocol would provide the same result while not directly exposing an API that most people will never need to interact with directly. Have ``__fspath__()`` only return strings ------------------------------------------ Much of the discussion that led to this PEP revolved around whether ``__fspath__()`` should be polymorphic and return ``bytes`` as well as ``str`` or only return ``str``. The general sentiment for this view was that ``bytes`` are difficult to work with due to their inherent lack of information about their encoding and PEP 383 makes it possible to represent all file system paths using ``str`` with the ``surrogateescape`` handler. Thus, it would be better to forcibly promote the use of ``str`` as the low-level path representation for high-level path objects. In the end, it was decided that using ``bytes`` to represent paths is simply not going to go away and thus they should be supported to some degree. The hope is that people will gravitate towards path objects like pathlib and that will move people away from operating directly with ``bytes``. A generic string encoding mechanism ----------------------------------- At one point there was a discussion of developing a generic mechanism to extract a string representation of an object that had semantic meaning (``__str__()`` does not necessarily return anything of semantic significance beyond what may be helpful for debugging). In the end, it was deemed to lack a motivating need beyond the one this PEP is trying to solve in a specific fashion. Have __fspath__ be an attribute ------------------------------- It was briefly considered to have ``__fspath__`` be an attribute instead of a method. This was rejected for two reasons. One, historically protocols have been implemented as "magic methods" and not "magic methods and attributes". Two, there is no guarantee that the lower-level representation of a path object will be pre-computed, potentially misleading users that there was no expensive computation behind the scenes in case the attribute was implemented as a property. This also indirectly ties into the idea of introducing a ``path`` attribute to accomplish the same thing. This idea has an added issue, though, of accidentally having any object with a ``path`` attribute meet the protocol's duck typing. Introducing a new magic method for the protocol helpfully avoids any accidental opting into the protocol. Provide specific type hinting support ------------------------------------- There was some consideration to provdinga generic ``typing.PathLike`` class which would allow for e.g. ``typing.PathLike[str]`` to specify a type hint for a path object which returned a string representation. While potentially beneficial, the usefulness was deemed too small to bother adding the type hint class. This also removed any desire to have a class in the ``typing`` module which represented the union of all acceptable path-representing types as that can be represented with ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope is users will slowly gravitate to path objects only. Acknowledgements ================ Thanks to everyone who participated in the various discussions related to this PEP that spanned both python-ideas and python-dev. Special thanks to Stephen Turnbull for direct feedback on early drafts of this PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not only feedback on early drafts of this PEP but also helping to drive the overall discussion on this topic across the two mailing lists. References ========== .. [#python-ideas-archive] The python-ideas mailing list archive (https://mail.python.org/pipermail/python-ideas/) .. [#python-dev-archive] The python-dev mailing list archive (https://mail.python.org/pipermail/python-dev/) .. [#libc-open] ``open()`` documention for the C standard library ( http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html ) .. [#pathlib] The ``pathlib`` module (https://docs.python.org/3/library/pathlib.html#module-pathlib) .. [#builtins-open] The ``builtins.open()`` function (https://docs.python.org/3/library/functions.html#open) .. [#os-fsencode] The ``os.fsencode()`` function (https://docs.python.org/3/library/os.html#os.fsencode) .. [#os-fsdecode] The ``os.fsdecode()`` function (https://docs.python.org/3/library/os.html#os.fsdecode) .. [#os-direntry] The ``os.DirEntry`` class (https://docs.python.org/3/library/os.html#os.DirEntry) .. [#os-path] The ``os.path`` module (https://docs.python.org/3/library/os.path.html#module-os.path) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From sjoerdjob at sjoerdjob.com Thu May 12 16:59:29 2016 From: sjoerdjob at sjoerdjob.com (Sjoerd Job Postmus) Date: Thu, 12 May 2016 22:59:29 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5734D9E5.6060701@stoneleaf.us> References: <20160512172225.GA16846@sjoerdjob.com> <5734D9E5.6060701@stoneleaf.us> Message-ID: > On 12 May 2016, at 21:30, Ethan Furman wrote: > > If you need bytes support for your paths, there's at least one [1] that has that support. So if I would need bytes support, I should submit a pull request to which replaces usage of the stdlib pathlib with another variant, upon which they will decline the pull request because it introduces another "useless" dependency. Good to know. From brett at python.org Thu May 12 17:34:04 2016 From: brett at python.org (Brett Cannon) Date: Thu, 12 May 2016 21:34:04 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <20160512172225.GA16846@sjoerdjob.com> <5734D9E5.6060701@stoneleaf.us> Message-ID: On Thu, 12 May 2016 at 14:04 Sjoerd Job Postmus wrote: > > > > On 12 May 2016, at 21:30, Ethan Furman wrote: > > > > If you need bytes support for your paths, there's at least one [1] that > has that support. > > So if I would need bytes support, I should submit a pull request to > which replaces usage of the stdlib pathlib > with another variant, upon which they will decline the pull request because > it introduces another "useless" dependency. > No, what you should do is ask them to create the pathlib instance lazily and only when duck typing shows they weren't given a compatible object. Then you could pass in some bytes-based solution like Ethan's and not worry about pathlib's refusal to work with bytes. Or you simply ask them to work with os.path after calling os.fspath() and be very careful to not use any strings with the functions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Thu May 12 19:58:53 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 12 May 2016 16:58:53 -0700 Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3) In-Reply-To: References: Message-ID: Is anyone going to mark the PEP as accepted? On Thu, May 12, 2016 at 8:11 AM, Guido van Rossum wrote: > OK, then PEP 515 is now officially accepted! Congratulations. Start the > implementation work! > > --Guido (mobile) > On May 11, 2016 10:33 PM, "Georg Brandl" wrote: > > I'm happy with the latest version. > > Georg > > On 05/11/2016 06:46 PM, Guido van Rossum wrote: > > If the authors are happy I'll accept it right away. > > > > (I vaguely recall there's another PEP that's ready for pronouncement -- > but > > which one?) > > > > On Wed, May 11, 2016 at 9:34 AM, Brett Cannon > > wrote: > > > > Is there anything holding up PEP 515 at this point in terms of > acceptance or > > implementation? > > > > On Sat, 19 Mar 2016 at 11:56 Guido van Rossum > > wrote: > > > > All that sounds fine! > > > > On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah < > stefan at bytereef.org > > > wrote: > > > Guido van Rossum python.org > > writes: > > >> So should the preprocessing step just be s.replace('_', ''), > or should > > >> it reject underscores that don't follow the rules from the PEP > > >> (perhaps augmented so they follow the spirit of the PEP and > the letter > > >> of the IBM spec)? > > >> > > >> Honestly I think it's also fine if specifying this exactly is > left out > > >> of the PEP, and handled by whoever adds this to Decimal. > Having a PEP > > >> to work from for the language spec and core builtins (int(), > float() > > >> complex()) is more important. > > > > > > I'd keep it simple for Decimal: Remove left and right > whitespace (we're > > > already doing this), then remove underscores from the > remaining string > > > (which must not contain any further whitespace), then use the > IBM grammar. > > > > > > > > > We could add a clause to the PEP that only those strings that > follow > > > the spirit of the PEP are guaranteed to be accepted in the > future. > > > > > > > > > One reason for keeping it simple is that I would not like to > slow down > > > string conversion, but thinking about two grammars is also a > problem -- > > > part of the string conversion in libmpdec is modeled in ACL2, > which > > > would be invalidated or at least complicated with two grammars. > > > > > > > > > > > > Stefan Krah > > > > > > _______________________________________________ > > > Python-Dev mailing list > > > Python-Dev at python.org > > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > > -- > > --Guido van Rossum (python.org/~guido >) > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > > https://mail.python.org/mailman/options/python-dev/brett%40python.org > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido ) > > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu May 12 20:25:51 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 00:25:51 +0000 Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3) In-Reply-To: References: Message-ID: Marked as accepted in https://hg.python.org/peps/rev/a52066565cc2 On Thu, 12 May 2016 at 16:59 Guido van Rossum wrote: > Is anyone going to mark the PEP as accepted? > > On Thu, May 12, 2016 at 8:11 AM, Guido van Rossum > wrote: > >> OK, then PEP 515 is now officially accepted! Congratulations. Start the >> implementation work! >> >> --Guido (mobile) >> On May 11, 2016 10:33 PM, "Georg Brandl" wrote: >> >> I'm happy with the latest version. >> >> Georg >> >> On 05/11/2016 06:46 PM, Guido van Rossum wrote: >> > If the authors are happy I'll accept it right away. >> > >> > (I vaguely recall there's another PEP that's ready for pronouncement -- >> but >> > which one?) >> > >> > On Wed, May 11, 2016 at 9:34 AM, Brett Cannon > > > wrote: >> > >> > Is there anything holding up PEP 515 at this point in terms of >> acceptance or >> > implementation? >> > >> > On Sat, 19 Mar 2016 at 11:56 Guido van Rossum > > > wrote: >> > >> > All that sounds fine! >> > >> > On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah < >> stefan at bytereef.org >> > > wrote: >> > > Guido van Rossum python.org > >> writes: >> > >> So should the preprocessing step just be s.replace('_', ''), >> or should >> > >> it reject underscores that don't follow the rules from the >> PEP >> > >> (perhaps augmented so they follow the spirit of the PEP and >> the letter >> > >> of the IBM spec)? >> > >> >> > >> Honestly I think it's also fine if specifying this exactly >> is left out >> > >> of the PEP, and handled by whoever adds this to Decimal. >> Having a PEP >> > >> to work from for the language spec and core builtins (int(), >> float() >> > >> complex()) is more important. >> > > >> > > I'd keep it simple for Decimal: Remove left and right >> whitespace (we're >> > > already doing this), then remove underscores from the >> remaining string >> > > (which must not contain any further whitespace), then use the >> IBM grammar. >> > > >> > > >> > > We could add a clause to the PEP that only those strings that >> follow >> > > the spirit of the PEP are guaranteed to be accepted in the >> future. >> > > >> > > >> > > One reason for keeping it simple is that I would not like to >> slow down >> > > string conversion, but thinking about two grammars is also a >> problem -- >> > > part of the string conversion in libmpdec is modeled in ACL2, >> which >> > > would be invalidated or at least complicated with two >> grammars. >> > > >> > > >> > > >> > > Stefan Krah >> > > >> > > _______________________________________________ >> > > Python-Dev mailing list >> > > Python-Dev at python.org >> > > https://mail.python.org/mailman/listinfo/python-dev >> > > Unsubscribe: >> > >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > >> > >> > >> > -- >> > --Guido van Rossum (python.org/~guido > >) >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > >> > >> > >> > >> > -- >> > --Guido van Rossum (python.org/~guido ) >> > >> > >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri May 13 00:04:24 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 May 2016 14:04:24 +1000 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: References: Message-ID: On 13 May 2016 at 06:53, Brett Cannon wrote: > Second draft that takes Guido's comments into consideration. The biggest > change is os.fspath() now returns whatever path.__fspath__() returns instead > of restricting it to only str. > > Minor changes: > - Renamed the C function to PyOS_FSPath() > - Added an Implementation section with a TODO list > - Bunch of things added to the Rejected Ideas section +1 for this version from me, as it means we have: - os.fsencode(obj) as the coerce-to-bytes API - os.fspath(obj) as the str/bytes hybrid API - os.fsdecode(obj) as the coerce-to-str API - os.fspath(pathlib.PurePath(obj)) as the error-on-bytes API That more strongly nudges people towards "use pathlib if you want to ensure cross-platform friendly path handling", which is an outcome I'm fine with. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Fri May 13 02:07:10 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 May 2016 23:07:10 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <20160512172225.GA16846@sjoerdjob.com> <5734D9E5.6060701@stoneleaf.us> Message-ID: <57356F0E.2030407@stoneleaf.us> On 05/12/2016 01:59 PM, Sjoerd Job Postmus wrote: >> On 12 May 2016, at 21:30, Ethan Furman wrote: >> >> If you need bytes support for your paths, there's at least one [1] that has that support. > > So if I would need bytes support, I should submit a pull request to library> which replaces usage of the stdlib pathlib with another variant, upon which they > will decline the pull request because it introduces another "useless" dependency. My apologies, I thought you were talking about your own code. As far as -- it wouldn't be awesome to me if I couldn't pass in the data types I need. -- ~Ethan~ From srkunze at mail.de Fri May 13 03:53:29 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 13 May 2016 09:53:29 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5734BCF9.6060305@stoneleaf.us> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <5734AEA6.5010308@mail.de> <5734B5C7.4060206@stoneleaf.us> <5734BBA9.2050807@mail.de> <5734BCF9.6060305@stoneleaf.us> Message-ID: <573587F9.7040201@mail.de> On 12.05.2016 19:27, Ethan Furman wrote: > Maybe, but a bad idea for two reasons: > > 1) Reducing a str to the exact same str is silly; and, more importantly Finding something silly is no technical argument. Best, Sven From srkunze at mail.de Fri May 13 03:58:13 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 13 May 2016 09:58:13 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: <57358915.7040606@mail.de> On 12.05.2016 18:24, Guido van Rossum wrote: > def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: > if isinstance(p, (str, bytes)): > return p > try: > return p.__fspath__ > except AttributeError: > raise TypeError(...) @Brett Would you think it makes sense to swap the str/bytes check and the __fspath__ check? I just thought of a class subclassing str/bytes and defines __fspath__. Its __fspath__ method would be ignored currently. Best, Sven From k7hoven at gmail.com Fri May 13 04:36:04 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 13 May 2016 11:36:04 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <57358915.7040606@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> Message-ID: This has just been discussed very recently in this thread (and earlier too). It may make sense, but it's not among our current worries. Besides, we already added the new fspath semantics to the PEP. While I hope Brett is asleep in his time zone, I'm guessing he will agree (just saying this because you write "@Brett"). -- Koos On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze wrote: > On 12.05.2016 18:24, Guido van Rossum wrote: >> >> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: >> if isinstance(p, (str, bytes)): >> return p >> try: >> return p.__fspath__ >> except AttributeError: >> raise TypeError(...) > > > @Brett > Would you think it makes sense to swap the str/bytes check and the > __fspath__ check? > > > I just thought of a class subclassing str/bytes and defines __fspath__. Its > __fspath__ method would be ignored currently. > > > Best, > Sven > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com From srkunze at mail.de Fri May 13 05:24:57 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 13 May 2016 11:24:57 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> Message-ID: <57359D69.6040108@mail.de> On 13.05.2016 10:36, Koos Zevenhoven wrote: > This has just been discussed very recently in this thread (and earlier > too). Could you point me to that? It seems I missed that part. I only found posts related to performance degradation. However, the proposed semantics will change if the checks are swapped. So, my actual question is: Is that an intended API inconsistency or a known bug supposed to be resolved later? > It may make sense, but it's not among our current worries. It might not be yours but mine. ;) That's why I was asking. > Besides, we already added the new fspath semantics to the PEP. > > While I hope Brett is asleep in his time zone, I'm guessing he will > agree (just saying this because you write "@Brett"). > > -- Koos > > > On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze wrote: >> On 12.05.2016 18:24, Guido van Rossum wrote: >>> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: >>> if isinstance(p, (str, bytes)): >>> return p >>> try: >>> return p.__fspath__ >>> except AttributeError: >>> raise TypeError(...) >> >> @Brett >> Would you think it makes sense to swap the str/bytes check and the >> __fspath__ check? >> >> >> I just thought of a class subclassing str/bytes and defines __fspath__. Its >> __fspath__ method would be ignored currently. >> >> >> Best, >> Sven >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com From larry at hastings.org Fri May 13 05:46:12 2016 From: larry at hastings.org (Larry Hastings) Date: Fri, 13 May 2016 11:46:12 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5734A46C.40306@stoneleaf.us> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> Message-ID: <5735A264.2020103@hastings.org> On 05/12/2016 05:42 PM, Ethan Furman wrote: > And even given all that, for smoother interoperability with the rest > of the stdlib, or at least the os.* portion, those functions would > still need to be upgraded to check for .path on the incoming arguments > -- at which point we may as well make a protocol to properly support > file system paths instead of relying on the rather generic attribute > name of 'path'. FWIW, this shouldn't be very difficult. Paths are already complicated enough that the parsing has been factored out into a PyArg_Parse "converter" called path_converter. path_converter isn't used everywhere in posixmodule yet, but it would probably make sense to convert everything to use it anyway. And then we would only need to implement Path object support once. It's remotely possible that converting functions to use path_converter will create slight incompatibilities (though I don't know how, this is just FUD really) so 3.6 would be a good opportunity to make this change. Optimistically, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri May 13 05:48:33 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 13 May 2016 12:48:33 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <57359D69.6040108@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> Message-ID: On Fri, May 13, 2016 at 12:24 PM, Sven R. Kunze wrote: > On 13.05.2016 10:36, Koos Zevenhoven wrote: >> >> This has just been discussed very recently in this thread (and earlier >> too). > > > Could you point me to that? It seems I missed that part. I only found posts > related to performance degradation. > This issue is coupled with the future optimization questions. > However, the proposed semantics will change if the checks are swapped. So, > my actual question is: > > Is that an intended API inconsistency or a known bug supposed to be resolved > later? > Taking into account the description (and the drafted type hint), which the documentation will probably reflect, the semantic effects of that are very minor or nonexistent. I do think the documentation of the protocol should say that str or bytes subclasses should not implement __fspath__. So no API inconsistency there. -- Koos From steve at pearwood.info Fri May 13 06:19:40 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 13 May 2016 20:19:40 +1000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <20160512172225.GA16846@sjoerdjob.com> References: <20160512172225.GA16846@sjoerdjob.com> Message-ID: <20160513101939.GG12028@ando.pearwood.info> On Thu, May 12, 2016 at 07:22:25PM +0200, Sjoerd Job Postmus wrote: > The whole point of adding `os.fspath` is to make it easier to use Path > objects. This is in an effort to gain greater adoption of pathlib in > libraries. Now, this is an excellent idea. > > However, if it were to reject bytes, that would mean that when libraries > start to use pathlib, it would suddenly become harder for people that > actually need bytes-support to use pathlib. I'm not sure how it would be *harder*. Currently, if you need bytes paths, you can't use pathlib. That won't change, so it will be exactly as hard as it is now. > Now, the claim 'if you need bytes, you should not be using pathlib` is a > reasonable one. But what if I need bytes *and* a specific library (say, > image handling, or a web framework, or ...). It's not up to me if that > library uses pathlib or plain old os.path.join. I feel this is Not Our Problem. Surely the stdlib cannot be held responsible for all the poor decisions made by third-party libraries? If a library hard-codes "\\" as their directory separator, because everyone uses Windows, you'd simply say "this library is not very good for my use-case". Likewise if it rejects bytes paths, then: - perhaps that library is not fit for your purpose; - perhaps you can use it, but with the limitation that you can no longer support all paths, but only the ones which can be accessed via a string path; - perhaps you can build a compatibility layer that lets you use both bytes and the library. I don't doubt that all three options are unpleasant, but I don't see how they are the fault of this PEP. This PEP supports Path-Like objects that return bytes paths. Its up to library authors to support bytes paths, or not, as they see fit. -- Steve From rosuav at gmail.com Fri May 13 06:54:05 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 13 May 2016 20:54:05 +1000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <20160513101939.GG12028@ando.pearwood.info> References: <20160512172225.GA16846@sjoerdjob.com> <20160513101939.GG12028@ando.pearwood.info> Message-ID: On Fri, May 13, 2016 at 8:19 PM, Steven D'Aprano wrote: > I feel this is Not Our Problem. Surely the stdlib cannot be held > responsible for all the poor decisions made by third-party libraries? If > a library hard-codes "\\" as their directory separator, because everyone > uses Windows, you'd simply say "this library is not very good for my > use-case". Likewise if it rejects bytes paths, then: > > - perhaps that library is not fit for your purpose; > > - perhaps you can use it, but with the limitation that you can no > longer support all paths, but only the ones which can be > accessed via a string path; > > - perhaps you can build a compatibility layer that lets you use > both bytes and the library. AFAICT, the compatibility layer would simply decode the bytes using surrogateescape handling, which should round-trip anything. Or am I wrong here somewhere? ChrisA From steve at pearwood.info Fri May 13 07:00:14 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 13 May 2016 21:00:14 +1000 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: References: Message-ID: <20160513110012.GH12028@ando.pearwood.info> On Thu, May 12, 2016 at 08:53:12PM +0000, Brett Cannon wrote: > Second draft that takes Guido's comments into consideration. The biggest > change is os.fspath() now returns whatever path.__fspath__() returns > instead of restricting it to only str. Counter suggestion: - __fspath__() method may return either bytes or str (no change from the PEP as it stands now); - but os.fspath() will only return str; - and os.fspathb() will only return bytes; - there is no os function that returns "str or bytes, I don't care which". (If you really need that, call __fspath__ directly.) Note that this differs from the already rejected suggestion that there should be two dunder methods, __fspath__() and __fspathb__(). Why? (1) Normally, the caller knows whether they want str or bytes. (That's been my experience, you may disagree.) If so, and they call os.fspath() expecting a str, they won't be surprised by it returning bytes. And visa versa for when you expect a bytes path. (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}. Cons: (3) Polymorphic code that truly doesn't care whether it gets bytes or str will have a slightly less convenient way of getting it, namely by calling __fspath__() itself, instead of os.fspath(). A few other comments below: > builtins > '''''''' > > ``open()`` [#builtins-open]_ will be updated to accept path objects as > well as continue to accept ``str`` and ``bytes``. I think it is a bit confusing to refer to "path objects", as that seems like you are referring only to pathlib.Path objects. It took me far too long to realise that here you mean generic path-like objects that obey the __fspath__ protocol rather than a specific concrete class. Since the ABC is called "PathLike", I suggest we refer to "path-like objects" rather than "path objects", both in the PEP and in the Python docs for this protocol. > def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]: > """Return the string representation of the path. > > If str or bytes is passed in, it is returned unchanged. > """ I've already suggested a change to this, above, but independent of that, a minor technical query: > try: > return path.__fspath__() Would I be right in saying that in practice this will actually end up being type(path).__fspath__() to match the behaviour of all(?) other dunder methods? -- Steve From rosuav at gmail.com Fri May 13 07:28:46 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 13 May 2016 21:28:46 +1000 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <20160513110012.GH12028@ando.pearwood.info> References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: On Fri, May 13, 2016 at 9:00 PM, Steven D'Aprano wrote: > Cons: > (3) Polymorphic code that truly doesn't care whether it gets bytes or > str will have a slightly less convenient way of getting it, namely by > calling __fspath__() itself, instead of os.fspath(). I don't like this; it goes against the general principle that dunders are for defining, not calling. Generally, a given dunder method has approximately one call site, eg __reduce__ in pickle.py, and everyone else defines it. (You might call super's dunder in the definition of your own, but that's still defining it, not calling it.) Having an official statement that it's appropriate to call a dunder confuses this. So this isn't a "slightly less convenient way", it's a bad way (IMO), and this is a very strong con. ChrisA From ncoghlan at gmail.com Fri May 13 07:56:34 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 May 2016 21:56:34 +1000 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <20160513110012.GH12028@ando.pearwood.info> References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: On 13 May 2016 at 21:00, Steven D'Aprano wrote: > On Thu, May 12, 2016 at 08:53:12PM +0000, Brett Cannon wrote: > >> Second draft that takes Guido's comments into consideration. The biggest >> change is os.fspath() now returns whatever path.__fspath__() returns >> instead of restricting it to only str. > > Counter suggestion: > > - __fspath__() method may return either bytes or str (no change > from the PEP as it stands now); > > - but os.fspath() will only return str; > > - and os.fspathb() will only return bytes; We don't have any use cases for a bytes-only API - only str-or-bytes (in the polymorphic low level functions) and str-only (in newer high level APIs like pathlib). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From k7hoven at gmail.com Fri May 13 09:51:44 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 13 May 2016 16:51:44 +0300 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: References: Message-ID: Thanks Brett! Now one thing is that, despite your suggestion, I had not added myself as an author in my big pull request. Originally, it was because I simply forgot to copy and paste it when I split my edits into separate commits ;-). Sorry about that (not sure if you care though, and I've been defending the PEP regardless). Anyway, especially now that my main worry regarding the open questions has been resolved, I would be more than happy to have my name on it. So Brett, could you add me as author? (Koos Zevenhoven and k7hoven at gmail.com will be fine) It looks like this is finally happening :) -- Koos On Thu, May 12, 2016 at 11:53 PM, Brett Cannon wrote: > Second draft that takes Guido's comments into consideration. The biggest > change is os.fspath() now returns whatever path.__fspath__() returns instead > of restricting it to only str. > > Minor changes: > - Renamed the C function to PyOS_FSPath() > - Added an Implementation section with a TODO list > - Bunch of things added to the Rejected Ideas section > From contrebasse at gmail.com Fri May 13 10:23:09 2016 From: contrebasse at gmail.com (Joseph Martinot-Lagarde) Date: Fri, 13 May 2016 14:23:09 +0000 (UTC) Subject: [Python-Dev] File system path PEP, part 2 References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: > - there is no os function that returns "str or bytes, I don't > care which". (If you really need that, call __fspath__ directly.) os.fspath() in the PEP works when given str or bytes directly, but those don't have a __fspath__ method, so directly calling the dunder method is not equivalent to calling os.fspath(). The whole point of having os.fspath() is to pass str or paths (or bytes) and get an appropriate object for lew-level functions. From sjoerdjob at sjoerdjob.com Fri May 13 02:22:12 2016 From: sjoerdjob at sjoerdjob.com (Sjoerd Job Postmus) Date: Fri, 13 May 2016 08:22:12 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <57356F0E.2030407@stoneleaf.us> References: <20160512172225.GA16846@sjoerdjob.com> <5734D9E5.6060701@stoneleaf.us> <57356F0E.2030407@stoneleaf.us> Message-ID: <6484E0A4-BF91-4643-8224-4E69D8AE354B@sjoerdjob.com> > On 13 May 2016, at 08:07, Ethan Furman wrote: > > On 05/12/2016 01:59 PM, Sjoerd Job Postmus wrote: >>> On 12 May 2016, at 21:30, Ethan Furman wrote: >>> >>> If you need bytes support for your paths, there's at least one [1] that has that support. >> >> So if I would need bytes support, I should submit a pull request to > library> which replaces usage of the stdlib pathlib with another variant, upon which they > > will decline the pull request because it introduces another "useless" dependency. > > My apologies, I thought you were talking about your own code. > > As far as -- it wouldn't be awesome to me if I couldn't pass in the data types I need. > > -- > ~Ethan~ I just hope you are right. Thank you (and Brett) for clearing things up for me. My apologies if I came off as (a bit) harsh. From random832 at fastmail.com Fri May 13 11:25:26 2016 From: random832 at fastmail.com (Random832) Date: Fri, 13 May 2016 11:25:26 -0400 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <20160513110012.GH12028@ando.pearwood.info> References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: <1463153126.2155923.607016537.1E02B685@webmail.messagingengine.com> On Fri, May 13, 2016, at 07:00, Steven D'Aprano wrote: > - but os.fspath() will only return str; > > - and os.fspathb() will only return bytes; And raise an exception if __fspath__ returns the other, I suppose. What's the use case for these functions? When would I call them rather than fsdecode and fsencode? (assuming the latter will support the path protocol - I've got more objections if they're not going to) Also, what happens if you pass a string to os.fspath? Statements like "str will not implement __fspath__" have thus far been light on details of what functions will accept "str or path-like-object-whose-__fspath__returns-str" and which ones will only accept the latter (and what, if any, will continue to only accept the former). If no-one's supposed to directly call dunder methods, and os.fspath accepts str, then what difference does it make whether this is implemented by having a special case within os.fspath or by calling str.__fspath__? From brett at python.org Fri May 13 11:29:29 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 15:29:29 +0000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <57359D69.6040108@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> Message-ID: On Fri, 13 May 2016 at 02:25 Sven R. Kunze wrote: > On 13.05.2016 10:36, Koos Zevenhoven wrote: > > This has just been discussed very recently in this thread (and earlier > > too). > > Could you point me to that? It seems I missed that part. I only found > posts related to performance degradation. > > However, the proposed semantics will change if the checks are swapped. > So, my actual question is: > > Is that an intended API inconsistency or a known bug supposed to be > resolved later? > Purposeful change. It was what I had in my head after I posted my "groups" breakdown email and what Guido suggested as well independently. This helps alleviate any perf worries as type checks in C are pointer checks that are cheap to make compared to attribute lookup. And even if we do get the community to move on to path objects the check is still cheap enough to not worry. And finally, this is simply how we have almost always coded this kind of special-casing at the C level throughout Python and the stdlib, so it now is more inline with standard coding practices than the original proposal in the PEP. > > > It may make sense, but it's not among our current worries. > > It might not be yours but mine. ;) That's why I was asking. > Yes, I was very much asleep. :) -Brett > > > Besides, we already added the new fspath semantics to the PEP. > > > > While I hope Brett is asleep in his time zone, I'm guessing he will > > agree (just saying this because you write "@Brett"). > > > > -- Koos > > > > > > On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze wrote: > >> On 12.05.2016 18:24, Guido van Rossum wrote: > >>> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: > >>> if isinstance(p, (str, bytes)): > >>> return p > >>> try: > >>> return p.__fspath__ > >>> except AttributeError: > >>> raise TypeError(...) > >> > >> @Brett > >> Would you think it makes sense to swap the str/bytes check and the > >> __fspath__ check? > >> > >> > >> I just thought of a class subclassing str/bytes and defines __fspath__. > Its > >> __fspath__ method would be ignored currently. > >> > >> > >> Best, > >> Sven > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 13 11:43:29 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 15:43:29 +0000 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <20160513110012.GH12028@ando.pearwood.info> References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: On Fri, 13 May 2016 at 04:00 Steven D'Aprano wrote: > On Thu, May 12, 2016 at 08:53:12PM +0000, Brett Cannon wrote: > > > Second draft that takes Guido's comments into consideration. The biggest > > change is os.fspath() now returns whatever path.__fspath__() returns > > instead of restricting it to only str. > > Counter suggestion: > > - __fspath__() method may return either bytes or str (no change > from the PEP as it stands now); > > - but os.fspath() will only return str; > > - and os.fspathb() will only return bytes; > > - there is no os function that returns "str or bytes, I don't > care which". (If you really need that, call __fspath__ directly.) > > Note that this differs from the already rejected suggestion that there > should be two dunder methods, __fspath__() and __fspathb__(). > > Why? > > (1) Normally, the caller knows whether they want str or bytes. (That's > been my experience, you may disagree.) If so, and they call os.fspath() > expecting a str, they won't be surprised by it returning bytes. And visa > versa for when you expect a bytes path. > > (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}. > > Cons: > > (3) Polymorphic code that truly doesn't care whether it gets bytes or > str will have a slightly less convenient way of getting it, namely by > calling __fspath__() itself, instead of os.fspath(). > I prefer what's in the PEP. I get where you coming from, Steven, but I don't think it will be common enough to worry about. Think of os.fspath() like next() where it truly is a very minor convenience function that happens to special-case str and bytes. > > > > > A few other comments below: > > > > builtins > > '''''''' > > > > ``open()`` [#builtins-open]_ will be updated to accept path objects as > > well as continue to accept ``str`` and ``bytes``. > > I think it is a bit confusing to refer to "path objects", as that seems > like you are referring only to pathlib.Path objects. It took me far too > long to realise that here you mean generic path-like objects that obey > the __fspath__ protocol rather than a specific concrete class. > > Since the ABC is called "PathLike", I suggest we refer to "path-like > objects" rather than "path objects", both in the PEP and in the Python > docs for this protocol. > I went back and forth with this in my head while writing the PEP. The problem with making "path-like" mean "objects implementing the PathLike ABC" becomes how do you refer to an argument of a function that accepts anything os.fspath() does (i.e. PathLike, str, and bytes)? > > > > > > def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, > bytes]: > > """Return the string representation of the path. > > > > If str or bytes is passed in, it is returned unchanged. > > """ > > I've already suggested a change to this, above, but independent of that, > a minor technical query: > > > try: > > return path.__fspath__() > > Would I be right in saying that in practice this will actually end up > being type(path).__fspath__() to match the behaviour of all(?) other > dunder methods? > I wasn't planning on it because for most types the accessing of the method directly off of the type for magic methods is because of some special struct field at the C level that we're pulling from. Since we're not planning to have an equivalent struct field I don't see any need to do the extra work of avoiding the instance participating in method lookup. Obviously if people disagree for some reason then please let me know (maybe for perf by avoiding the overhead of checking for the method on the instance?). -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Fri May 13 12:06:17 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 13 May 2016 18:06:17 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> Message-ID: <5735FB79.1010201@mail.de> On 13.05.2016 11:48, Koos Zevenhoven wrote: > This issue is coupled with the future optimization questions. AFAIC coupling API design to optimization is called premature optimization. >> However, the proposed semantics will change if the checks are swapped. So, >> my actual question is: >> >> Is that an intended API inconsistency or a known bug supposed to be resolved >> later? >> > Taking into account the description (and the drafted type hint), which > the documentation will probably reflect, the semantic effects of that > are very minor or nonexistent. From your perspective. As far as I remember, one goal of this proposal was to avoid wallet gardens. During the lengthy discussion on python-ideas people brought up that some third-party libs indeed subclass from str. They are currently locked out. > I do think the documentation of the protocol should say that str or > bytes subclasses should not implement __fspath__. Indeed. Just one minor note here: str or bytes subclasses *can* implement __fspath__ and currently it will be *ignored*. Maybe that changes in the future. So, that's the reason it should not be implemented. > So no API inconsistency there. API consistency is not defined by docs-matching-implementation but by implementation-matching-expectations. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri May 13 12:08:41 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 13 May 2016 18:08:41 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20160513160841.22A8F56AB1@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-05-06 - 2016-05-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5501 ( -3) closed 33270 (+47) total 38771 (+44) Open issues with patches: 2383 Issues opened (28) ================== #22570: Better stdlib support for Path objects http://bugs.python.org/issue22570 reopened by gvanrossum #26857: gethostbyname_r() is broken on android http://bugs.python.org/issue26857 reopened by xdegaye #26971: python v3.5.1: sys.paths not respecting DESTDIRS and DESTSHARE http://bugs.python.org/issue26971 opened by yaro-yaro #26972: mistakes in docstrings in the import machinery http://bugs.python.org/issue26972 opened by Oren Milman #26974: Crash in Decimal.from_float http://bugs.python.org/issue26974 opened by serhiy.storchaka #26975: Decimal.from_float works incorrectly for non-binary floats http://bugs.python.org/issue26975 opened by serhiy.storchaka #26978: Implement pathlib.Path.link (Using os.link) http://bugs.python.org/issue26978 opened by cool-RR #26979: The danger of PyType_FromSpec() http://bugs.python.org/issue26979 opened by serhiy.storchaka #26980: The path argument of asyncio.BaseEventLoop.create_unix_connect http://bugs.python.org/issue26980 opened by texttheater #26981: add compatibility shim for enum34 backport http://bugs.python.org/issue26981 opened by ethan.furman #26982: Clarify forward annotations in PEP 484 http://bugs.python.org/issue26982 opened by James.Tatum #26983: float() can return not exact float instance http://bugs.python.org/issue26983 opened by serhiy.storchaka #26984: int() can return not exact int instance http://bugs.python.org/issue26984 opened by serhiy.storchaka #26985: Information about CodeType in inspect documentation is outdate http://bugs.python.org/issue26985 opened by xiang.zhang #26988: Add AutoNumberedEnum to stdlib http://bugs.python.org/issue26988 opened by John Hagen #26990: file.tell affect decoding http://bugs.python.org/issue26990 opened by mfmain #26991: Possible reference leak in MAKE_FUNCTION http://bugs.python.org/issue26991 opened by xiang.zhang #26992: 64-bit Python 2.7.11 hangs in 64-bit Windows 10 - CMD and Git http://bugs.python.org/issue26992 opened by sam.taylor.bs #26993: Copy idlelib *.py files with new names http://bugs.python.org/issue26993 opened by terry.reedy #26995: Add tests for parsing float and object arguments http://bugs.python.org/issue26995 opened by serhiy.storchaka #26997: Docs for pdb should note that __future__ magic doesn't work http://bugs.python.org/issue26997 opened by Martin Jones #27002: Support different modes in posixpath.realpath() http://bugs.python.org/issue27002 opened by serhiy.storchaka #27004: Handle script shbang options http://bugs.python.org/issue27004 opened by opoplawski #27006: C implementation of Decimal.from_float() bypasses __new__ and http://bugs.python.org/issue27006 opened by serhiy.storchaka #27007: Alternate constructors bytes.fromhex() and bytearray.fromhex() http://bugs.python.org/issue27007 opened by serhiy.storchaka #27010: email library could "recover" from bad mime boundary like (som http://bugs.python.org/issue27010 opened by Fedele Mantuano #27012: Rename the posix module to _os http://bugs.python.org/issue27012 opened by brett.cannon #27014: maximum recursion depth when using typing options http://bugs.python.org/issue27014 opened by Roy Shmueli Most recent 15 issues with no replies (15) ========================================== #27014: maximum recursion depth when using typing options http://bugs.python.org/issue27014 #27006: C implementation of Decimal.from_float() bypasses __new__ and http://bugs.python.org/issue27006 #27002: Support different modes in posixpath.realpath() http://bugs.python.org/issue27002 #26997: Docs for pdb should note that __future__ magic doesn't work http://bugs.python.org/issue26997 #26995: Add tests for parsing float and object arguments http://bugs.python.org/issue26995 #26990: file.tell affect decoding http://bugs.python.org/issue26990 #26982: Clarify forward annotations in PEP 484 http://bugs.python.org/issue26982 #26980: The path argument of asyncio.BaseEventLoop.create_unix_connect http://bugs.python.org/issue26980 #26979: The danger of PyType_FromSpec() http://bugs.python.org/issue26979 #26978: Implement pathlib.Path.link (Using os.link) http://bugs.python.org/issue26978 #26964: Incorrect documentation for `-u`-flag http://bugs.python.org/issue26964 #26955: Implement equivalent to `pip.locations.distutils_scheme` in di http://bugs.python.org/issue26955 #26954: Add Guido's rejection notice to the "with" FAQ http://bugs.python.org/issue26954 #26942: android: test_ctypes crashes on armv7 http://bugs.python.org/issue26942 #26937: android: test_tarfile fails http://bugs.python.org/issue26937 Most recent 15 issues waiting for review (15) ============================================= #27007: Alternate constructors bytes.fromhex() and bytearray.fromhex() http://bugs.python.org/issue27007 #27002: Support different modes in posixpath.realpath() http://bugs.python.org/issue27002 #26995: Add tests for parsing float and object arguments http://bugs.python.org/issue26995 #26991: Possible reference leak in MAKE_FUNCTION http://bugs.python.org/issue26991 #26985: Information about CodeType in inspect documentation is outdate http://bugs.python.org/issue26985 #26983: float() can return not exact float instance http://bugs.python.org/issue26983 #26982: Clarify forward annotations in PEP 484 http://bugs.python.org/issue26982 #26974: Crash in Decimal.from_float http://bugs.python.org/issue26974 #26972: mistakes in docstrings in the import machinery http://bugs.python.org/issue26972 #26970: Replace OpenSSL's CPRNG with system entropy source http://bugs.python.org/issue26970 #26964: Incorrect documentation for `-u`-flag http://bugs.python.org/issue26964 #26959: pickle: respect dispatch for functions again http://bugs.python.org/issue26959 #26948: Simplify PyImport_ImportModuleLevelObject: avoid temporary tup http://bugs.python.org/issue26948 #26939: android: test_functools hangs on armv7 http://bugs.python.org/issue26939 #26935: android: test_os fails http://bugs.python.org/issue26935 Top 10 most discussed issues (10) ================================= #26969: ascynio should provide a policy to address pass-loop-everywher http://bugs.python.org/issue26969 36 msgs #26439: ctypes.util.find_library fails when ldconfig/glibc not availab http://bugs.python.org/issue26439 12 msgs #26993: Copy idlelib *.py files with new names http://bugs.python.org/issue26993 9 msgs #26906: format(object.__reduce__) fails intermittently http://bugs.python.org/issue26906 8 msgs #6422: timeit called from within Python should allow autoranging http://bugs.python.org/issue6422 7 msgs #18531: Undocumented different between METH_KEYWORDS and **kws http://bugs.python.org/issue18531 7 msgs #26632: __all__ decorator http://bugs.python.org/issue26632 7 msgs #26983: float() can return not exact float instance http://bugs.python.org/issue26983 7 msgs #26039: More flexibility in zipfile write interface http://bugs.python.org/issue26039 6 msgs #26369: unicode.decode and str.encode are unnecessarily confusing for http://bugs.python.org/issue26369 6 msgs Issues closed (45) ================== #5062: Rlcompleter.Completer does not use __dir__ magic method http://bugs.python.org/issue5062 closed by martin.panter #11643: Use |version| instead of X.Y in the doc http://bugs.python.org/issue11643 closed by serhiy.storchaka #15721: PEP 3121, 384 Refactoring applied to tkinter module http://bugs.python.org/issue15721 closed by serhiy.storchaka #17765: weakref.ref ignores a 'callback' keyword argument http://bugs.python.org/issue17765 closed by serhiy.storchaka #18864: Implementation for PEP 451 (importlib.machinery.ModuleSpec) http://bugs.python.org/issue18864 closed by eric.snow #19444: mmap.mmap() allocates a file descriptor that isn't CLOEXEC http://bugs.python.org/issue19444 closed by martin.panter #21012: Figure out how to best leverage pip in devinabox http://bugs.python.org/issue21012 closed by brett.cannon #22143: rlcompleter.Completer has duplicate matches http://bugs.python.org/issue22143 closed by martin.panter #22274: subprocess.Popen(stderr=STDOUT) fails to redirect subprocess s http://bugs.python.org/issue22274 closed by martin.panter #23213: subprocess communicate() hangs when stderr isn't closed http://bugs.python.org/issue23213 closed by martin.panter #23640: int.from_bytes() is broken for subclasses http://bugs.python.org/issue23640 closed by serhiy.storchaka #23815: Segmentation fault when create _tkinter objects http://bugs.python.org/issue23815 closed by serhiy.storchaka #23921: Standardize documentation whitespace, formatting http://bugs.python.org/issue23921 closed by serhiy.storchaka #24550: maxint on 64 bit platforms breaks os.read http://bugs.python.org/issue24550 closed by serhiy.storchaka #25745: Reference leaks in test_curses http://bugs.python.org/issue25745 closed by serhiy.storchaka #26352: getpass incorrectly displays password prompt on stderr on fall http://bugs.python.org/issue26352 closed by serhiy.storchaka #26512: Vocabulary: Using "integral" in library/stdtypes.html http://bugs.python.org/issue26512 closed by martin.panter #26708: Constify C string pointers in the posix module http://bugs.python.org/issue26708 closed by serhiy.storchaka #26736: Use HTTPS protocol in links http://bugs.python.org/issue26736 closed by serhiy.storchaka #26881: modulefinder should reuse the dis module http://bugs.python.org/issue26881 closed by serhiy.storchaka #26889: Improve Doc/library/xmlrpc.client.rst http://bugs.python.org/issue26889 closed by serhiy.storchaka #26892: debuglevel not honored in urllib http://bugs.python.org/issue26892 closed by orsenthil #26911: lib2to3/tests/pytree_idempotency.py has broken imports http://bugs.python.org/issue26911 closed by terry.reedy #26912: Broken imports in test/test_email/torture_test.py, test_asian_ http://bugs.python.org/issue26912 closed by terry.reedy #26924: android: test_concurrent_futures fails http://bugs.python.org/issue26924 closed by berker.peksag #26925: android: test_multiprocessing_main_handling fails http://bugs.python.org/issue26925 closed by xdegaye #26960: urllib hangs when closing connection http://bugs.python.org/issue26960 closed by python-dev #26973: Dict View binops permit non-set types http://bugs.python.org/issue26973 closed by r.david.murray #26976: pathlib equivalent for os.path.realpath() http://bugs.python.org/issue26976 closed by serhiy.storchaka #26977: Unnecessary line in statistics.pvariance http://bugs.python.org/issue26977 closed by steven.daprano #26986: Enhance PyFunction_New documentation http://bugs.python.org/issue26986 closed by python-dev #26987: Comment/implementation disagreement in Thread._bootstrap_inner http://bugs.python.org/issue26987 closed by python-dev #26989: error in installation of ez_setup.py http://bugs.python.org/issue26989 closed by benjamin.peterson #26994: unexpected behavior for booleans in argparse http://bugs.python.org/issue26994 closed by r.david.murray #26996: Add secrets module as per PEP 506. http://bugs.python.org/issue26996 closed by steven.daprano #26998: Unable to import "site" in an embedded application http://bugs.python.org/issue26998 closed by steve.dower #26999: Add child separator keyword to logging.basicConfig and use it http://bugs.python.org/issue26999 closed by vinay.sajip #27000: improve document of filter http://bugs.python.org/issue27000 closed by r.david.murray #27001: Python Interface X-Plane 10.45 http://bugs.python.org/issue27001 closed by r.david.murray #27003: Python 3.5.1 fails at HTTPSTest with SSL CERT error http://bugs.python.org/issue27003 closed by martin.panter #27005: Faster float.fromhex() http://bugs.python.org/issue27005 closed by serhiy.storchaka #27008: error message of ord is not intuitive http://bugs.python.org/issue27008 closed by r.david.murray #27009: multiprocessing Connection data length limited to max(int32) http://bugs.python.org/issue27009 closed by serhiy.storchaka #27011: Identation error http://bugs.python.org/issue27011 closed by python-dev #27013: IDLE completion for dict keys http://bugs.python.org/issue27013 closed by terry.reedy From guido at python.org Fri May 13 12:21:09 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 13 May 2016 09:21:09 -0700 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <20160513110012.GH12028@ando.pearwood.info> References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: On Fri, May 13, 2016 at 4:00 AM, Steven D'Aprano wrote: > On Thu, May 12, 2016 at 08:53:12PM +0000, Brett Cannon wrote: > > > Second draft that takes Guido's comments into consideration. The biggest > > change is os.fspath() now returns whatever path.__fspath__() returns > > instead of restricting it to only str. > > Counter suggestion: > > - __fspath__() method may return either bytes or str (no change > from the PEP as it stands now); > > - but os.fspath() will only return str; > > - and os.fspathb() will only return bytes; > > - there is no os function that returns "str or bytes, I don't > care which". (If you really need that, call __fspath__ directly.) > > Note that this differs from the already rejected suggestion that there > should be two dunder methods, __fspath__() and __fspathb__(). > > Why? > > (1) Normally, the caller knows whether they want str or bytes. (That's > been my experience, you may disagree.) If so, and they call os.fspath() > expecting a str, they won't be surprised by it returning bytes. And visa > versa for when you expect a bytes path. > > (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}. > > Cons: > > (3) Polymorphic code that truly doesn't care whether it gets bytes or > str will have a slightly less convenient way of getting it, namely by > calling __fspath__() itself, instead of os.fspath(). > More cons: - It would be confusing that there'd be no direct corresponding between os.fspath(x) and x.__fspath__(), unlike for most other dunders (next(x) -> x.__next__(), iter(x) -> x.__iter__(), and so on). - Your examples like os.getcwd() have no input to determine whether to return bytes or str, so for those we use an alternate name to request bytes. However for most os methods the same method handles both, e.g. os.listdir(b'.') will return a list of bytes objects while os.listdir('.') will return a list of strings. os.fspath() firmly falls in the latter camp: it takes an input whose __fspath__() method will return either str or bytes, so that's all the guidance it needs. Really, if you want bytes, you should use os.fsencode(); if you want strings, use os.fsencode(); if you want to be polymorphic, use os.fspath() and check the type it returns. I find the case where you'd explicitly want to exclude bytes unusual; Nick already proposed pathlib.Path(os.fspath(x)) for that. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri May 13 12:24:54 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 13 May 2016 19:24:54 +0300 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <20160513110012.GH12028@ando.pearwood.info> References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: On Fri, May 13, 2016 at 2:00 PM, Steven D'Aprano wrote: > Counter suggestion: > > - __fspath__() method may return either bytes or str (no change > from the PEP as it stands now); > > - but os.fspath() will only return str; > > - and os.fspathb() will only return bytes; > > - there is no os function that returns "str or bytes, I don't > care which". (If you really need that, call __fspath__ directly.) > > Note that this differs from the already rejected suggestion that there > should be two dunder methods, __fspath__() and __fspathb__(). > > Why? > > (1) Normally, the caller knows whether they want str or bytes. (That's > been my experience, you may disagree.) If so, and they call os.fspath() > expecting a str, they won't be surprised by it returning bytes. And visa > versa for when you expect a bytes path. > > (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}. I would think these have the b suffix because there is no good way to infer which type should be returned. In things like os.path.join or os.path.dirname you pass in the object(s) that determine the return type. In os.fspath, you pass in an object, whose type (str/bytes) or "underlying path string type" (as returned by __fspath__()) determines the return type of fspath. I think this is well in line with os.path functions. -- Koos > > Cons: > > (3) Polymorphic code that truly doesn't care whether it gets bytes or > str will have a slightly less convenient way of getting it, namely by > calling __fspath__() itself, instead of os.fspath(). > From larry at hastings.org Fri May 13 12:33:10 2016 From: larry at hastings.org (Larry Hastings) Date: Fri, 13 May 2016 18:33:10 +0200 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: <573601C6.8080804@hastings.org> On 05/13/2016 06:21 PM, Guido van Rossum wrote: > Really, if you want bytes, you should use os.fsencode(); if you want > strings, use os.fsencode(); if you want to be polymorphic, use > os.fspath() and check the type it returns. Am I severely misunderstanding the API, or did you mean "if you want strings, use os.fsdecode()" here? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Fri May 13 12:33:47 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 13 May 2016 18:33:47 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> Message-ID: <573601EB.50904@mail.de> On 13.05.2016 17:29, Brett Cannon wrote: > Purposeful change. It was what I had in my head after I posted my > "groups" breakdown email and what Guido suggested as well > independently. This helps alleviate any perf worries as type checks in > C are pointer checks that are cheap to make compared to attribute > lookup. And even if we do get the community to move on to path objects > the check is still cheap enough to not worry. And finally, this is > simply how we have almost always coded this kind of special-casing at > the C level throughout Python and the stdlib, so it now is more inline > with standard coding practices than the original proposal in the PEP. Maybe, we have a misunderstanding here. Does "PyUnicode_Check(path)" respect subclass hierarchies such as isinstance(path, str) does? Could not find the definition of it in the source. You said it's just a pointer comparison. To me that implies it could not respect subclasses as isinstance does. If that is the case, the C implementation differs from the Python version in semantics. I'm sorry if I misunderstood the whole implementation here. Best, Sven From ethan at stoneleaf.us Fri May 13 12:34:36 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 May 2016 09:34:36 -0700 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: <5736021C.1070300@stoneleaf.us> On 05/13/2016 08:43 AM, Brett Cannon wrote: >> a minor technical query: >> >> try: >> return path.__fspath__() >> >> Would I be right in saying that in practice this will actually end up >> being type(path).__fspath__() to match the behaviour of all(?) other >> dunder methods? > > I wasn't planning on it because for most types the accessing of the > method directly off of the type for magic methods is because of some > special struct field at the C level that we're pulling from. Since we're > not planning to have an equivalent struct field I don't see any need to > do the extra work of avoiding the instance participating in method > lookup. Obviously if people disagree for some reason then please let me > know (maybe for perf by avoiding the overhead of checking for the method > on the instance?). I would say use `type(x).__fspath__`. I'm not aware of any other __dunder__ method that doesn't access the attribute from the type instead of the instance, and I see no point in making this one different. I know there's a line in the Zen about foolish conistencies, but I suspect there's a corollary about foolish inconsistencies. ;) -- ~Ethan~ From rosuav at gmail.com Fri May 13 12:43:19 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 14 May 2016 02:43:19 +1000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <573601EB.50904@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <573601EB.50904@mail.de> Message-ID: On Sat, May 14, 2016 at 2:33 AM, Sven R. Kunze wrote: > On 13.05.2016 17:29, Brett Cannon wrote: >> >> Purposeful change. It was what I had in my head after I posted my "groups" >> breakdown email and what Guido suggested as well independently. This helps >> alleviate any perf worries as type checks in C are pointer checks that are >> cheap to make compared to attribute lookup. And even if we do get the >> community to move on to path objects the check is still cheap enough to not >> worry. And finally, this is simply how we have almost always coded this kind >> of special-casing at the C level throughout Python and the stdlib, so it now >> is more inline with standard coding practices than the original proposal in >> the PEP. > > > Maybe, we have a misunderstanding here. Does "PyUnicode_Check(path)" respect > subclass hierarchies such as isinstance(path, str) does? Could not find the > definition of it in the source. > > You said it's just a pointer comparison. To me that implies it could not > respect subclasses as isinstance does. If that is the case, the C > implementation differs from the Python version in semantics. I'm sorry if I > misunderstood the whole implementation here. https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Check https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_CheckExact "Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x) is str"). The question is, which one SHOULD be being done? What should this do: class TmpPath(str): def __fspath__(self): return "/tmp/"+self x = TmpPath("foo/bar") open(x, "w") Does that respect __fspath__, or respect the fact that it's a string? ChrisA From rosuav at gmail.com Fri May 13 12:45:40 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 14 May 2016 02:45:40 +1000 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <5736021C.1070300@stoneleaf.us> References: <20160513110012.GH12028@ando.pearwood.info> <5736021C.1070300@stoneleaf.us> Message-ID: On Sat, May 14, 2016 at 2:34 AM, Ethan Furman wrote: > I would say use `type(x).__fspath__`. I'm not aware of any other __dunder__ > method that doesn't access the attribute from the type instead of the > instance, and I see no point in making this one different. > __reduce__ / __reduce_ex__ in pickle.py is accessed with a straight-forward getattr() call. It's the ones that are called from deep within the interpreter core (eg __iter__) that are always looked up on the type. ChrisA From p.f.moore at gmail.com Fri May 13 12:52:20 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 May 2016 17:52:20 +0100 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5735FB79.1010201@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <5735FB79.1010201@mail.de> Message-ID: On 13 May 2016 at 17:06, Sven R. Kunze wrote: > As far as I remember, one goal of this proposal was to avoid wallet gardens. > During the lengthy discussion on python-ideas people brought up that some > third-party libs indeed subclass from str. They are currently locked out. Hardly. Libraries that subclass from str will provide their string representation precisely by being (a subclass of) a string. So they don't need the __fspath__ protocol, and can continue to work exactly as now. This PEP merely provides the same capability for libraries that prefer *not* to subclass str. Paul From ethan at stoneleaf.us Fri May 13 12:57:19 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 May 2016 09:57:19 -0700 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5735FB79.1010201@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <5735FB79.1010201@mail.de> Message-ID: <5736076F.6010905@stoneleaf.us> On 05/13/2016 09:06 AM, Sven R. Kunze wrote: > On 13.05.2016 11:48, Koos Zevenhoven wrote: >> Sven R Kunze had previously queried: >>> However, the proposed semantics will change if the checks are swapped. So, >>> my actual question is: >>> >>> Is that an intended API inconsistency or a known bug supposed to be resolved >>> later? >> >> Taking into account the description (and the drafted type hint), which >> the documentation will probably reflect, the semantic effects of that >> are very minor or nonexistent. > > From your perspective. As far as I remember, one goal of this proposal > was to avoid wallet gardens. During the lengthy discussion on > python-ideas people brought up that some third-party libs indeed > subclass from str. They are currently locked out. Two points: 1) What is a wallet garden? 2) If a third-party path lib subclasses either str or bytes, and the os.* and other stdlib functions accept str or bytes, how are they locked out by not having an __fspath__? Assuming, of course, that those functions use isinstance and not type (if do use type a bug should be filed against them. > Indeed. Just one minor note here: str or bytes subclasses *can* > implement __fspath__ and currently it will be *ignored*. Sure. And int subclasses can implement __index__, which will likewise be ignored. >> So no API inconsistency there. > > API consistency is not defined by docs-matching-implementation but by > implementation-matching-expectations. Well, probably both. ;) -- ~Ethan~ From k7hoven at gmail.com Fri May 13 13:05:38 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 13 May 2016 20:05:38 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5735FB79.1010201@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <5735FB79.1010201@mail.de> Message-ID: On Fri, May 13, 2016 at 7:06 PM, Sven R. Kunze wrote: > On 13.05.2016 11:48, Koos Zevenhoven wrote: >> >> This issue is coupled with the future optimization questions. >> > > AFAIC coupling API design to optimization is called premature optimization. > I suppose so, but currently I consider the API not to make guarantees about what happens if you pass in something other than specified as valid input. I'm not sure what level of precision is usual at this point, but I think this is sufficient for now. I'm sure we will pay attention to this (as we have been doing, both while optimizing prematurely and for other reasons). -- Koos From guido at python.org Fri May 13 13:38:35 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 13 May 2016 10:38:35 -0700 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <573601C6.8080804@hastings.org> References: <20160513110012.GH12028@ando.pearwood.info> <573601C6.8080804@hastings.org> Message-ID: On Fri, May 13, 2016 at 9:33 AM, Larry Hastings wrote: > > > On 05/13/2016 06:21 PM, Guido van Rossum wrote: > > Really, if you want bytes, you should use os.fsencode(); if you want > strings, use os.fsencode(); if you want to be polymorphic, use os.fspath() > and check the type it returns. > > > Am I severely misunderstanding the API, or did you mean "if you want > strings, use os.fsdecode()" here? > encode, decode -- poh-tay-toe, poh-tah-toe. :-) Another slip of the fingers, I did mean os.fsdecode() when you want strings. encode, schmencode... -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri May 13 13:39:23 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 13 May 2016 10:39:23 -0700 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <5736021C.1070300@stoneleaf.us> References: <20160513110012.GH12028@ando.pearwood.info> <5736021C.1070300@stoneleaf.us> Message-ID: On Fri, May 13, 2016 at 9:34 AM, Ethan Furman wrote: > I would say use `type(x).__fspath__`. I'm not aware of any other > __dunder__ method that doesn't access the attribute from the type instead > of the instance, and I see no point in making this one different. > Agreed. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri May 13 13:42:23 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 May 2016 18:42:23 +0100 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5736076F.6010905@stoneleaf.us> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <5735FB79.1010201@mail.de> <5736076F.6010905@stoneleaf.us> Message-ID: On 13 May 2016 at 17:57, Ethan Furman wrote: > 1) What is a wallet garden? I assumed he meant "walled garden" (for people who don't know the phrase, it refers to an area or capability that's only made accessible to "favoured" clients - think of something like Apple's App Store, where you have to be approved to publish there). Paul From steve at pearwood.info Fri May 13 13:52:26 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 May 2016 03:52:26 +1000 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: References: <20160513110012.GH12028@ando.pearwood.info> Message-ID: <20160513175226.GI12028@ando.pearwood.info> On Fri, May 13, 2016 at 03:43:29PM +0000, Brett Cannon wrote: > On Fri, 13 May 2016 at 04:00 Steven D'Aprano wrote: [...] > > - but os.fspath() will only return str; > > - and os.fspathb() will only return bytes; > I prefer what's in the PEP. I get where you coming from, Steven, but I > don't think it will be common enough to worry about. Think of os.fspath() > like next() where it truly is a very minor convenience function that > happens to special-case str and bytes. Okay, I'm satisfied by the various arguments against this idea. [...] > > I think it is a bit confusing to refer to "path objects", as that seems > > like you are referring only to pathlib.Path objects. It took me far too > > long to realise that here you mean generic path-like objects that obey > > the __fspath__ protocol rather than a specific concrete class. > > > > Since the ABC is called "PathLike", I suggest we refer to "path-like > > objects" rather than "path objects", both in the PEP and in the Python > > docs for this protocol. > > > > I went back and forth with this in my head while writing the PEP. The > problem with making "path-like" mean "objects implementing the PathLike > ABC" becomes how do you refer to an argument of a function that accepts > anything os.fspath() does (i.e. PathLike, str, and bytes)? On further reflection, I think the right language is to use "path-like" for the union of Pathlike, str and bytes. That will, I think, cover the majority of cases: most functions which want to work on a file system path should accept all three. When you want to specify only an object which implements the PathLike ABC, that's called a (virtual) instance of PathLike. I see this as analogous to "iterable". An iterable is anything which can be iterated over. Sometimes that's an iterator. Sometimes its not an iterator. We don't have a special term for "iterable which is not an iterator", mostly because its rare to care about the distinction, but on those rare cases that we do care, we can describe it explicitly. In this case, "path-like" would be equivalent to iterable: anything that can be used as a file system path. Some path-like objects are actual pathlib.Path objects, some are some other PathLike object, and some are strings or bytes. [...] > > > try: > > > return path.__fspath__() > > > > Would I be right in saying that in practice this will actually end up > > being type(path).__fspath__() to match the behaviour of all(?) other > > dunder methods? > > > > I wasn't planning on it because for most types the accessing of the method > directly off of the type for magic methods is because of some special > struct field at the C level that we're pulling from. Since we're not > planning to have an equivalent struct field I don't see any need to do the > extra work of avoiding the instance participating in method lookup. > Obviously if people disagree for some reason then please let me know (maybe > for perf by avoiding the overhead of checking for the method on the > instance?). The reasons I would disagree are: (1) It took me a long time to learn the rule that dunder methods are always called from the class, not the instance, and now I have to learn that there are exceptions? Grrrr argggh. (2) If we ever do change to a C struct field, the behaviour will change. Maybe it's better to emulate the same behaviour from the start? (3) If there's a performance speed up, that's a bonus! -- Steven From brett at python.org Fri May 13 13:56:35 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 17:56:35 +0000 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <20160513175226.GI12028@ando.pearwood.info> References: <20160513110012.GH12028@ando.pearwood.info> <20160513175226.GI12028@ando.pearwood.info> Message-ID: On Fri, 13 May 2016 at 10:53 Steven D'Aprano wrote: > On Fri, May 13, 2016 at 03:43:29PM +0000, Brett Cannon wrote: > > On Fri, 13 May 2016 at 04:00 Steven D'Aprano > wrote: > > [...] > > > - but os.fspath() will only return str; > > > - and os.fspathb() will only return bytes; > > > I prefer what's in the PEP. I get where you coming from, Steven, but I > > don't think it will be common enough to worry about. Think of os.fspath() > > like next() where it truly is a very minor convenience function that > > happens to special-case str and bytes. > > Okay, I'm satisfied by the various arguments against this idea. > I'll add a note in the Rejected Ideas section about this. > > > [...] > > > I think it is a bit confusing to refer to "path objects", as that seems > > > like you are referring only to pathlib.Path objects. It took me far too > > > long to realise that here you mean generic path-like objects that obey > > > the __fspath__ protocol rather than a specific concrete class. > > > > > > Since the ABC is called "PathLike", I suggest we refer to "path-like > > > objects" rather than "path objects", both in the PEP and in the Python > > > docs for this protocol. > > > > > > > I went back and forth with this in my head while writing the PEP. The > > problem with making "path-like" mean "objects implementing the PathLike > > ABC" becomes how do you refer to an argument of a function that accepts > > anything os.fspath() does (i.e. PathLike, str, and bytes)? > > On further reflection, I think the right language is to use "path-like" > for the union of Pathlike, str and bytes. That will, I think, cover the > majority of cases: most functions which want to work on a file system > path should accept all three. When you want to specify only an object > which implements the PathLike ABC, that's called a (virtual) instance of > PathLike. > > I see this as analogous to "iterable". An iterable is anything which can > be iterated over. Sometimes that's an iterator. Sometimes its not an > iterator. We don't have a special term for "iterable which is not an > iterator", mostly because its rare to care about the distinction, but on > those rare cases that we do care, we can describe it explicitly. > > In this case, "path-like" would be equivalent to iterable: anything that > can be used as a file system path. Some path-like objects are actual > pathlib.Path objects, some are some other PathLike object, and some are > strings or bytes. > That was my general thinking. I'll add a TODO to add an entry in the glossary for "path-like". > > > [...] > > > > try: > > > > return path.__fspath__() > > > > > > Would I be right in saying that in practice this will actually end up > > > being type(path).__fspath__() to match the behaviour of all(?) other > > > dunder methods? > > > > > > > I wasn't planning on it because for most types the accessing of the > method > > directly off of the type for magic methods is because of some special > > struct field at the C level that we're pulling from. Since we're not > > planning to have an equivalent struct field I don't see any need to do > the > > extra work of avoiding the instance participating in method lookup. > > Obviously if people disagree for some reason then please let me know > (maybe > > for perf by avoiding the overhead of checking for the method on the > > instance?). > > The reasons I would disagree are: > > (1) It took me a long time to learn the rule that dunder methods are > always called from the class, not the instance, and now I have to learn > that there are exceptions? Grrrr argggh. > > (2) If we ever do change to a C struct field, the behaviour will change. > Maybe it's better to emulate the same behaviour from the start? > > (3) If there's a performance speed up, that's a bonus! > And Guido agrees with this as well so I'll be updating the PEP shortly after I remember how to do the equivalent from C. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 13 14:37:33 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 18:37:33 +0000 Subject: [Python-Dev] File system path PEP, 3rd draft Message-ID: Biggest changes since the second draft: 1. Resolve __fspath__() from the type, not the instance (for Guido) 2. Updated the TypeError messages to say "os.PathLike object" instead of "path object" (implicitly for Steven) 3. TODO item to define "path-like" in the glossary (for Steven) 4. Various more things added to Rejected Ideas 5. Added Koos as a co-author (for Koos :) ---------- PEP: NNN Title: Adding a file system path protocol Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon , Koos Zevenhoven Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-May-2016 Post-History: 11-May-2016, 12-May-2016, 13-May-2016 Abstract ======== This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only ``str`` and/or ``bytes`` file system paths are accepted. The goal is to facilitate the migration of users towards rich path objects while providing an easy way to work with code expecting ``str`` or ``bytes``. Rationale ========= Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path. This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not. To help elevate the representation of file system paths from their representation as strings and bytes to a richer object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone. One issue in converting path objects to strings comes from the fact that the only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file. Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library has become an issue. A solution that allows any path-representing object to declare that it is a path and a way to extract a low-level representation that all path objects could support is desired. This PEP then proposes to introduce a new protocol to be followed by objects which represent file system paths. Providing a protocol allows for explicit signaling of what objects represent file system paths as well as a way to extract a lower-level representation that can be used with older APIs which only support strings or bytes. Discussions regarding path objects that led to this PEP can be found in multiple threads on the python-ideas mailing list archive [#python-ideas-archive]_ for the months of March and April 2016 and on the python-dev mailing list archives [#python-dev-archive]_ during April 2016. Proposal ======== This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation. The other part deals with changes to Python's standard library to support the new protocol. These changes will also lead to the pathlib module dropping its provisional status. Protocol -------- The following abstract base class defines the protocol for an object to be considered a path object:: import abc import typing as t class PathLike(abc.ABC): """Abstract base class for implementing the file system path protocol.""" @abc.abstractmethod def __fspath__(self) -> t.Union[str, bytes]: """Return the file system path representation of the object.""" raise NotImplementedError Objects representing file system paths will implement the ``__fspath__()`` method which will return the ``str`` or ``bytes`` representation of the path. The ``str`` representation is the preferred low-level path representation as it is human-readable and what people historically represent paths as. Standard library changes ------------------------ It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary). The modules mentioned below, though, deserve specific details as they have either fundamental changes that empower the ability to use path objects, or entail additions/removal of APIs. builtins '''''''' ``open()`` [#builtins-open]_ will be updated to accept path objects as well as continue to accept ``str`` and ``bytes``. os ''' The ``fspath()`` function will be added with the following semantics:: import typing as t def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]: """Return the string representation of the path. If str or bytes is passed in, it is returned unchanged. """ if isinstance(path, (str, bytes)): return path # Work from the object's type to match method resolution of other magic # methods. path_type = type(path) try: return path_type.__fspath__(path) except AttributeError: if hasattr(path_type, '__fspath__'): raise raise TypeError("expected str, bytes or os.PathLike object, not " + path_type.__name__) The ``os.fsencode()`` [#os-fsencode]_ and ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept path objects. As both functions coerce their arguments to ``bytes`` and ``str``, respectively, they will be updated to call ``__fspath__()`` if present to convert the path object to a ``str`` or ``bytes`` representation, and then perform their appropriate coercion operations as if the return value from ``__fspath__()`` had been the original argument to the coercion function in question. The addition of ``os.fspath()``, the updates to ``os.fsencode()``/``os.fsdecode()``, and the current semantics of ``pathlib.PurePath`` provide the semantics necessary to get the path representation one prefers. For a path object, ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or ``bytes`` representation without any coersion, then ``os.fspath()`` can be used. If a ``str`` is desired and the encoding of ``bytes`` should be assumed to be the default file system encoding, then ``os.fsdecode()`` should be used. If a ``bytes`` representation is desired and any strings should be encoded using the default file system encoding, then ``os.fsencode()`` is used. This PEP recommends using path objects when possible and falling back to string paths as necessary and using ``bytes`` as a last resort. Another way to view this is as a hierarchy of file system path representations (highest- to lowest-level): path ? str ? bytes. The functions and classes under discussion can all accept objects on the same level of the hierarchy, but they vary in whether they promote or demote objects to another level. The ``pathlib.PurePath`` class can promote a ``str`` to a path object. The ``os.fspath()`` function can demote a path object to a ``str`` or ``bytes`` instance, depending on what ``__fspath__()`` returns. The ``os.fsdecode()`` function will demote a path object to a string or promote a ``bytes`` object to a ``str``. The ``os.fsencode()`` function will demote a path or string object to ``bytes``. There is no function that provides a way to demote a path object directly to ``bytes`` while bypassing string demotion. The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` method. It will return the same value as currently found on the ``path`` attribute of ``DirEntry`` instances. The Protocol_ ABC will be added to the ``os`` module under the name ``os.PathLike``. os.path ''''''' The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use ``os.fspath()``. During the discussions leading up to this PEP it was suggested that ``os.path`` not be updated using an "explicit is better than implicit" argument. The thinking was that since ``__fspath__()`` is polymorphic itself it may be better to have code working with ``os.path`` extract the path representation from path objects explicitly. There is also the consideration that adding support this deep into the low-level OS APIs will lead to code magically supporting path objects without requiring any documentation updated, leading to potential complaints when it doesn't work, unbeknownst to the project author. But it is the view of this PEP that "practicality beats purity" in this instance. To help facilitate the transition to supporting path objects, it is better to make the transition as easy as possible than to worry about unexpected/undocumented duck typing support for path objects by projects. There has also been the suggestion that ``os.path`` functions could be used in a tight loop and the overhead of checking or calling ``__fspath__()`` would be too costly. In this scenario only path-consuming APIs would be directly updated and path-manipulating APIs like the ones in ``os.path`` would go unmodified. This would require library authors to update their code to support path objects if they performed any path manipulations, but if the library code passed the path straight through then the library wouldn't need to be updated. It is the view of this PEP and Guido, though, that this is an unnecessary worry and that performance will still be acceptable. pathlib ''''''' The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path`` will continue to not accept ``bytes`` path representations, and so if ``__fspath__()`` returns ``bytes`` it will raise an exception. The ``path`` attribute will be removed as this PEP makes it redundant (it has not been included in any released version of Python and so is not a backwards-compatibility concern). C API ''''' The C API will gain an equivalent function to ``os.fspath()``:: /* Return the file system path of the object. If the object is str or bytes, then allow it to pass through with an incremented refcount. If the object defines __fspath__(), then return the result of that method. All other types raise a TypeError. */ PyObject * PyOS_FSPath(PyObject *path) { if (PyUnicode_Check(path) || PyBytes_Check(path)) { Py_INCREF(path); return path; } if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__", path, NULL); } return PyErr_Format(PyExc_TypeError, "expected a str, bytes, or os.PathLike object, not %S", path->ob_type); } Backwards compatibility ======================= There are no explicit backwards-compatibility concerns. Unless an object incidentally already defines a ``__fspath__()`` method there is no reason to expect the pre-existing code to break or expect to have its semantics implicitly changed. Libraries wishing to support path objects and a version of Python prior to Python 3.6 and the existence of ``os.fspath()`` can use the idiom of ``path.__fspath__() if hasattr(path, "__fspath__") else path``. Implementation ============== This is the task list for what this PEP proposes: #. Remove the ``path`` attribute from pathlib #. Remove the provisional status of pathlib #. Add ``os.PathLike`` #. Add ``os.fspath()`` #. Add ``PyOS_FSPath()`` #. Update ``os.fsencode()`` #. Update ``os.fsdecode()`` #. Update ``pathlib.PurePath`` and ``pathlib.Path`` #. Update ``builtins.open()`` #. Update ``os.DirEntry`` #. Update ``os.path`` #. Add a glossary entry for "path-like" Rejected Ideas ============== Other names for the protocol's method ------------------------------------- Various names were proposed during discussions leading to this PEP, including ``__path__``, ``__pathname__``, and ``__fspathname__``. In the end people seemed to gravitate towards ``__fspath__`` for being unambiguous without being unnecessarily long. Separate str/bytes methods -------------------------- At one point it was suggested that ``__fspath__()`` only return strings and another method named ``__fspathb__()`` be introduced to return bytes. The thinking is that by making ``__fspath__()`` not be polymorphic it could make dealing with the potential string or bytes representations easier. But the general consensus was that returning bytes will more than likely be rare and that the various functions in the os module are the better abstraction to promote over direct calls to ``__fspath__()``. Providing a ``path`` attribute ------------------------------ To help deal with the issue of ``pathlib.PurePath`` not inheriting from ``str``, originally it was proposed to introduce a ``path`` attribute to mirror what ``os.DirEntry`` provides. In the end, though, it was determined that a protocol would provide the same result while not directly exposing an API that most people will never need to interact with directly. Have ``__fspath__()`` only return strings ------------------------------------------ Much of the discussion that led to this PEP revolved around whether ``__fspath__()`` should be polymorphic and return ``bytes`` as well as ``str`` or only return ``str``. The general sentiment for this view was that ``bytes`` are difficult to work with due to their inherent lack of information about their encoding and PEP 383 makes it possible to represent all file system paths using ``str`` with the ``surrogateescape`` handler. Thus, it would be better to forcibly promote the use of ``str`` as the low-level path representation for high-level path objects. In the end, it was decided that using ``bytes`` to represent paths is simply not going to go away and thus they should be supported to some degree. The hope is that people will gravitate towards path objects like pathlib and that will move people away from operating directly with ``bytes``. A generic string encoding mechanism ----------------------------------- At one point there was a discussion of developing a generic mechanism to extract a string representation of an object that had semantic meaning (``__str__()`` does not necessarily return anything of semantic significance beyond what may be helpful for debugging). In the end, it was deemed to lack a motivating need beyond the one this PEP is trying to solve in a specific fashion. Have __fspath__ be an attribute ------------------------------- It was briefly considered to have ``__fspath__`` be an attribute instead of a method. This was rejected for two reasons. One, historically protocols have been implemented as "magic methods" and not "magic methods and attributes". Two, there is no guarantee that the lower-level representation of a path object will be pre-computed, potentially misleading users that there was no expensive computation behind the scenes in case the attribute was implemented as a property. This also indirectly ties into the idea of introducing a ``path`` attribute to accomplish the same thing. This idea has an added issue, though, of accidentally having any object with a ``path`` attribute meet the protocol's duck typing. Introducing a new magic method for the protocol helpfully avoids any accidental opting into the protocol. Provide specific type hinting support ------------------------------------- There was some consideration to provdinga generic ``typing.PathLike`` class which would allow for e.g. ``typing.PathLike[str]`` to specify a type hint for a path object which returned a string representation. While potentially beneficial, the usefulness was deemed too small to bother adding the type hint class. This also removed any desire to have a class in the ``typing`` module which represented the union of all acceptable path-representing types as that can be represented with ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope is users will slowly gravitate to path objects only. Provide ``os.fspathb()`` ------------------------ It was suggested that to mirror the structure of e.g. ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return ``str`` and that another function named ``os.fspathb()`` be introduced that only returned ``bytes``. This was rejected as the purposes of the ``*b()`` functions are tied to querying the file system where there is a need to get the raw bytes back. As this PEP does not work directly with data on a file system (but which *may* be), the view was taken this distinction is unnecessary. It's also believed that the need for only bytes will not be common enough to need to support in such a specific manner as ``os.fsencode()`` will provide similar functionality. Call ``__fspath__()`` off of the instance ----------------------------------------- An earlier draft of this PEP had ``os.fspath()`` calling ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The changed to be consistent with how other magic methods in Python are resolved. Acknowledgements ================ Thanks to everyone who participated in the various discussions related to this PEP that spanned both python-ideas and python-dev. Special thanks to Stephen Turnbull for direct feedback on early drafts of this PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not only feedback on early drafts of this PEP but also helping to drive the overall discussion on this topic across the two mailing lists. References ========== .. [#python-ideas-archive] The python-ideas mailing list archive (https://mail.python.org/pipermail/python-ideas/) .. [#python-dev-archive] The python-dev mailing list archive (https://mail.python.org/pipermail/python-dev/) .. [#libc-open] ``open()`` documention for the C standard library ( http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html ) .. [#pathlib] The ``pathlib`` module (https://docs.python.org/3/library/pathlib.html#module-pathlib) .. [#builtins-open] The ``builtins.open()`` function (https://docs.python.org/3/library/functions.html#open) .. [#os-fsencode] The ``os.fsencode()`` function (https://docs.python.org/3/library/os.html#os.fsencode) .. [#os-fsdecode] The ``os.fsdecode()`` function (https://docs.python.org/3/library/os.html#os.fsdecode) .. [#os-direntry] The ``os.DirEntry`` class (https://docs.python.org/3/library/os.html#os.DirEntry) .. [#os-path] The ``os.path`` module (https://docs.python.org/3/library/os.path.html#module-os.path) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri May 13 15:08:10 2016 From: barry at python.org (Barry Warsaw) Date: Fri, 13 May 2016 15:08:10 -0400 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: <20160513150810.2ae51e79.barry@wooz.org> On May 13, 2016, at 06:37 PM, Brett Cannon wrote: >PEP: NNN >Title: Adding a file system path protocol >Version: $Revision$ >Last-Modified: $Date$ >Author: Brett Cannon , > Koos Zevenhoven >Status: Draft >Type: Standards Track >Content-Type: text/x-rst >Created: 11-May-2016 >Post-History: 11-May-2016, > 12-May-2016, > 13-May-2016 I think this is far enough along that you should self-assign a PEP number and check it into the official repo. Cheers, -Barry From brett at python.org Fri May 13 15:24:17 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 19:24:17 +0000 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: <20160513150810.2ae51e79.barry@wooz.org> References: <20160513150810.2ae51e79.barry@wooz.org> Message-ID: On Fri, 13 May 2016 at 12:08 Barry Warsaw wrote: > On May 13, 2016, at 06:37 PM, Brett Cannon wrote: > > >PEP: NNN > >Title: Adding a file system path protocol > >Version: $Revision$ > >Last-Modified: $Date$ > >Author: Brett Cannon , > > Koos Zevenhoven > >Status: Draft > >Type: Standards Track > >Content-Type: text/x-rst > >Created: 11-May-2016 > >Post-History: 11-May-2016, > > 12-May-2016, > > 13-May-2016 > > I think this is far enough along that you should self-assign a PEP number > and > check it into the official repo. > Paraphrasing *someone* from Twitter yesterday, I'm just going to work on GitHub until I absolutely have to check into hg, m'kay? ;) -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri May 13 15:26:24 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 13 May 2016 22:26:24 +0300 Subject: [Python-Dev] File system path PEP, part 2 In-Reply-To: <20160513175226.GI12028@ando.pearwood.info> References: <20160513110012.GH12028@ando.pearwood.info> <20160513175226.GI12028@ando.pearwood.info> Message-ID: On Fri, May 13, 2016 at 8:52 PM, Steven D'Aprano wrote: > On Fri, May 13, 2016 at 03:43:29PM +0000, Brett Cannon wrote: >> On Fri, 13 May 2016 at 04:00 Steven D'Aprano wrote: > > [...] >> > I think it is a bit confusing to refer to "path objects", as that seems >> > like you are referring only to pathlib.Path objects. It took me far too >> > long to realise that here you mean generic path-like objects that obey >> > the __fspath__ protocol rather than a specific concrete class. >> > This terminology is indeed a bit difficult, not least because there are 6 different path classes in pathlib. A couple of months ago, I decided to start to call these pathlib objects and path objects, because I did not know what else to call them without doubling the length. >> > Since the ABC is called "PathLike", I suggest we refer to "path-like >> > objects" rather than "path objects", both in the PEP and in the Python >> > docs for this protocol. >> >> I went back and forth with this in my head while writing the PEP. The >> problem with making "path-like" mean "objects implementing the PathLike >> ABC" becomes how do you refer to an argument of a function that accepts >> anything os.fspath() does (i.e. PathLike, str, and bytes)? > > On further reflection, I think the right language is to use "path-like" > for the union of Pathlike, str and bytes. That will, I think, cover the > majority of cases: most functions which want to work on a file system > path should accept all three. When you want to specify only an object > which implements the PathLike ABC, that's called a (virtual) instance of > PathLike. > As I've told Brett before, I think exactly this reasoning would be a good enough reason not to call the ABC PathLike. [...] >> > Would I be right in saying that in practice this will actually end up >> > being type(path).__fspath__() to match the behaviour of all(?) other >> > dunder methods? >> >> I wasn't planning on it because for most types the accessing of the method >> directly off of the type for magic methods is because of some special >> struct field at the C level that we're pulling from. Since we're not >> planning to have an equivalent struct field I don't see any need to do the >> extra work of avoiding the instance participating in method lookup. >> Obviously if people disagree for some reason then please let me know (maybe >> for perf by avoiding the overhead of checking for the method on the >> instance?). > > The reasons I would disagree are: > > (1) It took me a long time to learn the rule that dunder methods are > always called from the class, not the instance, and now I have to learn > that there are exceptions? Grrrr argggh. Who knows, maybe the exception made it take a longer time.. IIRC the docs say that dunder methods are *not guaranteed* to be called on the instance, because they may be called on the class. > (2) If we ever do change to a C struct field, the behaviour will change. > Maybe it's better to emulate the same behaviour from the start? > > (3) If there's a performance speed up, that's a bonus! > > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com From guido at python.org Fri May 13 15:31:04 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 13 May 2016 12:31:04 -0700 Subject: [Python-Dev] Adding Type[C] to PEP 484 Message-ID: I've got another proposed update to PEP 484 ready. There was previously some discussion on python-ideas (Bcc'ed for closure) and quite a bit on the typing issue tracker: https://github.com/python/typing/issues/107 I've worked all that into a patch for PEP 484: https://github.com/python/typing/pull/218 Once we're okay with that I will work on getting the implementation ready in time for Python 3.5.2. Here's the added text: Meta-types ---------- A meta-type can be used to indicate a value that is itself a class object that is a subclass of a given class. It is spelled as ``Type[C]`` where ``C`` is a class. To clarify: while ``C`` (when used as an annotation) refers to instances of class ``C``, ``Type[C]`` refers to *subclasses* of ``C``. (This is a similar distinction as between ``object`` and ``type``.) For example, suppose we have the following classes:: class User: ... # Abstract base for User classes class BasicUser(User): ... class ProUser(User): ... class TeamUser(User): ... And suppose we have a function that creates an instance of one of these classes if you pass it a class object:: def new_user(user_class): user = user_class() # (Here we could write the user object to a database) return user Without ``Type[]`` the best we could do to annotate ``new_user()`` would be:: def new_user(user_class: type) -> User: ... However using ``Type[]`` and a type variable with an upper bound we can do much better:: U = TypeVar('U', bound=User) def new_user(user_class: Type[U]) -> U: ... Now when we call ``new_user()`` with a specific subclass of ``User`` a type checker will infer the correct type of the result:: joe = new_user(BasicUser) # Inferred type is BasicUser At runtime the value corresponding to ``Type[C]`` must be an actual class object that's a subtype of ``C``, not a special form. IOW, in the above example calling e.g. ``new_user(Union[BasicUser, ProUser])`` is not allowed. There are some concerns with this feature: for example when ``new_user()`` calls ``user_class()`` this implies that all subclasses of ``User`` must support this in their constructor signature. However this is not unique to ``Type[]``: class methods have similar concerns. A type checker ought to flag violations of such assumptions, but by default constructor calls that match the constructor signature in the indicated base class (``User`` in the example above) should be allowed. A program containing a complex or extensible class hierarchy might also handle this by using a factory class method. Plain ``Type`` without brackets is equivalent to using ``type`` (the root of Python's metaclass hierarchy). This equivalence also motivates the name, ``Type``, as opposed to alternatives like ``Class`` or ``SubType``, which were proposed while this feature was under discussion; this is similar to the relationship between e.g. ``List`` and ``list``. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 13 15:53:03 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 19:53:03 +0000 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: References: Message-ID: On Fri, 13 May 2016 at 12:38 Guido van Rossum wrote: > I've got another proposed update to PEP 484 ready. There was previously > some discussion on python-ideas (Bcc'ed for closure) and quite a bit on the > typing issue tracker: https://github.com/python/typing/issues/107 > > I've worked all that into a patch for PEP 484: > https://github.com/python/typing/pull/218 > > Once we're okay with that I will work on getting the implementation ready > in time for Python 3.5.2. > +1 from me. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri May 13 15:55:40 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 13 May 2016 12:55:40 -0700 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: <20160513150810.2ae51e79.barry@wooz.org> Message-ID: Can't wait until the peps repo is on GitHub! On Fri, May 13, 2016 at 12:24 PM, Brett Cannon wrote: > > > On Fri, 13 May 2016 at 12:08 Barry Warsaw wrote: > >> On May 13, 2016, at 06:37 PM, Brett Cannon wrote: >> >> >PEP: NNN >> >Title: Adding a file system path protocol >> >Version: $Revision$ >> >Last-Modified: $Date$ >> >Author: Brett Cannon , >> > Koos Zevenhoven >> >Status: Draft >> >Type: Standards Track >> >Content-Type: text/x-rst >> >Created: 11-May-2016 >> >Post-History: 11-May-2016, >> > 12-May-2016, >> > 13-May-2016 >> >> I think this is far enough along that you should self-assign a PEP number >> and >> check it into the official repo. >> > > Paraphrasing *someone* from Twitter yesterday, I'm just going to work on > GitHub until I absolutely have to check into hg, m'kay? ;) > > -Brett > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 13 16:31:33 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 20:31:33 +0000 Subject: [Python-Dev] quick GitHub transition update (was: File system path PEP, 3rd draft) In-Reply-To: References: <20160513150810.2ae51e79.barry@wooz.org> Message-ID: On Fri, 13 May 2016 at 12:56 Guido van Rossum wrote: > Can't wait until the peps repo is on GitHub! > With devinabox now moved over to GitHub, the peps repo is next on the list to move (and the devguide after that). I don't know if I'm going to be able to make the migration happen before PyCon US, but my goal is to have it either done before the language summit or have every little step lined up by the end of the sprints and then finished up the week following. I got one of the coveted 20 minute timeslots at the language summit to give an update on everything and soliciting feedback on what needs to be in place to migrate the cpython repo so I can have a clear idea of what work is left to do. If people want to prep for this discussion (which will be reported back here so everyone can participate in the end), think about what you *absolutely* must have in place to make working with GitHub for the cpython repo work for you. I don't want any emails on this now (I've got 3 PEPs at various stages plus travel between now and PyCon at this point so I'm busy enough as it is, so any emails discussing this now *will be ignored* as I don't have the mental bandwidth for that discussion prior to PyCon US). Please just reflect on the required parts of your workflow over the next 2 weeks to know how much work you are going to ask of *me* as minimum requirements to make the GitHub migration of the cpython repo happen (i.e. I don't care about "nice to have", only about "without this I flat-out won't contribute to Python on GitHub"). > On Fri, May 13, 2016 at 12:24 PM, Brett Cannon wrote: > >> >> >> On Fri, 13 May 2016 at 12:08 Barry Warsaw wrote: >> >>> On May 13, 2016, at 06:37 PM, Brett Cannon wrote: >>> >>> >PEP: NNN >>> >Title: Adding a file system path protocol >>> >Version: $Revision$ >>> >Last-Modified: $Date$ >>> >Author: Brett Cannon , >>> > Koos Zevenhoven >>> >Status: Draft >>> >Type: Standards Track >>> >Content-Type: text/x-rst >>> >Created: 11-May-2016 >>> >Post-History: 11-May-2016, >>> > 12-May-2016, >>> > 13-May-2016 >>> >>> I think this is far enough along that you should self-assign a PEP >>> number and >>> check it into the official repo. >>> >> >> Paraphrasing *someone* from Twitter yesterday, I'm just going to work on >> GitHub until I absolutely have to check into hg, m'kay? ;) >> >> -Brett >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjenvey at underboss.org Fri May 13 17:30:10 2016 From: pjenvey at underboss.org (Philip Jenvey) Date: Fri, 13 May 2016 14:30:10 -0700 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: > On May 13, 2016, at 11:37 AM, Brett Cannon wrote: > > Biggest changes since the second draft: > Resolve __fspath__() from the type, not the instance (for Guido) > if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { > return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__", path, > NULL); _PyObject_LookupSpecial would be preferable then. -- Philip Jenvey -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 13 17:48:04 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 21:48:04 +0000 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: On Fri, 13 May 2016 at 14:30 Philip Jenvey wrote: > > On May 13, 2016, at 11:37 AM, Brett Cannon wrote: > > Biggest changes since the second draft: > > 1. Resolve __fspath__() from the type, not the instance (for Guido) > > > if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { > return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__", > path, > NULL); > > > _PyObject_LookupSpecial would be preferable then. > Yes it would be. :) I'll add it to the PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri May 13 18:49:24 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 14 May 2016 01:49:24 +0300 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: FYI, I recently sent a couple of emails in my earlier type hinting thread on -ideas. What I wrote now is about the path ABC regarding type hints. -- Koos On Sat, May 14, 2016 at 12:48 AM, Brett Cannon wrote: > > > On Fri, 13 May 2016 at 14:30 Philip Jenvey wrote: >> >> >> On May 13, 2016, at 11:37 AM, Brett Cannon wrote: >> >> Biggest changes since the second draft: >> >> Resolve __fspath__() from the type, not the instance (for Guido) >> >> >> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { >> return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__", >> path, >> NULL); >> >> >> _PyObject_LookupSpecial would be preferable then. > > > Yes it would be. :) I'll add it to the PEP. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com > From nad at python.org Fri May 13 19:21:40 2016 From: nad at python.org (Ned Deily) Date: Fri, 13 May 2016 19:21:40 -0400 Subject: [Python-Dev] First 3.6.0 alpha release coming up: 2016-05-16 12:00 UTC Message-ID: <9B21FE29-226F-4DC6-AA8E-4651CDCD1BDB@python.org> Hi everyone! Thank you for all of your work so far on our next feature release cycle, Python 3.6. Code checkins for 3.6 began 12 months ago, after the feature code cutoff and beta phase for Python 3.5.0. We are now about to do 3.6.0 alpha 1, our first of 4 alpha and 4 beta releases planned for the next 6 months. As a reminder, alpha releases are intended to make it easier for the wider community to test the current state of new features and bug fixes for an upcoming Python release as a whole and for us to test the release process. The start of the alpha phase does *not* mean feature development is complete. During the alpha phase, features may be added, modified, or deleted up until the start of the beta phase. Alpha users beware! 3.6.0 beta 1, currently scheduled for 2016-09-07, marks feature code cutoff, at which time new feature development is expected to be complete and no additional features are to be added to 3.6. At the same time, feature development will begin for the following release cycle, 3.7. During the 3.6.0 beta phase, which is planned to end on 2016-12-04 with the first final release candidate, the focus will be on bug fixes and regressions and preparing for the final release. The goal is for the first release candidate to be the same as the final release: that is, no changes will be accepted for 3.6.0 after release candidate 1 without discussion, review, and release manager approval. Please plan accordingly! Although I've been involved behind the scenes in most releases over the past several years, 3.6.0a1 will be my first as a release manager so no doubt there will be bumps along the way. Many thanks to my fellow release managers, particularly Benjamin, Georg, Larry, and Barry, for their help and support. Your understanding, help, and feedback will be greatly appreciated! For this first alpha release at least, the release team (primarily Steve Dower and myself) are going to try a slightly different timetable for producing the release. We are planning for the code cutoff for alpha 1 to be on 2016-05-16 around 12:00 UTC; that allows for code checkins to continue through Sunday almost anywhere on the planet. We will then go off and manufacture the release bits and make them available via the python.org website as soon as possible thereafter, assuming no major glitches, that should be no later than 24 hours thereafter. In the remaining hours until the code cutoff, please get your commits in as soon as possible and please pay attention to the buildbots for possible regressions. Looking ahead, the next alpha release, 3.6.0a2, will follow in about a month on 2016-06-12, which is a week after the end of the development sprints at US PyCon 2016 in Portland, Oregon. Hope to see you there! To summarize: 2016-05-16 ~12:00 UTC: code snapshot for 3.6.0 alpha 1 now to 2016-09-07: Alpha phase (unrestricted feature development) 2016-09-07: 3.6.0 feature code freeze, 3.7.0 feature development begins 2016-09-07 to 2016-12-04: 3.6.0 beta phase (bug and regression fixes, no new features) 2016-12-04 3.6.0 release candidate 1 (3.6.0 code freeze) 2016-12-16 3.6.0 release (3.6.0rc1 plus, if necessary, any dire emergency fixes) Let me know if you have questions or comments. And thanks again for all of your work towards making Python even better! --Ned D AKA The Other Ned https://www.python.org/dev/peps/pep-0494/ https://docs.python.org/devguide/devcycle.html#stages https://en.wikipedia.org/wiki/Software_release_life_cycle#Alpha https://docs.python.org/devguide/buildbots.html -- Ned Deily nad at python.org -- [] Freenode IRC: ned_deily, #python-dev From greg.ewing at canterbury.ac.nz Fri May 13 21:02:55 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 14 May 2016 13:02:55 +1200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <5735FB79.1010201@mail.de> <5736076F.6010905@stoneleaf.us> Message-ID: <5736793F.5030401@canterbury.ac.nz> Paul Moore wrote: > On 13 May 2016 at 17:57, Ethan Furman wrote: > >>1) What is a wallet garden? > > I assumed he meant "walled garden" Works either way -- you'd want a wall around your wallet garden to stop people stealing your wallets. -- Greg From stephen at xemacs.org Sat May 14 02:56:30 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 14 May 2016 15:56:30 +0900 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <20160512172225.GA16846@sjoerdjob.com> <20160513101939.GG12028@ando.pearwood.info> Message-ID: <22326.52254.70620.535515@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > AFAICT, the compatibility layer would simply decode the bytes using > surrogateescape handling, which should round-trip anything. By design. See PEP 383. Or rather, the OP should; he has not done his homework and is confused by his own FUD. This whole subthread is really python-list territory. Since a lot of people I respect seem uncertain about the facts, for the record, let's lay out the (putative) issues remaining for post-PEP-383 Python vs. str-y path objects. (0) "Can't work with some POSIX (bytes) paths" is closed by PEP 383, forget it. os.fsdecode(bytespath) as soon as you get one, os.fsencode(strpath) just before you need one, done. Surrogates embedded in strpath may need special handling depending on the application (see (1)). (1) str.encode(errors='strict') (the default) will blow up on embedded surrogates. Yes, but that's a *good* thing if you're mixing str derived from filesystem paths with other text. There's no way to avoid it. If you're just passing it back to open(), it Just Works, done. (2) You're using bytes as text a la 2.x for "efficiency's" sake, and you're worried that you might pass a str-y Path deep into bytes territory and it will explode there. I don't think there is any sympathy left for that use case on Python dev channels. Define a clear boundary with well-defined entry and exit gates, and convert there. Then you can get some sleep. (How-to example: your "compatibility layer".) (3) You're worried about inefficiency of decoding/encoding to the same or trivially changed bytes (ie, you didn't need pathlib in the first place, but you got it anyway) -- this especially matters for 2.7, but is significant for 3.x too, if you're using a bunch of paths in a tight loop. I don't have sympathy for that use case, but Brett and Guido do, and Brett's PEP handles it by making __fspath__ polymorphic in the usual os.path-y way, with Guido's modification. This is always a tradeoff. If you know your JPEGs all have extension '.JPG' and png_path = jpeg_path[:-4] + b'.png' is readable enough for you, use that, not pathlib or Antipathy, and you get your efficiency. (Doing jpeg_path.rindex(b'.') is left as an exercise for the reader. Part (i): Is it really worth it?) If you want the readability of a rich path library and the efficiency of bytes, you *may* have the option of using Ethan's Antipathy (or whatever). If you can't use Antipathy, use bytes methods directly, or accept that it isn't *that* inefficient and use pathlib. At this point, I think this subcase is just FUD, no real examples were presented where the efficiency hit of encoding/decoding gets in the way of getting work done using pathlib. If you need to stick to stdlib for some reason (eg, to use a higher-level library that uses pathlib), live with the "compatibility layer"'s inefficiency. Decoding and encoding are actually rather low-cost operations at path lengths (PATHMAX=256 was common, not so long ago!). Most high-level libraries will impose a lot more overhead elsewhere, and calling into pathlib by itself will add a certain amount of overhead as well. (4) Lack of transparency/readability for "simple" operations. If Antipathy is something you can use, I agree it's plausible that avoiding a few os.fsdecode and os.fsencode calls would look nicer, but this is really a style question. My take: I think of paths as human-readable, so presenting them as str (not bytes) is important to me, important enough that I advocate that position to other developers. If you do the conversion at the boundary between a bytes-y module and pathlib ("compatibility layer") I don't see how it affects readability of the path manipulation code, while data marshaling at boundaries is a expected fact of software development. YMMV. (0) is thus a non-issue. (1) is not something that can be addressed by general principles, let alone language design. (2)-(4) are all real issues regardless of how I feel they should be resolved :-), but they're all design trade-offs, not things that can completely block you from getting some kinds of work done in your own style (eg, the situation str-minded people were in before PEP 383). Python 3 is an example of how language design can help alleviate issues like (2), by discouraging that use case in various ways. Brett's PEP is an example of how language design can help alleviate issues like (3) and (4). In particular, it helps us to interface pathlib to open() and friends in a very natural, readable way, without explicit conversions that should be unnecessary by the nature of the operation and its arguments. By contrast, the conversion of bytes to str is important to do explicitly because they are different representations of the same thing, and it's important that readers be notified of that change of representation. > Or am I wrong here somewhere? Well, considering the length of this irrelevant-to-the-PEP subthread, arguably you are feeding a successful troll. I hope that having posted the above, in the future there will be *one*, *short* reply to such questions: Not a problem. Read PEP 383. and the thread will end there. Steve From srkunze at mail.de Sat May 14 05:36:08 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Sat, 14 May 2016 11:36:08 +0200 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <573601EB.50904@mail.de> Message-ID: <5736F188.90705@mail.de> On 13.05.2016 18:43, Chris Angelico wrote: > https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Check > > https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_CheckExact Thanks for pointing me at this. I searched via github and found usages only: https://github.com/python/cpython/search?utf8=%E2%9C%93&q=PyUnicode_Check > "Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x) > is str"). The question is, which one SHOULD be being done? What should > this do: > > class TmpPath(str): > def __fspath__(self): > return "/tmp/"+self > x = TmpPath("foo/bar") > open(x, "w") > > Does that respect __fspath__, or respect the fact that it's a string? Fine example. Thinking naively, I would say, when somebody made an effort to write __fspath__, it should be respected. Maybe, that's just me. Not sure if that can be a source of errors, when somebody adds str as baseclass later because the team needs str functionality. At least I would expect __fspath__ to still work . Best, Sven From ncoghlan at gmail.com Sat May 14 08:24:04 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 May 2016 22:24:04 +1000 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5736F188.90705@mail.de> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <573601EB.50904@mail.de> <5736F188.90705@mail.de> Message-ID: On 14 May 2016 at 19:36, Sven R. Kunze wrote: > Fine example. Thinking naively, I would say, when somebody made an effort to > write __fspath__, it should be respected. Maybe, that's just me. Over time, we've generally moved towards treating subclasses as if they were the base class where protocols are concerned, and encouraging the use of composition if people don't actually want that "is-a" behaviour. For example, compare the int protocol to the last major type protocol we added, index: >>> from operator import index >>> class MyInt(int): ... def __int__(self): ... return 42 ... def __index__(self): ... return 42 ... >>> x = MyInt(5) >>> x 5 >>> int(x) 42 >>> index(x) 5 In the case of retroactively added protocols like operator.index() and os.fspath(), where inheritance from the relevant builtin type was previously the only option, retaining that behaviour is also the only way to avoid a performance regression for those subclasses. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pludemann at google.com Sat May 14 14:18:59 2016 From: pludemann at google.com (Peter Ludemann) Date: Sat, 14 May 2016 11:18:59 -0700 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: References: Message-ID: Is Type[C,D] allowed? Or should multiple types be restricted to TypeVar? -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Sat May 14 14:30:00 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat, 14 May 2016 11:30:00 -0700 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: References: Message-ID: What would Type[C, D] mean? --Guido (mobile) On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" < python-dev at python.org> wrote: > Is Type[C,D] allowed? Or should multiple types be restricted to TypeVar? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pludemann at google.com Sat May 14 14:47:15 2016 From: pludemann at google.com (Peter Ludemann) Date: Sat, 14 May 2016 11:47:15 -0700 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: References: Message-ID: I think that Type[C,D] would mean Union[Type[C], Type[D]], but I'm not sure ... I should probably talk to a typing expert about this. (Sorry for thinking out loud; but if we decide that Type[C,D] doesn't make sense, we need to prohibit it) I suppose I'm asking: do we allow new_user(Type[BasicUser, ProUser])? Here's a trivial example that off the top of my head makes sense: BasicOrProUser = TypeVar('BasicOrProUser', BasicUser, ProUser) def new_user(name: str, user_factory: Type[BasicOrProuser]) -> BasicOrProUser: return user_factory(name) Or must the Type[...] item have been defined with a TypeVar(..., bound=...), in which case multiple types aren't allowed with Type[...]? On 14 May 2016 at 11:30, Guido van Rossum wrote: > What would Type[C, D] mean? > > --Guido (mobile) > On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" < > python-dev at python.org> wrote: > >> Is Type[C,D] allowed? Or should multiple types be restricted to TypeVar? >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Sat May 14 16:28:46 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat, 14 May 2016 13:28:46 -0700 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: References: Message-ID: On Sat, May 14, 2016 at 11:47 AM, Peter Ludemann wrote: > I think that Type[C,D] would mean Union[Type[C], Type[D]], but I'm not > sure ... I should probably talk to a typing expert about this. > (Sorry for thinking out loud; but if we decide that Type[C,D] doesn't make > sense, we need to prohibit it) > It sounds iffy enough to prohibit. > I suppose I'm asking: do we allow new_user(Type[BasicUser, ProUser])? > That looks like a call, but Type[] is something to use in the definition. > Here's a trivial example that off the top of my head makes sense: > > BasicOrProUser = TypeVar('BasicOrProUser', BasicUser, ProUser) > def new_user(name: str, user_factory: Type[BasicOrProuser]) -> > BasicOrProUser: > return user_factory(name) > That looks reasonable and only has one parameter to Type[]. > Or must the Type[...] item have been defined with a TypeVar(..., > bound=...), in which case multiple types aren't allowed with Type[...]? > You can use whatever you want. If you want to write Type[int] go right ahead. Even Type[Union[int, str]] should be allowed. There is *usually* a good reason to use a type variable though -- without one the return type would be the base class. (But then again maybe that could be what you want, or you might not even have a return type, e.g. when the thing just gets thrown into the database.) --Guido > > > On 14 May 2016 at 11:30, Guido van Rossum wrote: > >> What would Type[C, D] mean? >> >> --Guido (mobile) >> On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" < >> python-dev at python.org> wrote: >> >>> Is Type[C,D] allowed? Or should multiple types be restricted to TypeVar? >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>> >>> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Sun May 15 03:23:15 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sun, 15 May 2016 09:23:15 +0200 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: <56A90B97.7090001@gmail.com> Message-ID: 2016-02-02 10:28 GMT+01:00 Victor Stinner : > 2016-01-27 19:25 GMT+01:00 Yury Selivanov : > > tl;dr The summary is that I have a patch that improves CPython > performance > > up to 5-10% on macro benchmarks. Benchmarks results on Macbook Pro/Mac > OS > > X, desktop CPU/Linux, server CPU/Linux are available at [1]. There are > no > > slowdowns that I could reproduce consistently. > > That's really impressive, great job Yury :-) Getting non-negligible > speedup on large macrobenchmarks became really hard in CPython. > CPython is already well optimized in all corners. It's long time since I took a look at CPython (3.2), but if it didn't changed a lot then there might be some corner cases still waiting to be optimized. ;-) Just one thing that comes to my mind: is the stack depth calculation routine changed? It was suboptimal, and calculating a better number decreases stack allocation, and increases the frame usage. > It looks like the > overall Python performance still depends heavily on the performance of > dictionary and attribute lookups. Even if it was well known, I didn't > expect up to 10% speedup on *macro* benchmarks. > True, but it might be mitigated in some ways, at least for built-in types. There are ideas about that, but they are a bit complicated to implement. The problem is with functions like len, which IMO should become attribute lookups ('foo'.len) or method executions ('foo'.len()). Then it'll be easier to accelerate their execution, with one of the above ideas. However such kind of changes belong to Guido, which defines the language structure/philosophy. IMO something like len should be part of the attributes exposed by an object: it's more "object-oriented". Whereas other things like open, file, sum, etc., are "general facilities". Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Sun May 15 07:57:39 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 15 May 2016 14:57:39 +0300 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <573601EB.50904@mail.de> Message-ID: On Fri, May 13, 2016 at 7:43 PM, Chris Angelico wrote: [...] > "Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x) > is str"). The question is, which one SHOULD be being done? Indeed it should do "Check", so that path libraries that do inherit from str will still work (see also below). > What should this do: > > class TmpPath(str): > def __fspath__(self): > return "/tmp/"+self > x = TmpPath("foo/bar") > open(x, "w") > > Does that respect __fspath__, or respect the fact that it's a string? It should not respect __fspath__ because that would mean the behavior is different in older versions of Python(!). This also means that such code should never be written (except for that example you wrote ;-). So indeed, composition would be the way to go. So yes, I still think str subclasses should not implement __fspath__, and that, at the very least, the docs should say that :). -- Koos > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com From cesare.di.mauro at gmail.com Sun May 15 08:12:46 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sun, 15 May 2016 14:12:46 +0200 Subject: [Python-Dev] Wordcode v2 In-Reply-To: References: Message-ID: 2016-02-15 1:20 GMT+01:00 Demur Rumed : > Saw recent discussion: > https://mail.python.org/pipermail/python-dev/2016-February/143013.html > > I remember trying WPython; it was fast. Unfortunately it feels it came at > the wrong time when development was invested in getting py3k out the door. > Not only that. IMO the primary problem was related to the fact the "patch" was too big to be reviewed. Unfortunately it was my first attempt, and having worked alone I introduced too much optimizations and (heavy) changes to the code. An incremental approach should have worked better, albeit I believe that such drastic move from the consolidated bytecodes to the new wordcodes would have produced strong resistance anyway. > It also had a lot of other ideas like *_INT instructions which allowed > having oparg to be a constant int rather than needing to LOAD_CONST one. > This, specifically, was an experiment that I made with WPython 1.1, which I recommend to do not follow. There are other, more general, ways to speedup the execution when dealing with integers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Sun May 15 08:27:52 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sun, 15 May 2016 14:27:52 +0200 Subject: [Python-Dev] Wordcode v2 In-Reply-To: References: Message-ID: 2016-02-15 8:14 GMT+01:00 Andrew Barnert via Python-Dev < python-dev at python.org>: > Despite the name (and inspiration), my fork has very little to do with > WPython. I'm just focused on simpler (hopefully = faster) fetch code; he > started with that, but ended up going the exact opposite direction, > accepting more complicated (and much slower) fetch code as a reasonable > cost for drastically reducing the number of instructions. (If you double > the 30% fetch-and-parse overhead per instruction, but cut the number of > instructions to 40%, the net is a huge win.) I don't know why you consider slower the WPython's code that fetches the more complicated instructions. On the contrary, I've structured such "superinstructions" in order to simplify their decoding. Arguments are decoded as they are needed in a specific moment, in order to reduce or completely avoid the usage of temporary variables to keep such values. Can you provide some example about your claim? Regarding the WPython goal, it wasn't only about introducing simpler instructions. As I've written also in my presentation, it's an hybrid VM: stack and register-based. I've introduced a new instruction format for the existing CPython's instructions, which are now easier to fetch, decode, execute, and provide a better density too (for the most common case: arguments with a maximum of 255 as value/index). However I've also added superinstructions to better pack more "useful work", which provides more code density and they are the primary responsible for improving the execution speed. Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Sun May 15 08:32:02 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sun, 15 May 2016 14:32:02 +0200 Subject: [Python-Dev] Wordcode v2 In-Reply-To: References: Message-ID: 2016-02-17 12:04 GMT+01:00 Antoine Pitrou : > Demur Rumed gmail.com> writes: > > I've personally benchmarked this fork with positive results. > > I'm skeptical of claims like this. What did you benchmark exactly, and with > which results? > I don't think changing the opcode encoding per se will bring any large > benefit... > > Regards > > Antoine. With WPython I've introduced several optimizations which improved a lot the execution speed (+25% with PyStone, at the time, compared to CPython 2.6), but most of the benefits came from the new opcode format. Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Sun May 15 09:10:40 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sun, 15 May 2016 15:10:40 +0200 Subject: [Python-Dev] Wordcode: new regular bytecode using 16-bit units In-Reply-To: References: Message-ID: 2016-04-13 18:24 GMT+02:00 Victor Stinner : > Demur Rumed proposes a different change to use a regular bytecode > using 16-bit units: an instruction has always one 8-bit argument, it's > zero if the instruction doesn't have an argument: > > http://bugs.python.org/issue26647 > > According to benchmarks, it looks faster: > > http://bugs.python.org/issue26647#msg263339 > > IMHO it's a nice enhancement: it makes the code simpler. The most > interesting change is made in Python/ceval.c: > > - if (HAS_ARG(opcode)) > - oparg = NEXTARG(); > + oparg = NEXTARG(); > > This code is the very hot loop evaluating Python bytecode. I expect > that removing a conditional branch here can reduce the CPU branch > misprediction. > Correct. The old bytecode format wasn't so much predictable for the CPU. > > Right now, ceval.c still fetchs opcode and then oparg with two 8-bit > instructions. Later, we can discuss if it would be possible to ensure > that the bytecode is always aligned to 16-bit in memory to fetch the > two bytes using a uint16_t* pointer. > > Maybe we can overallocate 1 byte in codeobject.c and align manually > the memory block if needed. Or ceval.c should maybe copy the code if > it's not aligned? > > Raymond Hettinger proposes something like that, but it looks like > there are concerns about non-aligned memory accesses: > > http://bugs.python.org/issue25823 > > The cost of non-aligned memory accesses depends on the CPU > architecture, but it can raise a SIGBUS on some arch (MIPS and > SPARC?). > > Victor > It should not be a problem, since every PyObject is allocated with PyAlloc (however I don't remember if it's the correct name) which AFAIK guarantees a base 8 bytes alignment. So, it's safe to use an unsigned int for keeping/referencing a word at the time. The only problem with such approach is related to the processor endianess, but it can be solved with proper macros (like I did with WPython). Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Sun May 15 09:25:27 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sun, 15 May 2016 15:25:27 +0200 Subject: [Python-Dev] Wordcode: new regular bytecode using 16-bit units In-Reply-To: References: <030201d195c7$ca9de130$5fd9a390$@gmail.com> Message-ID: 2016-04-13 23:23 GMT+02:00 Victor Stinner : > Hopefully, I don't expect 32-bit parameters in the wild, only 24-bit > parameter for function with annotation. > I never found 32-bit parameters, and not even 24-bit ones. I think that their usage is as rare as all planets alignment. ;-) That's why with in WPython I supported only 8, 16, and 32-bit parameters (which are 6 bytes long). Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Sun May 15 10:00:09 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sun, 15 May 2016 16:00:09 +0200 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <56AF8DB9.30900@gmail.com> References: <56A90B97.7090001@gmail.com> <20160130042835.GJ4619@ando.pearwood.info> <56AF8DB9.30900@gmail.com> Message-ID: 2016-02-01 17:54 GMT+01:00 Yury Selivanov : > Thanks for bringing this up! > > IIRC wpython was about using "fat" bytecodes, i.e. using 64bits per > bytecode instead of 8. No, it used 16, 32, and 48-bit per opcode (1, 2, or 3 16-bit words). > That allows to minimize the number of bytecodes, thus having some > performance increase. TBH, I don't think it was "significantly faster". > Please, take a look at the benchmarks, or compile it and check yourself. ;-) If I were to do some big refactoring of the ceval loop, I'd probably > consider implementing a register VM. While register VMs are a bit faster > than stack VMs (up to 20-30%), they would also allow us to apply more > optimizations, and even bolt on a simple JIT compiler. > > Yury WPython was an hybrid-VM: it supported both a stack-based and a register-based approach. I think that it's needed, since the nature of Python, because you can have operations with intermixed operands: constants, locals, globals, names. It's quite difficult to handle all possible cases with a register-based VM. Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun May 15 13:21:13 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 16 May 2016 02:21:13 +0900 Subject: [Python-Dev] file system path protocol PEP In-Reply-To: <5736793F.5030401@canterbury.ac.nz> References: <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us> <57358915.7040606@mail.de> <57359D69.6040108@mail.de> <5735FB79.1010201@mail.de> <5736076F.6010905@stoneleaf.us> <5736793F.5030401@canterbury.ac.nz> Message-ID: <22328.45065.716113.338675@turnbull.sk.tsukuba.ac.jp> Greg Ewing writes: > Paul Moore wrote: > > On 13 May 2016 at 17:57, Ethan Furman wrote: > > > >>1) What is a wallet garden? > > > > I assumed he meant "walled garden" > > Works either way -- you'd want a wall around your wallet > garden to stop people stealing your wallets. Actually, I expect it's a wall intended to keep your clients' wallets in a garden where you can harvest them at your convenience. From pludemann at google.com Sun May 15 13:24:08 2016 From: pludemann at google.com (Peter Ludemann) Date: Sun, 15 May 2016 10:24:08 -0700 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: References: Message-ID: On 14 May 2016 at 13:28, Guido van Rossum wrote: > On Sat, May 14, 2016 at 11:47 AM, Peter Ludemann > wrote: > >> I think that Type[C,D] would mean Union[Type[C], Type[D]], but I'm not >> sure ... I should probably talk to a typing expert about this. >> (Sorry for thinking out loud; but if we decide that Type[C,D] doesn't >> make sense, we need to prohibit it) >> > > It sounds iffy enough to prohibit. > > >> I suppose I'm asking: do we allow new_user(Type[BasicUser, ProUser])? >> > > That looks like a call, but Type[] is something to use in the definition. > > >> Here's a trivial example that off the top of my head makes sense: >> >> BasicOrProUser = TypeVar('BasicOrProUser', BasicUser, ProUser) >> def new_user(name: str, user_factory: Type[BasicOrProuser]) -> >> BasicOrProUser: >> return user_factory(name) >> > > That looks reasonable and only has one parameter to Type[]. > So, should we allow this (by analogy to TypeVar allowing multiple types): ? ? def new_user(name: str, user_factory: Type[BasicUser, ProUser]) -> Union[BasicUser, ProUser]: ? ? return user_factory(name) ? ?... ? ?or should it be: ? def new_user(name: str, ? ? user_factory: Type[ ? Union[ BasicUser, ProUser ?]? ]) -> Union[BasicUser, ProUser]: return user_factory(name) ? ? Although both of these lose the information that the output type is related to the input type; and hopefully the verbosity would encourage people to use TypeVar. (From these examples, my preference is for Type to be limited to a single choice and anything more complicated should be done with TypeVar, Union, etc.) > > >> Or must the Type[...] item have been defined with a TypeVar(..., >> bound=...), in which case multiple types aren't allowed with Type[...]? >> > > You can use whatever you want. If you want to write Type[int] go right > ahead. Even Type[Union[int, str]] should be allowed. There is *usually* a > good reason to use a type variable though -- without one the return type > would be the base class. (But then again maybe that could be what you want, > or you might not even have a return type, e.g. when the thing just gets > thrown into the database.) > > --Guido > > >> >> >> On 14 May 2016 at 11:30, Guido van Rossum wrote: >> >>> What would Type[C, D] mean? >>> >>> --Guido (mobile) >>> On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" < >>> python-dev at python.org> wrote: >>> >>>> Is Type[C,D] allowed? Or should multiple types be restricted to >>>> TypeVar? >>>> >>>> _______________________________________________ >>>> Python-Dev mailing list >>>> Python-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/python-dev >>>> Unsubscribe: >>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>>> >>>> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Sun May 15 13:30:05 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun, 15 May 2016 10:30:05 -0700 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: References: Message-ID: Right. I still have no good intuition for what Type[BasicUser, ProUser] would mean so I think you should be required to use the Union, which is clear. Type[] should only allow one parameter. Most real uses of Type[] use a type variable anyway. I really recommend that you read through https://github.com/python/typing/issues/107 even though it's long and meanders a bit; it comes to the right conclusion in the end. On Sun, May 15, 2016 at 10:24 AM, Peter Ludemann wrote: > > > On 14 May 2016 at 13:28, Guido van Rossum wrote: > >> On Sat, May 14, 2016 at 11:47 AM, Peter Ludemann >> wrote: >> >>> I think that Type[C,D] would mean Union[Type[C], Type[D]], but I'm not >>> sure ... I should probably talk to a typing expert about this. >>> (Sorry for thinking out loud; but if we decide that Type[C,D] doesn't >>> make sense, we need to prohibit it) >>> >> >> It sounds iffy enough to prohibit. >> >> >>> I suppose I'm asking: do we allow new_user(Type[BasicUser, ProUser])? >>> >> >> That looks like a call, but Type[] is something to use in the definition. >> >> >>> Here's a trivial example that off the top of my head makes sense: >>> >>> BasicOrProUser = TypeVar('BasicOrProUser', BasicUser, ProUser) >>> def new_user(name: str, user_factory: Type[BasicOrProuser]) -> >>> BasicOrProUser: >>> return user_factory(name) >>> >> >> That looks reasonable and only has one parameter to Type[]. >> > > So, should we allow this (by analogy to TypeVar allowing multiple types): > > ? ? > def new_user(name: str, user_factory: Type[BasicUser, ProUser]) -> > Union[BasicUser, ProUser]: > > ? ? > return user_factory(name) > > ? > ?... ? > ?or should it be: > > ? > def new_user(name: str, > ? ? > user_factory: Type[ > ? > Union[ > BasicUser, ProUser > ?]? > ]) -> Union[BasicUser, ProUser]: > return user_factory(name) > ? > ? > > Although both of these lose the information that the output type is > related to the input type; and hopefully the verbosity would encourage > people to use TypeVar. > > (From these examples, my preference is for Type to be limited to a single > choice and anything more complicated should be done with TypeVar, Union, > etc.) > > > >> >> >>> Or must the Type[...] item have been defined with a TypeVar(..., >>> bound=...), in which case multiple types aren't allowed with Type[...]? >>> >> >> You can use whatever you want. If you want to write Type[int] go right >> ahead. Even Type[Union[int, str]] should be allowed. There is *usually* a >> good reason to use a type variable though -- without one the return type >> would be the base class. (But then again maybe that could be what you want, >> or you might not even have a return type, e.g. when the thing just gets >> thrown into the database.) >> >> --Guido >> >> >>> >>> >>> On 14 May 2016 at 11:30, Guido van Rossum wrote: >>> >>>> What would Type[C, D] mean? >>>> >>>> --Guido (mobile) >>>> On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" < >>>> python-dev at python.org> wrote: >>>> >>>>> Is Type[C,D] allowed? Or should multiple types be restricted to >>>>> TypeVar? >>>>> >>>>> _______________________________________________ >>>>> Python-Dev mailing list >>>>> Python-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/python-dev >>>>> Unsubscribe: >>>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>>>> >>>>> >>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Mon May 16 00:48:50 2016 From: nas at arctrix.com (Neil Schemenauer) Date: Mon, 16 May 2016 04:48:50 +0000 (UTC) Subject: [Python-Dev] PyGC_Collect ignores state of `enabled` References: Message-ID: Hi, I intended for gc.collect() to actually collect cycles even if the auto-GC was disabled. Having Py_Finalize() run GC even when it has been disabled seems wrong to me. Originally, cyclic GC was supposed to be optional. Back then, most programs did not leak cycles. I would vote for Py_Finalize() checking the 'enabled' flag and not calling PyGC_Collect if false. Regards, Neil From meadori at gmail.com Mon May 16 11:55:18 2016 From: meadori at gmail.com (Meador Inge) Date: Mon, 16 May 2016 10:55:18 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: <56A90B97.7090001@gmail.com> Message-ID: On Sun, May 15, 2016 at 2:23 AM, Cesare Di Mauro wrote: > Just one thing that comes to my mind: is the stack depth calculation > routine changed? It was suboptimal, and calculating a better number > decreases stack allocation, and increases the frame usage. > This is still a problem and came up again recently: http://bugs.python.org/issue26549 -- Meador -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 16 16:00:58 2016 From: brett at python.org (Brett Cannon) Date: Mon, 16 May 2016 20:00:58 +0000 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: Recent discussions have been about type hints which are orthogonal to the PEP, so things have seemed to have reached a steady state. Was there anything else that needed clarification, Guido, or are you ready to pronounce? Or did you want to wait until the language summit? Or did you want to assign a BDFL delegate? On Fri, 13 May 2016 at 11:37 Brett Cannon wrote: > Biggest changes since the second draft: > > 1. Resolve __fspath__() from the type, not the instance (for Guido) > 2. Updated the TypeError messages to say "os.PathLike object" instead > of "path object" (implicitly for Steven) > 3. TODO item to define "path-like" in the glossary (for Steven) > 4. Various more things added to Rejected Ideas > 5. Added Koos as a co-author (for Koos :) > > ---------- > PEP: NNN > Title: Adding a file system path protocol > Version: $Revision$ > Last-Modified: $Date$ > Author: Brett Cannon , > Koos Zevenhoven > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 11-May-2016 > Post-History: 11-May-2016, > 12-May-2016, > 13-May-2016 > > > Abstract > ======== > > This PEP proposes a protocol for classes which represent a file system > path to be able to provide a ``str`` or ``bytes`` representation. > Changes to Python's standard library are also proposed to utilize this > protocol where appropriate to facilitate the use of path objects where > historically only ``str`` and/or ``bytes`` file system paths are > accepted. The goal is to facilitate the migration of users towards > rich path objects while providing an easy way to work with code > expecting ``str`` or ``bytes``. > > > Rationale > ========= > > Historically in Python, file system paths have been represented as > strings or bytes. This choice of representation has stemmed from C's > own decision to represent file system paths as > ``const char *`` [#libc-open]_. While that is a totally serviceable > format to use for file system paths, it's not necessarily optimal. At > issue is the fact that while all file system paths can be represented > as strings or bytes, not all strings or bytes represent a file system > path. This can lead to issues where any e.g. string duck-types to a > file system path whether it actually represents a path or not. > > To help elevate the representation of file system paths from their > representation as strings and bytes to a richer object representation, > the pathlib module [#pathlib]_ was provisionally introduced in > Python 3.4 through PEP 428. While considered by some as an improvement > over strings and bytes for file system paths, it has suffered from a > lack of adoption. Typically the key issue listed for the low adoption > rate has been the lack of support in the standard library. This lack > of support required users of pathlib to manually convert path objects > to strings by calling ``str(path)`` which many found error-prone. > > One issue in converting path objects to strings comes from > the fact that the only generic way to get a string representation of > the path was to pass the object to ``str()``. This can pose a > problem when done blindly as nearly all Python objects have some > string representation whether they are a path or not, e.g. > ``str(None)`` will give a result that > ``builtins.open()`` [#builtins-open]_ will happily use to create a new > file. > > Exacerbating this whole situation is the > ``DirEntry`` object [#os-direntry]_. While path objects have a > representation that can be extracted using ``str()``, ``DirEntry`` > objects expose a ``path`` attribute instead. Having no common > interface between path objects, ``DirEntry``, and any other > third-party path library has become an issue. A solution that allows > any path-representing object to declare that it is a path and a way > to extract a low-level representation that all path objects could > support is desired. > > This PEP then proposes to introduce a new protocol to be followed by > objects which represent file system paths. Providing a protocol allows > for explicit signaling of what objects represent file system paths as > well as a way to extract a lower-level representation that can be used > with older APIs which only support strings or bytes. > > Discussions regarding path objects that led to this PEP can be found > in multiple threads on the python-ideas mailing list archive > [#python-ideas-archive]_ for the months of March and April 2016 and on > the python-dev mailing list archives [#python-dev-archive]_ during > April 2016. > > > Proposal > ======== > > This proposal is split into two parts. One part is the proposal of a > protocol for objects to declare and provide support for exposing a > file system path representation. The other part deals with changes to > Python's standard library to support the new protocol. These changes > will also lead to the pathlib module dropping its provisional status. > > Protocol > -------- > > The following abstract base class defines the protocol for an object > to be considered a path object:: > > import abc > import typing as t > > > class PathLike(abc.ABC): > > """Abstract base class for implementing the file system path > protocol.""" > > @abc.abstractmethod > def __fspath__(self) -> t.Union[str, bytes]: > """Return the file system path representation of the object.""" > raise NotImplementedError > > > Objects representing file system paths will implement the > ``__fspath__()`` method which will return the ``str`` or ``bytes`` > representation of the path. The ``str`` representation is the > preferred low-level path representation as it is human-readable and > what people historically represent paths as. > > > Standard library changes > ------------------------ > > It is expected that most APIs in Python's standard library that > currently accept a file system path will be updated appropriately to > accept path objects (whether that requires code or simply an update > to documentation will vary). The modules mentioned below, though, > deserve specific details as they have either fundamental changes that > empower the ability to use path objects, or entail additions/removal > of APIs. > > > builtins > '''''''' > > ``open()`` [#builtins-open]_ will be updated to accept path objects as > well as continue to accept ``str`` and ``bytes``. > > > os > ''' > > The ``fspath()`` function will be added with the following semantics:: > > import typing as t > > > def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]: > """Return the string representation of the path. > > If str or bytes is passed in, it is returned unchanged. > """ > if isinstance(path, (str, bytes)): > return path > > # Work from the object's type to match method resolution of other > magic > # methods. > path_type = type(path) > try: > return path_type.__fspath__(path) > except AttributeError: > if hasattr(path_type, '__fspath__'): > raise > > raise TypeError("expected str, bytes or os.PathLike object, > not " > + path_type.__name__) > > The ``os.fsencode()`` [#os-fsencode]_ and > ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept > path objects. As both functions coerce their arguments to > ``bytes`` and ``str``, respectively, they will be updated to call > ``__fspath__()`` if present to convert the path object to a ``str`` or > ``bytes`` representation, and then perform their appropriate > coercion operations as if the return value from ``__fspath__()`` had > been the original argument to the coercion function in question. > > The addition of ``os.fspath()``, the updates to > ``os.fsencode()``/``os.fsdecode()``, and the current semantics of > ``pathlib.PurePath`` provide the semantics necessary to > get the path representation one prefers. For a path object, > ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or > ``bytes`` representation without any coersion, then ``os.fspath()`` > can be used. If a ``str`` is desired and the encoding of ``bytes`` > should be assumed to be the default file system encoding, then > ``os.fsdecode()`` should be used. If a ``bytes`` representation is > desired and any strings should be encoded using the default file > system encoding, then ``os.fsencode()`` is used. This PEP recommends > using path objects when possible and falling back to string paths as > necessary and using ``bytes`` as a last resort. > > Another way to view this is as a hierarchy of file system path > representations (highest- to lowest-level): path ? str ? bytes. The > functions and classes under discussion can all accept objects on the > same level of the hierarchy, but they vary in whether they promote or > demote objects to another level. The ``pathlib.PurePath`` class can > promote a ``str`` to a path object. The ``os.fspath()`` function can > demote a path object to a ``str`` or ``bytes`` instance, depending > on what ``__fspath__()`` returns. > The ``os.fsdecode()`` function will demote a path object to > a string or promote a ``bytes`` object to a ``str``. The > ``os.fsencode()`` function will demote a path or string object to > ``bytes``. There is no function that provides a way to demote a path > object directly to ``bytes`` while bypassing string demotion. > > The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` > method. It will return the same value as currently found on the > ``path`` attribute of ``DirEntry`` instances. > > The Protocol_ ABC will be added to the ``os`` module under the name > ``os.PathLike``. > > > os.path > ''''''' > > The various path-manipulation functions of ``os.path`` [#os-path]_ > will be updated to accept path objects. For polymorphic functions that > accept both bytes and strings, they will be updated to simply use > ``os.fspath()``. > > During the discussions leading up to this PEP it was suggested that > ``os.path`` not be updated using an "explicit is better than implicit" > argument. The thinking was that since ``__fspath__()`` is polymorphic > itself it may be better to have code working with ``os.path`` extract > the path representation from path objects explicitly. There is also > the consideration that adding support this deep into the low-level OS > APIs will lead to code magically supporting path objects without > requiring any documentation updated, leading to potential complaints > when it doesn't work, unbeknownst to the project author. > > But it is the view of this PEP that "practicality beats purity" in > this instance. To help facilitate the transition to supporting path > objects, it is better to make the transition as easy as possible than > to worry about unexpected/undocumented duck typing support for > path objects by projects. > > There has also been the suggestion that ``os.path`` functions could be > used in a tight loop and the overhead of checking or calling > ``__fspath__()`` would be too costly. In this scenario only > path-consuming APIs would be directly updated and path-manipulating > APIs like the ones in ``os.path`` would go unmodified. This would > require library authors to update their code to support path objects > if they performed any path manipulations, but if the library code > passed the path straight through then the library wouldn't need to be > updated. It is the view of this PEP and Guido, though, that this is an > unnecessary worry and that performance will still be acceptable. > > > pathlib > ''''''' > > The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be > updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path`` > will continue to not accept ``bytes`` path representations, and so if > ``__fspath__()`` returns ``bytes`` it will raise an exception. > > The ``path`` attribute will be removed as this PEP makes it > redundant (it has not been included in any released version of Python > and so is not a backwards-compatibility concern). > > > C API > ''''' > > The C API will gain an equivalent function to ``os.fspath()``:: > > /* > Return the file system path of the object. > > If the object is str or bytes, then allow it to pass through with > an incremented refcount. If the object defines __fspath__(), then > return the result of that method. All other types raise a > TypeError. > */ > PyObject * > PyOS_FSPath(PyObject *path) > { > if (PyUnicode_Check(path) || PyBytes_Check(path)) { > Py_INCREF(path); > return path; > } > > if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { > return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__", > path, > NULL); > } > > return PyErr_Format(PyExc_TypeError, > "expected a str, bytes, or os.PathLike object, > not %S", > path->ob_type); > } > > > > Backwards compatibility > ======================= > > There are no explicit backwards-compatibility concerns. Unless an > object incidentally already defines a ``__fspath__()`` method there is > no reason to expect the pre-existing code to break or expect to have > its semantics implicitly changed. > > Libraries wishing to support path objects and a version of Python > prior to Python 3.6 and the existence of ``os.fspath()`` can use the > idiom of > ``path.__fspath__() if hasattr(path, "__fspath__") else path``. > > > Implementation > ============== > > This is the task list for what this PEP proposes: > > #. Remove the ``path`` attribute from pathlib > #. Remove the provisional status of pathlib > #. Add ``os.PathLike`` > #. Add ``os.fspath()`` > #. Add ``PyOS_FSPath()`` > #. Update ``os.fsencode()`` > #. Update ``os.fsdecode()`` > #. Update ``pathlib.PurePath`` and ``pathlib.Path`` > #. Update ``builtins.open()`` > #. Update ``os.DirEntry`` > #. Update ``os.path`` > #. Add a glossary entry for "path-like" > > > Rejected Ideas > ============== > > Other names for the protocol's method > ------------------------------------- > > Various names were proposed during discussions leading to this PEP, > including ``__path__``, ``__pathname__``, and ``__fspathname__``. In > the end people seemed to gravitate towards ``__fspath__`` for being > unambiguous without being unnecessarily long. > > > Separate str/bytes methods > -------------------------- > > At one point it was suggested that ``__fspath__()`` only return > strings and another method named ``__fspathb__()`` be introduced to > return bytes. The thinking is that by making ``__fspath__()`` not be > polymorphic it could make dealing with the potential string or bytes > representations easier. But the general consensus was that returning > bytes will more than likely be rare and that the various functions in > the os module are the better abstraction to promote over direct > calls to ``__fspath__()``. > > > Providing a ``path`` attribute > ------------------------------ > > To help deal with the issue of ``pathlib.PurePath`` not inheriting > from ``str``, originally it was proposed to introduce a ``path`` > attribute to mirror what ``os.DirEntry`` provides. In the end, > though, it was determined that a protocol would provide the same > result while not directly exposing an API that most people will never > need to interact with directly. > > > Have ``__fspath__()`` only return strings > ------------------------------------------ > > Much of the discussion that led to this PEP revolved around whether > ``__fspath__()`` should be polymorphic and return ``bytes`` as well as > ``str`` or only return ``str``. The general sentiment for this view > was that ``bytes`` are difficult to work with due to their > inherent lack of information about their encoding and PEP 383 makes > it possible to represent all file system paths using ``str`` with the > ``surrogateescape`` handler. Thus, it would be better to forcibly > promote the use of ``str`` as the low-level path representation for > high-level path objects. > > In the end, it was decided that using ``bytes`` to represent paths is > simply not going to go away and thus they should be supported to some > degree. The hope is that people will gravitate towards path objects > like pathlib and that will move people away from operating directly > with ``bytes``. > > > A generic string encoding mechanism > ----------------------------------- > > At one point there was a discussion of developing a generic mechanism > to extract a string representation of an object that had semantic > meaning (``__str__()`` does not necessarily return anything of > semantic significance beyond what may be helpful for debugging). In > the end, it was deemed to lack a motivating need beyond the one this > PEP is trying to solve in a specific fashion. > > > Have __fspath__ be an attribute > ------------------------------- > > It was briefly considered to have ``__fspath__`` be an attribute > instead of a method. This was rejected for two reasons. One, > historically protocols have been implemented as "magic methods" and > not "magic methods and attributes". Two, there is no guarantee that > the lower-level representation of a path object will be pre-computed, > potentially misleading users that there was no expensive computation > behind the scenes in case the attribute was implemented as a property. > > This also indirectly ties into the idea of introducing a ``path`` > attribute to accomplish the same thing. This idea has an added issue, > though, of accidentally having any object with a ``path`` attribute > meet the protocol's duck typing. Introducing a new magic method for > the protocol helpfully avoids any accidental opting into the protocol. > > > Provide specific type hinting support > ------------------------------------- > > There was some consideration to provdinga generic ``typing.PathLike`` > class which would allow for e.g. ``typing.PathLike[str]`` to specify > a type hint for a path object which returned a string representation. > While potentially beneficial, the usefulness was deemed too small to > bother adding the type hint class. > > This also removed any desire to have a class in the ``typing`` module > which represented the union of all acceptable path-representing types > as that can be represented with > ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope > is users will slowly gravitate to path objects only. > > > Provide ``os.fspathb()`` > ------------------------ > > It was suggested that to mirror the structure of e.g. > ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return > ``str`` and that another function named ``os.fspathb()`` be > introduced that only returned ``bytes``. This was rejected as the > purposes of the ``*b()`` functions are tied to querying the file > system where there is a need to get the raw bytes back. As this PEP > does not work directly with data on a file system (but which *may* > be), the view was taken this distinction is unnecessary. It's also > believed that the need for only bytes will not be common enough to > need to support in such a specific manner as ``os.fsencode()`` will > provide similar functionality. > > > Call ``__fspath__()`` off of the instance > ----------------------------------------- > > An earlier draft of this PEP had ``os.fspath()`` calling > ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The > changed to be consistent with how other magic methods in Python are > resolved. > > > Acknowledgements > ================ > > Thanks to everyone who participated in the various discussions related > to this PEP that spanned both python-ideas and python-dev. Special > thanks to Stephen Turnbull for direct feedback on early drafts of this > PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not > only feedback on early drafts of this PEP but also helping to drive > the overall discussion on this topic across the two mailing lists. > > > References > ========== > > .. [#python-ideas-archive] The python-ideas mailing list archive > (https://mail.python.org/pipermail/python-ideas/) > > .. [#python-dev-archive] The python-dev mailing list archive > (https://mail.python.org/pipermail/python-dev/) > > .. [#libc-open] ``open()`` documention for the C standard library > ( > http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html > ) > > .. [#pathlib] The ``pathlib`` module > (https://docs.python.org/3/library/pathlib.html#module-pathlib) > > .. [#builtins-open] The ``builtins.open()`` function > (https://docs.python.org/3/library/functions.html#open) > > .. [#os-fsencode] The ``os.fsencode()`` function > (https://docs.python.org/3/library/os.html#os.fsencode) > > .. [#os-fsdecode] The ``os.fsdecode()`` function > (https://docs.python.org/3/library/os.html#os.fsdecode) > > .. [#os-direntry] The ``os.DirEntry`` class > (https://docs.python.org/3/library/os.html#os.DirEntry) > > .. [#os-path] The ``os.path`` module > (https://docs.python.org/3/library/os.path.html#module-os.path) > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 16 16:11:48 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 16 May 2016 13:11:48 -0700 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: Once you assign yourself a PEP number I'll do one more pass and then I expect to accept it -- the draft looks good to me! On Mon, May 16, 2016 at 1:00 PM, Brett Cannon wrote: > Recent discussions have been about type hints which are orthogonal to the > PEP, so things have seemed to have reached a steady state. > > Was there anything else that needed clarification, Guido, or are you ready > to pronounce? Or did you want to wait until the language summit? Or did you > want to assign a BDFL delegate? > > > On Fri, 13 May 2016 at 11:37 Brett Cannon wrote: > >> Biggest changes since the second draft: >> >> 1. Resolve __fspath__() from the type, not the instance (for Guido) >> 2. Updated the TypeError messages to say "os.PathLike object" instead >> of "path object" (implicitly for Steven) >> 3. TODO item to define "path-like" in the glossary (for Steven) >> 4. Various more things added to Rejected Ideas >> 5. Added Koos as a co-author (for Koos :) >> >> ---------- >> PEP: NNN >> Title: Adding a file system path protocol >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Brett Cannon , >> Koos Zevenhoven >> Status: Draft >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 11-May-2016 >> Post-History: 11-May-2016, >> 12-May-2016, >> 13-May-2016 >> >> >> Abstract >> ======== >> >> This PEP proposes a protocol for classes which represent a file system >> path to be able to provide a ``str`` or ``bytes`` representation. >> Changes to Python's standard library are also proposed to utilize this >> protocol where appropriate to facilitate the use of path objects where >> historically only ``str`` and/or ``bytes`` file system paths are >> accepted. The goal is to facilitate the migration of users towards >> rich path objects while providing an easy way to work with code >> expecting ``str`` or ``bytes``. >> >> >> Rationale >> ========= >> >> Historically in Python, file system paths have been represented as >> strings or bytes. This choice of representation has stemmed from C's >> own decision to represent file system paths as >> ``const char *`` [#libc-open]_. While that is a totally serviceable >> format to use for file system paths, it's not necessarily optimal. At >> issue is the fact that while all file system paths can be represented >> as strings or bytes, not all strings or bytes represent a file system >> path. This can lead to issues where any e.g. string duck-types to a >> file system path whether it actually represents a path or not. >> >> To help elevate the representation of file system paths from their >> representation as strings and bytes to a richer object representation, >> the pathlib module [#pathlib]_ was provisionally introduced in >> Python 3.4 through PEP 428. While considered by some as an improvement >> over strings and bytes for file system paths, it has suffered from a >> lack of adoption. Typically the key issue listed for the low adoption >> rate has been the lack of support in the standard library. This lack >> of support required users of pathlib to manually convert path objects >> to strings by calling ``str(path)`` which many found error-prone. >> >> One issue in converting path objects to strings comes from >> the fact that the only generic way to get a string representation of >> the path was to pass the object to ``str()``. This can pose a >> problem when done blindly as nearly all Python objects have some >> string representation whether they are a path or not, e.g. >> ``str(None)`` will give a result that >> ``builtins.open()`` [#builtins-open]_ will happily use to create a new >> file. >> >> Exacerbating this whole situation is the >> ``DirEntry`` object [#os-direntry]_. While path objects have a >> representation that can be extracted using ``str()``, ``DirEntry`` >> objects expose a ``path`` attribute instead. Having no common >> interface between path objects, ``DirEntry``, and any other >> third-party path library has become an issue. A solution that allows >> any path-representing object to declare that it is a path and a way >> to extract a low-level representation that all path objects could >> support is desired. >> >> This PEP then proposes to introduce a new protocol to be followed by >> objects which represent file system paths. Providing a protocol allows >> for explicit signaling of what objects represent file system paths as >> well as a way to extract a lower-level representation that can be used >> with older APIs which only support strings or bytes. >> >> Discussions regarding path objects that led to this PEP can be found >> in multiple threads on the python-ideas mailing list archive >> [#python-ideas-archive]_ for the months of March and April 2016 and on >> the python-dev mailing list archives [#python-dev-archive]_ during >> April 2016. >> >> >> Proposal >> ======== >> >> This proposal is split into two parts. One part is the proposal of a >> protocol for objects to declare and provide support for exposing a >> file system path representation. The other part deals with changes to >> Python's standard library to support the new protocol. These changes >> will also lead to the pathlib module dropping its provisional status. >> >> Protocol >> -------- >> >> The following abstract base class defines the protocol for an object >> to be considered a path object:: >> >> import abc >> import typing as t >> >> >> class PathLike(abc.ABC): >> >> """Abstract base class for implementing the file system path >> protocol.""" >> >> @abc.abstractmethod >> def __fspath__(self) -> t.Union[str, bytes]: >> """Return the file system path representation of the >> object.""" >> raise NotImplementedError >> >> >> Objects representing file system paths will implement the >> ``__fspath__()`` method which will return the ``str`` or ``bytes`` >> representation of the path. The ``str`` representation is the >> preferred low-level path representation as it is human-readable and >> what people historically represent paths as. >> >> >> Standard library changes >> ------------------------ >> >> It is expected that most APIs in Python's standard library that >> currently accept a file system path will be updated appropriately to >> accept path objects (whether that requires code or simply an update >> to documentation will vary). The modules mentioned below, though, >> deserve specific details as they have either fundamental changes that >> empower the ability to use path objects, or entail additions/removal >> of APIs. >> >> >> builtins >> '''''''' >> >> ``open()`` [#builtins-open]_ will be updated to accept path objects as >> well as continue to accept ``str`` and ``bytes``. >> >> >> os >> ''' >> >> The ``fspath()`` function will be added with the following semantics:: >> >> import typing as t >> >> >> def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, >> bytes]: >> """Return the string representation of the path. >> >> If str or bytes is passed in, it is returned unchanged. >> """ >> if isinstance(path, (str, bytes)): >> return path >> >> # Work from the object's type to match method resolution of other >> magic >> # methods. >> path_type = type(path) >> try: >> return path_type.__fspath__(path) >> except AttributeError: >> if hasattr(path_type, '__fspath__'): >> raise >> >> raise TypeError("expected str, bytes or os.PathLike object, >> not " >> + path_type.__name__) >> >> The ``os.fsencode()`` [#os-fsencode]_ and >> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept >> path objects. As both functions coerce their arguments to >> ``bytes`` and ``str``, respectively, they will be updated to call >> ``__fspath__()`` if present to convert the path object to a ``str`` or >> ``bytes`` representation, and then perform their appropriate >> coercion operations as if the return value from ``__fspath__()`` had >> been the original argument to the coercion function in question. >> >> The addition of ``os.fspath()``, the updates to >> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of >> ``pathlib.PurePath`` provide the semantics necessary to >> get the path representation one prefers. For a path object, >> ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or >> ``bytes`` representation without any coersion, then ``os.fspath()`` >> can be used. If a ``str`` is desired and the encoding of ``bytes`` >> should be assumed to be the default file system encoding, then >> ``os.fsdecode()`` should be used. If a ``bytes`` representation is >> desired and any strings should be encoded using the default file >> system encoding, then ``os.fsencode()`` is used. This PEP recommends >> using path objects when possible and falling back to string paths as >> necessary and using ``bytes`` as a last resort. >> >> Another way to view this is as a hierarchy of file system path >> representations (highest- to lowest-level): path ? str ? bytes. The >> functions and classes under discussion can all accept objects on the >> same level of the hierarchy, but they vary in whether they promote or >> demote objects to another level. The ``pathlib.PurePath`` class can >> promote a ``str`` to a path object. The ``os.fspath()`` function can >> demote a path object to a ``str`` or ``bytes`` instance, depending >> on what ``__fspath__()`` returns. >> The ``os.fsdecode()`` function will demote a path object to >> a string or promote a ``bytes`` object to a ``str``. The >> ``os.fsencode()`` function will demote a path or string object to >> ``bytes``. There is no function that provides a way to demote a path >> object directly to ``bytes`` while bypassing string demotion. >> >> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` >> method. It will return the same value as currently found on the >> ``path`` attribute of ``DirEntry`` instances. >> >> The Protocol_ ABC will be added to the ``os`` module under the name >> ``os.PathLike``. >> >> >> os.path >> ''''''' >> >> The various path-manipulation functions of ``os.path`` [#os-path]_ >> will be updated to accept path objects. For polymorphic functions that >> accept both bytes and strings, they will be updated to simply use >> ``os.fspath()``. >> >> During the discussions leading up to this PEP it was suggested that >> ``os.path`` not be updated using an "explicit is better than implicit" >> argument. The thinking was that since ``__fspath__()`` is polymorphic >> itself it may be better to have code working with ``os.path`` extract >> the path representation from path objects explicitly. There is also >> the consideration that adding support this deep into the low-level OS >> APIs will lead to code magically supporting path objects without >> requiring any documentation updated, leading to potential complaints >> when it doesn't work, unbeknownst to the project author. >> >> But it is the view of this PEP that "practicality beats purity" in >> this instance. To help facilitate the transition to supporting path >> objects, it is better to make the transition as easy as possible than >> to worry about unexpected/undocumented duck typing support for >> path objects by projects. >> >> There has also been the suggestion that ``os.path`` functions could be >> used in a tight loop and the overhead of checking or calling >> ``__fspath__()`` would be too costly. In this scenario only >> path-consuming APIs would be directly updated and path-manipulating >> APIs like the ones in ``os.path`` would go unmodified. This would >> require library authors to update their code to support path objects >> if they performed any path manipulations, but if the library code >> passed the path straight through then the library wouldn't need to be >> updated. It is the view of this PEP and Guido, though, that this is an >> unnecessary worry and that performance will still be acceptable. >> >> >> pathlib >> ''''''' >> >> The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be >> updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path`` >> will continue to not accept ``bytes`` path representations, and so if >> ``__fspath__()`` returns ``bytes`` it will raise an exception. >> >> The ``path`` attribute will be removed as this PEP makes it >> redundant (it has not been included in any released version of Python >> and so is not a backwards-compatibility concern). >> >> >> C API >> ''''' >> >> The C API will gain an equivalent function to ``os.fspath()``:: >> >> /* >> Return the file system path of the object. >> >> If the object is str or bytes, then allow it to pass through with >> an incremented refcount. If the object defines __fspath__(), then >> return the result of that method. All other types raise a >> TypeError. >> */ >> PyObject * >> PyOS_FSPath(PyObject *path) >> { >> if (PyUnicode_Check(path) || PyBytes_Check(path)) { >> Py_INCREF(path); >> return path; >> } >> >> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { >> return PyObject_CallMethodObjArgs(path->ob_type, >> "__fspath__", path, >> NULL); >> } >> >> return PyErr_Format(PyExc_TypeError, >> "expected a str, bytes, or os.PathLike >> object, not %S", >> path->ob_type); >> } >> >> >> >> Backwards compatibility >> ======================= >> >> There are no explicit backwards-compatibility concerns. Unless an >> object incidentally already defines a ``__fspath__()`` method there is >> no reason to expect the pre-existing code to break or expect to have >> its semantics implicitly changed. >> >> Libraries wishing to support path objects and a version of Python >> prior to Python 3.6 and the existence of ``os.fspath()`` can use the >> idiom of >> ``path.__fspath__() if hasattr(path, "__fspath__") else path``. >> >> >> Implementation >> ============== >> >> This is the task list for what this PEP proposes: >> >> #. Remove the ``path`` attribute from pathlib >> #. Remove the provisional status of pathlib >> #. Add ``os.PathLike`` >> #. Add ``os.fspath()`` >> #. Add ``PyOS_FSPath()`` >> #. Update ``os.fsencode()`` >> #. Update ``os.fsdecode()`` >> #. Update ``pathlib.PurePath`` and ``pathlib.Path`` >> #. Update ``builtins.open()`` >> #. Update ``os.DirEntry`` >> #. Update ``os.path`` >> #. Add a glossary entry for "path-like" >> >> >> Rejected Ideas >> ============== >> >> Other names for the protocol's method >> ------------------------------------- >> >> Various names were proposed during discussions leading to this PEP, >> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In >> the end people seemed to gravitate towards ``__fspath__`` for being >> unambiguous without being unnecessarily long. >> >> >> Separate str/bytes methods >> -------------------------- >> >> At one point it was suggested that ``__fspath__()`` only return >> strings and another method named ``__fspathb__()`` be introduced to >> return bytes. The thinking is that by making ``__fspath__()`` not be >> polymorphic it could make dealing with the potential string or bytes >> representations easier. But the general consensus was that returning >> bytes will more than likely be rare and that the various functions in >> the os module are the better abstraction to promote over direct >> calls to ``__fspath__()``. >> >> >> Providing a ``path`` attribute >> ------------------------------ >> >> To help deal with the issue of ``pathlib.PurePath`` not inheriting >> from ``str``, originally it was proposed to introduce a ``path`` >> attribute to mirror what ``os.DirEntry`` provides. In the end, >> though, it was determined that a protocol would provide the same >> result while not directly exposing an API that most people will never >> need to interact with directly. >> >> >> Have ``__fspath__()`` only return strings >> ------------------------------------------ >> >> Much of the discussion that led to this PEP revolved around whether >> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as >> ``str`` or only return ``str``. The general sentiment for this view >> was that ``bytes`` are difficult to work with due to their >> inherent lack of information about their encoding and PEP 383 makes >> it possible to represent all file system paths using ``str`` with the >> ``surrogateescape`` handler. Thus, it would be better to forcibly >> promote the use of ``str`` as the low-level path representation for >> high-level path objects. >> >> In the end, it was decided that using ``bytes`` to represent paths is >> simply not going to go away and thus they should be supported to some >> degree. The hope is that people will gravitate towards path objects >> like pathlib and that will move people away from operating directly >> with ``bytes``. >> >> >> A generic string encoding mechanism >> ----------------------------------- >> >> At one point there was a discussion of developing a generic mechanism >> to extract a string representation of an object that had semantic >> meaning (``__str__()`` does not necessarily return anything of >> semantic significance beyond what may be helpful for debugging). In >> the end, it was deemed to lack a motivating need beyond the one this >> PEP is trying to solve in a specific fashion. >> >> >> Have __fspath__ be an attribute >> ------------------------------- >> >> It was briefly considered to have ``__fspath__`` be an attribute >> instead of a method. This was rejected for two reasons. One, >> historically protocols have been implemented as "magic methods" and >> not "magic methods and attributes". Two, there is no guarantee that >> the lower-level representation of a path object will be pre-computed, >> potentially misleading users that there was no expensive computation >> behind the scenes in case the attribute was implemented as a property. >> >> This also indirectly ties into the idea of introducing a ``path`` >> attribute to accomplish the same thing. This idea has an added issue, >> though, of accidentally having any object with a ``path`` attribute >> meet the protocol's duck typing. Introducing a new magic method for >> the protocol helpfully avoids any accidental opting into the protocol. >> >> >> Provide specific type hinting support >> ------------------------------------- >> >> There was some consideration to provdinga generic ``typing.PathLike`` >> class which would allow for e.g. ``typing.PathLike[str]`` to specify >> a type hint for a path object which returned a string representation. >> While potentially beneficial, the usefulness was deemed too small to >> bother adding the type hint class. >> >> This also removed any desire to have a class in the ``typing`` module >> which represented the union of all acceptable path-representing types >> as that can be represented with >> ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope >> is users will slowly gravitate to path objects only. >> >> >> Provide ``os.fspathb()`` >> ------------------------ >> >> It was suggested that to mirror the structure of e.g. >> ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return >> ``str`` and that another function named ``os.fspathb()`` be >> introduced that only returned ``bytes``. This was rejected as the >> purposes of the ``*b()`` functions are tied to querying the file >> system where there is a need to get the raw bytes back. As this PEP >> does not work directly with data on a file system (but which *may* >> be), the view was taken this distinction is unnecessary. It's also >> believed that the need for only bytes will not be common enough to >> need to support in such a specific manner as ``os.fsencode()`` will >> provide similar functionality. >> >> >> Call ``__fspath__()`` off of the instance >> ----------------------------------------- >> >> An earlier draft of this PEP had ``os.fspath()`` calling >> ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The >> changed to be consistent with how other magic methods in Python are >> resolved. >> >> >> Acknowledgements >> ================ >> >> Thanks to everyone who participated in the various discussions related >> to this PEP that spanned both python-ideas and python-dev. Special >> thanks to Stephen Turnbull for direct feedback on early drafts of this >> PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not >> only feedback on early drafts of this PEP but also helping to drive >> the overall discussion on this topic across the two mailing lists. >> >> >> References >> ========== >> >> .. [#python-ideas-archive] The python-ideas mailing list archive >> (https://mail.python.org/pipermail/python-ideas/) >> >> .. [#python-dev-archive] The python-dev mailing list archive >> (https://mail.python.org/pipermail/python-dev/) >> >> .. [#libc-open] ``open()`` documention for the C standard library >> ( >> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html >> ) >> >> .. [#pathlib] The ``pathlib`` module >> (https://docs.python.org/3/library/pathlib.html#module-pathlib) >> >> .. [#builtins-open] The ``builtins.open()`` function >> (https://docs.python.org/3/library/functions.html#open) >> >> .. [#os-fsencode] The ``os.fsencode()`` function >> (https://docs.python.org/3/library/os.html#os.fsencode) >> >> .. [#os-fsdecode] The ``os.fsdecode()`` function >> (https://docs.python.org/3/library/os.html#os.fsdecode) >> >> .. [#os-direntry] The ``os.DirEntry`` class >> (https://docs.python.org/3/library/os.html#os.DirEntry) >> >> .. [#os-path] The ``os.path`` module >> (https://docs.python.org/3/library/os.path.html#module-os.path) >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> >> .. >> Local Variables: >> mode: indented-text >> indent-tabs-mode: nil >> sentence-end-double-space: t >> fill-column: 70 >> coding: utf-8 >> End: >> >> -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 16 16:26:25 2016 From: brett at python.org (Brett Cannon) Date: Mon, 16 May 2016 20:26:25 +0000 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: On Mon, 16 May 2016 at 13:12 Guido van Rossum wrote: > Once you assign yourself a PEP number I'll do one more pass and then I > expect to accept it -- the draft looks good to me! > Done: https://hg.python.org/peps/rev/b41cb718054a > > On Mon, May 16, 2016 at 1:00 PM, Brett Cannon wrote: > >> Recent discussions have been about type hints which are orthogonal to the >> PEP, so things have seemed to have reached a steady state. >> >> Was there anything else that needed clarification, Guido, or are you >> ready to pronounce? Or did you want to wait until the language summit? Or >> did you want to assign a BDFL delegate? >> >> >> On Fri, 13 May 2016 at 11:37 Brett Cannon wrote: >> >>> Biggest changes since the second draft: >>> >>> 1. Resolve __fspath__() from the type, not the instance (for Guido) >>> 2. Updated the TypeError messages to say "os.PathLike object" >>> instead of "path object" (implicitly for Steven) >>> 3. TODO item to define "path-like" in the glossary (for Steven) >>> 4. Various more things added to Rejected Ideas >>> 5. Added Koos as a co-author (for Koos :) >>> >>> ---------- >>> PEP: NNN >>> Title: Adding a file system path protocol >>> Version: $Revision$ >>> Last-Modified: $Date$ >>> Author: Brett Cannon , >>> Koos Zevenhoven >>> Status: Draft >>> Type: Standards Track >>> Content-Type: text/x-rst >>> Created: 11-May-2016 >>> Post-History: 11-May-2016, >>> 12-May-2016, >>> 13-May-2016 >>> >>> >>> Abstract >>> ======== >>> >>> This PEP proposes a protocol for classes which represent a file system >>> path to be able to provide a ``str`` or ``bytes`` representation. >>> Changes to Python's standard library are also proposed to utilize this >>> protocol where appropriate to facilitate the use of path objects where >>> historically only ``str`` and/or ``bytes`` file system paths are >>> accepted. The goal is to facilitate the migration of users towards >>> rich path objects while providing an easy way to work with code >>> expecting ``str`` or ``bytes``. >>> >>> >>> Rationale >>> ========= >>> >>> Historically in Python, file system paths have been represented as >>> strings or bytes. This choice of representation has stemmed from C's >>> own decision to represent file system paths as >>> ``const char *`` [#libc-open]_. While that is a totally serviceable >>> format to use for file system paths, it's not necessarily optimal. At >>> issue is the fact that while all file system paths can be represented >>> as strings or bytes, not all strings or bytes represent a file system >>> path. This can lead to issues where any e.g. string duck-types to a >>> file system path whether it actually represents a path or not. >>> >>> To help elevate the representation of file system paths from their >>> representation as strings and bytes to a richer object representation, >>> the pathlib module [#pathlib]_ was provisionally introduced in >>> Python 3.4 through PEP 428. While considered by some as an improvement >>> over strings and bytes for file system paths, it has suffered from a >>> lack of adoption. Typically the key issue listed for the low adoption >>> rate has been the lack of support in the standard library. This lack >>> of support required users of pathlib to manually convert path objects >>> to strings by calling ``str(path)`` which many found error-prone. >>> >>> One issue in converting path objects to strings comes from >>> the fact that the only generic way to get a string representation of >>> the path was to pass the object to ``str()``. This can pose a >>> problem when done blindly as nearly all Python objects have some >>> string representation whether they are a path or not, e.g. >>> ``str(None)`` will give a result that >>> ``builtins.open()`` [#builtins-open]_ will happily use to create a new >>> file. >>> >>> Exacerbating this whole situation is the >>> ``DirEntry`` object [#os-direntry]_. While path objects have a >>> representation that can be extracted using ``str()``, ``DirEntry`` >>> objects expose a ``path`` attribute instead. Having no common >>> interface between path objects, ``DirEntry``, and any other >>> third-party path library has become an issue. A solution that allows >>> any path-representing object to declare that it is a path and a way >>> to extract a low-level representation that all path objects could >>> support is desired. >>> >>> This PEP then proposes to introduce a new protocol to be followed by >>> objects which represent file system paths. Providing a protocol allows >>> for explicit signaling of what objects represent file system paths as >>> well as a way to extract a lower-level representation that can be used >>> with older APIs which only support strings or bytes. >>> >>> Discussions regarding path objects that led to this PEP can be found >>> in multiple threads on the python-ideas mailing list archive >>> [#python-ideas-archive]_ for the months of March and April 2016 and on >>> the python-dev mailing list archives [#python-dev-archive]_ during >>> April 2016. >>> >>> >>> Proposal >>> ======== >>> >>> This proposal is split into two parts. One part is the proposal of a >>> protocol for objects to declare and provide support for exposing a >>> file system path representation. The other part deals with changes to >>> Python's standard library to support the new protocol. These changes >>> will also lead to the pathlib module dropping its provisional status. >>> >>> Protocol >>> -------- >>> >>> The following abstract base class defines the protocol for an object >>> to be considered a path object:: >>> >>> import abc >>> import typing as t >>> >>> >>> class PathLike(abc.ABC): >>> >>> """Abstract base class for implementing the file system path >>> protocol.""" >>> >>> @abc.abstractmethod >>> def __fspath__(self) -> t.Union[str, bytes]: >>> """Return the file system path representation of the >>> object.""" >>> raise NotImplementedError >>> >>> >>> Objects representing file system paths will implement the >>> ``__fspath__()`` method which will return the ``str`` or ``bytes`` >>> representation of the path. The ``str`` representation is the >>> preferred low-level path representation as it is human-readable and >>> what people historically represent paths as. >>> >>> >>> Standard library changes >>> ------------------------ >>> >>> It is expected that most APIs in Python's standard library that >>> currently accept a file system path will be updated appropriately to >>> accept path objects (whether that requires code or simply an update >>> to documentation will vary). The modules mentioned below, though, >>> deserve specific details as they have either fundamental changes that >>> empower the ability to use path objects, or entail additions/removal >>> of APIs. >>> >>> >>> builtins >>> '''''''' >>> >>> ``open()`` [#builtins-open]_ will be updated to accept path objects as >>> well as continue to accept ``str`` and ``bytes``. >>> >>> >>> os >>> ''' >>> >>> The ``fspath()`` function will be added with the following semantics:: >>> >>> import typing as t >>> >>> >>> def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, >>> bytes]: >>> """Return the string representation of the path. >>> >>> If str or bytes is passed in, it is returned unchanged. >>> """ >>> if isinstance(path, (str, bytes)): >>> return path >>> >>> # Work from the object's type to match method resolution of >>> other magic >>> # methods. >>> path_type = type(path) >>> try: >>> return path_type.__fspath__(path) >>> except AttributeError: >>> if hasattr(path_type, '__fspath__'): >>> raise >>> >>> raise TypeError("expected str, bytes or os.PathLike object, >>> not " >>> + path_type.__name__) >>> >>> The ``os.fsencode()`` [#os-fsencode]_ and >>> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept >>> path objects. As both functions coerce their arguments to >>> ``bytes`` and ``str``, respectively, they will be updated to call >>> ``__fspath__()`` if present to convert the path object to a ``str`` or >>> ``bytes`` representation, and then perform their appropriate >>> coercion operations as if the return value from ``__fspath__()`` had >>> been the original argument to the coercion function in question. >>> >>> The addition of ``os.fspath()``, the updates to >>> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of >>> ``pathlib.PurePath`` provide the semantics necessary to >>> get the path representation one prefers. For a path object, >>> ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or >>> ``bytes`` representation without any coersion, then ``os.fspath()`` >>> can be used. If a ``str`` is desired and the encoding of ``bytes`` >>> should be assumed to be the default file system encoding, then >>> ``os.fsdecode()`` should be used. If a ``bytes`` representation is >>> desired and any strings should be encoded using the default file >>> system encoding, then ``os.fsencode()`` is used. This PEP recommends >>> using path objects when possible and falling back to string paths as >>> necessary and using ``bytes`` as a last resort. >>> >>> Another way to view this is as a hierarchy of file system path >>> representations (highest- to lowest-level): path ? str ? bytes. The >>> functions and classes under discussion can all accept objects on the >>> same level of the hierarchy, but they vary in whether they promote or >>> demote objects to another level. The ``pathlib.PurePath`` class can >>> promote a ``str`` to a path object. The ``os.fspath()`` function can >>> demote a path object to a ``str`` or ``bytes`` instance, depending >>> on what ``__fspath__()`` returns. >>> The ``os.fsdecode()`` function will demote a path object to >>> a string or promote a ``bytes`` object to a ``str``. The >>> ``os.fsencode()`` function will demote a path or string object to >>> ``bytes``. There is no function that provides a way to demote a path >>> object directly to ``bytes`` while bypassing string demotion. >>> >>> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` >>> method. It will return the same value as currently found on the >>> ``path`` attribute of ``DirEntry`` instances. >>> >>> The Protocol_ ABC will be added to the ``os`` module under the name >>> ``os.PathLike``. >>> >>> >>> os.path >>> ''''''' >>> >>> The various path-manipulation functions of ``os.path`` [#os-path]_ >>> will be updated to accept path objects. For polymorphic functions that >>> accept both bytes and strings, they will be updated to simply use >>> ``os.fspath()``. >>> >>> During the discussions leading up to this PEP it was suggested that >>> ``os.path`` not be updated using an "explicit is better than implicit" >>> argument. The thinking was that since ``__fspath__()`` is polymorphic >>> itself it may be better to have code working with ``os.path`` extract >>> the path representation from path objects explicitly. There is also >>> the consideration that adding support this deep into the low-level OS >>> APIs will lead to code magically supporting path objects without >>> requiring any documentation updated, leading to potential complaints >>> when it doesn't work, unbeknownst to the project author. >>> >>> But it is the view of this PEP that "practicality beats purity" in >>> this instance. To help facilitate the transition to supporting path >>> objects, it is better to make the transition as easy as possible than >>> to worry about unexpected/undocumented duck typing support for >>> path objects by projects. >>> >>> There has also been the suggestion that ``os.path`` functions could be >>> used in a tight loop and the overhead of checking or calling >>> ``__fspath__()`` would be too costly. In this scenario only >>> path-consuming APIs would be directly updated and path-manipulating >>> APIs like the ones in ``os.path`` would go unmodified. This would >>> require library authors to update their code to support path objects >>> if they performed any path manipulations, but if the library code >>> passed the path straight through then the library wouldn't need to be >>> updated. It is the view of this PEP and Guido, though, that this is an >>> unnecessary worry and that performance will still be acceptable. >>> >>> >>> pathlib >>> ''''''' >>> >>> The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be >>> updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path`` >>> will continue to not accept ``bytes`` path representations, and so if >>> ``__fspath__()`` returns ``bytes`` it will raise an exception. >>> >>> The ``path`` attribute will be removed as this PEP makes it >>> redundant (it has not been included in any released version of Python >>> and so is not a backwards-compatibility concern). >>> >>> >>> C API >>> ''''' >>> >>> The C API will gain an equivalent function to ``os.fspath()``:: >>> >>> /* >>> Return the file system path of the object. >>> >>> If the object is str or bytes, then allow it to pass through with >>> an incremented refcount. If the object defines __fspath__(), then >>> return the result of that method. All other types raise a >>> TypeError. >>> */ >>> PyObject * >>> PyOS_FSPath(PyObject *path) >>> { >>> if (PyUnicode_Check(path) || PyBytes_Check(path)) { >>> Py_INCREF(path); >>> return path; >>> } >>> >>> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { >>> return PyObject_CallMethodObjArgs(path->ob_type, >>> "__fspath__", path, >>> NULL); >>> } >>> >>> return PyErr_Format(PyExc_TypeError, >>> "expected a str, bytes, or os.PathLike >>> object, not %S", >>> path->ob_type); >>> } >>> >>> >>> >>> Backwards compatibility >>> ======================= >>> >>> There are no explicit backwards-compatibility concerns. Unless an >>> object incidentally already defines a ``__fspath__()`` method there is >>> no reason to expect the pre-existing code to break or expect to have >>> its semantics implicitly changed. >>> >>> Libraries wishing to support path objects and a version of Python >>> prior to Python 3.6 and the existence of ``os.fspath()`` can use the >>> idiom of >>> ``path.__fspath__() if hasattr(path, "__fspath__") else path``. >>> >>> >>> Implementation >>> ============== >>> >>> This is the task list for what this PEP proposes: >>> >>> #. Remove the ``path`` attribute from pathlib >>> #. Remove the provisional status of pathlib >>> #. Add ``os.PathLike`` >>> #. Add ``os.fspath()`` >>> #. Add ``PyOS_FSPath()`` >>> #. Update ``os.fsencode()`` >>> #. Update ``os.fsdecode()`` >>> #. Update ``pathlib.PurePath`` and ``pathlib.Path`` >>> #. Update ``builtins.open()`` >>> #. Update ``os.DirEntry`` >>> #. Update ``os.path`` >>> #. Add a glossary entry for "path-like" >>> >>> >>> Rejected Ideas >>> ============== >>> >>> Other names for the protocol's method >>> ------------------------------------- >>> >>> Various names were proposed during discussions leading to this PEP, >>> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In >>> the end people seemed to gravitate towards ``__fspath__`` for being >>> unambiguous without being unnecessarily long. >>> >>> >>> Separate str/bytes methods >>> -------------------------- >>> >>> At one point it was suggested that ``__fspath__()`` only return >>> strings and another method named ``__fspathb__()`` be introduced to >>> return bytes. The thinking is that by making ``__fspath__()`` not be >>> polymorphic it could make dealing with the potential string or bytes >>> representations easier. But the general consensus was that returning >>> bytes will more than likely be rare and that the various functions in >>> the os module are the better abstraction to promote over direct >>> calls to ``__fspath__()``. >>> >>> >>> Providing a ``path`` attribute >>> ------------------------------ >>> >>> To help deal with the issue of ``pathlib.PurePath`` not inheriting >>> from ``str``, originally it was proposed to introduce a ``path`` >>> attribute to mirror what ``os.DirEntry`` provides. In the end, >>> though, it was determined that a protocol would provide the same >>> result while not directly exposing an API that most people will never >>> need to interact with directly. >>> >>> >>> Have ``__fspath__()`` only return strings >>> ------------------------------------------ >>> >>> Much of the discussion that led to this PEP revolved around whether >>> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as >>> ``str`` or only return ``str``. The general sentiment for this view >>> was that ``bytes`` are difficult to work with due to their >>> inherent lack of information about their encoding and PEP 383 makes >>> it possible to represent all file system paths using ``str`` with the >>> ``surrogateescape`` handler. Thus, it would be better to forcibly >>> promote the use of ``str`` as the low-level path representation for >>> high-level path objects. >>> >>> In the end, it was decided that using ``bytes`` to represent paths is >>> simply not going to go away and thus they should be supported to some >>> degree. The hope is that people will gravitate towards path objects >>> like pathlib and that will move people away from operating directly >>> with ``bytes``. >>> >>> >>> A generic string encoding mechanism >>> ----------------------------------- >>> >>> At one point there was a discussion of developing a generic mechanism >>> to extract a string representation of an object that had semantic >>> meaning (``__str__()`` does not necessarily return anything of >>> semantic significance beyond what may be helpful for debugging). In >>> the end, it was deemed to lack a motivating need beyond the one this >>> PEP is trying to solve in a specific fashion. >>> >>> >>> Have __fspath__ be an attribute >>> ------------------------------- >>> >>> It was briefly considered to have ``__fspath__`` be an attribute >>> instead of a method. This was rejected for two reasons. One, >>> historically protocols have been implemented as "magic methods" and >>> not "magic methods and attributes". Two, there is no guarantee that >>> the lower-level representation of a path object will be pre-computed, >>> potentially misleading users that there was no expensive computation >>> behind the scenes in case the attribute was implemented as a property. >>> >>> This also indirectly ties into the idea of introducing a ``path`` >>> attribute to accomplish the same thing. This idea has an added issue, >>> though, of accidentally having any object with a ``path`` attribute >>> meet the protocol's duck typing. Introducing a new magic method for >>> the protocol helpfully avoids any accidental opting into the protocol. >>> >>> >>> Provide specific type hinting support >>> ------------------------------------- >>> >>> There was some consideration to provdinga generic ``typing.PathLike`` >>> class which would allow for e.g. ``typing.PathLike[str]`` to specify >>> a type hint for a path object which returned a string representation. >>> While potentially beneficial, the usefulness was deemed too small to >>> bother adding the type hint class. >>> >>> This also removed any desire to have a class in the ``typing`` module >>> which represented the union of all acceptable path-representing types >>> as that can be represented with >>> ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope >>> is users will slowly gravitate to path objects only. >>> >>> >>> Provide ``os.fspathb()`` >>> ------------------------ >>> >>> It was suggested that to mirror the structure of e.g. >>> ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return >>> ``str`` and that another function named ``os.fspathb()`` be >>> introduced that only returned ``bytes``. This was rejected as the >>> purposes of the ``*b()`` functions are tied to querying the file >>> system where there is a need to get the raw bytes back. As this PEP >>> does not work directly with data on a file system (but which *may* >>> be), the view was taken this distinction is unnecessary. It's also >>> believed that the need for only bytes will not be common enough to >>> need to support in such a specific manner as ``os.fsencode()`` will >>> provide similar functionality. >>> >>> >>> Call ``__fspath__()`` off of the instance >>> ----------------------------------------- >>> >>> An earlier draft of this PEP had ``os.fspath()`` calling >>> ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The >>> changed to be consistent with how other magic methods in Python are >>> resolved. >>> >>> >>> Acknowledgements >>> ================ >>> >>> Thanks to everyone who participated in the various discussions related >>> to this PEP that spanned both python-ideas and python-dev. Special >>> thanks to Stephen Turnbull for direct feedback on early drafts of this >>> PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not >>> only feedback on early drafts of this PEP but also helping to drive >>> the overall discussion on this topic across the two mailing lists. >>> >>> >>> References >>> ========== >>> >>> .. [#python-ideas-archive] The python-ideas mailing list archive >>> (https://mail.python.org/pipermail/python-ideas/) >>> >>> .. [#python-dev-archive] The python-dev mailing list archive >>> (https://mail.python.org/pipermail/python-dev/) >>> >>> .. [#libc-open] ``open()`` documention for the C standard library >>> ( >>> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html >>> ) >>> >>> .. [#pathlib] The ``pathlib`` module >>> (https://docs.python.org/3/library/pathlib.html#module-pathlib) >>> >>> .. [#builtins-open] The ``builtins.open()`` function >>> (https://docs.python.org/3/library/functions.html#open) >>> >>> .. [#os-fsencode] The ``os.fsencode()`` function >>> (https://docs.python.org/3/library/os.html#os.fsencode) >>> >>> .. [#os-fsdecode] The ``os.fsdecode()`` function >>> (https://docs.python.org/3/library/os.html#os.fsdecode) >>> >>> .. [#os-direntry] The ``os.DirEntry`` class >>> (https://docs.python.org/3/library/os.html#os.DirEntry) >>> >>> .. [#os-path] The ``os.path`` module >>> (https://docs.python.org/3/library/os.path.html#module-os.path) >>> >>> >>> Copyright >>> ========= >>> >>> This document has been placed in the public domain. >>> >>> >>> .. >>> Local Variables: >>> mode: indented-text >>> indent-tabs-mode: nil >>> sentence-end-double-space: t >>> fill-column: 70 >>> coding: utf-8 >>> End: >>> >>> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 16 17:02:42 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 16 May 2016 14:02:42 -0700 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: For those following along on the list, it's PEP 519: https://www.python.org/dev/peps/pep-0519/ I've read it once more and found nothing worth bickering about, so I am hereby approving PEP 519. Thanks Brett and Koos for getting this one over the finish line, and congrats! It's been quite an effort (I personally did not have the patience to keep up with the thread...). On Mon, May 16, 2016 at 1:26 PM, Brett Cannon wrote: > > > On Mon, 16 May 2016 at 13:12 Guido van Rossum wrote: > >> Once you assign yourself a PEP number I'll do one more pass and then I >> expect to accept it -- the draft looks good to me! >> > > Done: https://hg.python.org/peps/rev/b41cb718054a > > >> >> On Mon, May 16, 2016 at 1:00 PM, Brett Cannon wrote: >> >>> Recent discussions have been about type hints which are orthogonal to >>> the PEP, so things have seemed to have reached a steady state. >>> >>> Was there anything else that needed clarification, Guido, or are you >>> ready to pronounce? Or did you want to wait until the language summit? Or >>> did you want to assign a BDFL delegate? >>> >>> >>> On Fri, 13 May 2016 at 11:37 Brett Cannon wrote: >>> >>>> Biggest changes since the second draft: >>>> >>>> 1. Resolve __fspath__() from the type, not the instance (for Guido) >>>> 2. Updated the TypeError messages to say "os.PathLike object" >>>> instead of "path object" (implicitly for Steven) >>>> 3. TODO item to define "path-like" in the glossary (for Steven) >>>> 4. Various more things added to Rejected Ideas >>>> 5. Added Koos as a co-author (for Koos :) >>>> >>>> ---------- >>>> PEP: NNN >>>> Title: Adding a file system path protocol >>>> Version: $Revision$ >>>> Last-Modified: $Date$ >>>> Author: Brett Cannon , >>>> Koos Zevenhoven >>>> Status: Draft >>>> Type: Standards Track >>>> Content-Type: text/x-rst >>>> Created: 11-May-2016 >>>> Post-History: 11-May-2016, >>>> 12-May-2016, >>>> 13-May-2016 >>>> >>>> >>>> Abstract >>>> ======== >>>> >>>> This PEP proposes a protocol for classes which represent a file system >>>> path to be able to provide a ``str`` or ``bytes`` representation. >>>> Changes to Python's standard library are also proposed to utilize this >>>> protocol where appropriate to facilitate the use of path objects where >>>> historically only ``str`` and/or ``bytes`` file system paths are >>>> accepted. The goal is to facilitate the migration of users towards >>>> rich path objects while providing an easy way to work with code >>>> expecting ``str`` or ``bytes``. >>>> >>>> >>>> Rationale >>>> ========= >>>> >>>> Historically in Python, file system paths have been represented as >>>> strings or bytes. This choice of representation has stemmed from C's >>>> own decision to represent file system paths as >>>> ``const char *`` [#libc-open]_. While that is a totally serviceable >>>> format to use for file system paths, it's not necessarily optimal. At >>>> issue is the fact that while all file system paths can be represented >>>> as strings or bytes, not all strings or bytes represent a file system >>>> path. This can lead to issues where any e.g. string duck-types to a >>>> file system path whether it actually represents a path or not. >>>> >>>> To help elevate the representation of file system paths from their >>>> representation as strings and bytes to a richer object representation, >>>> the pathlib module [#pathlib]_ was provisionally introduced in >>>> Python 3.4 through PEP 428. While considered by some as an improvement >>>> over strings and bytes for file system paths, it has suffered from a >>>> lack of adoption. Typically the key issue listed for the low adoption >>>> rate has been the lack of support in the standard library. This lack >>>> of support required users of pathlib to manually convert path objects >>>> to strings by calling ``str(path)`` which many found error-prone. >>>> >>>> One issue in converting path objects to strings comes from >>>> the fact that the only generic way to get a string representation of >>>> the path was to pass the object to ``str()``. This can pose a >>>> problem when done blindly as nearly all Python objects have some >>>> string representation whether they are a path or not, e.g. >>>> ``str(None)`` will give a result that >>>> ``builtins.open()`` [#builtins-open]_ will happily use to create a new >>>> file. >>>> >>>> Exacerbating this whole situation is the >>>> ``DirEntry`` object [#os-direntry]_. While path objects have a >>>> representation that can be extracted using ``str()``, ``DirEntry`` >>>> objects expose a ``path`` attribute instead. Having no common >>>> interface between path objects, ``DirEntry``, and any other >>>> third-party path library has become an issue. A solution that allows >>>> any path-representing object to declare that it is a path and a way >>>> to extract a low-level representation that all path objects could >>>> support is desired. >>>> >>>> This PEP then proposes to introduce a new protocol to be followed by >>>> objects which represent file system paths. Providing a protocol allows >>>> for explicit signaling of what objects represent file system paths as >>>> well as a way to extract a lower-level representation that can be used >>>> with older APIs which only support strings or bytes. >>>> >>>> Discussions regarding path objects that led to this PEP can be found >>>> in multiple threads on the python-ideas mailing list archive >>>> [#python-ideas-archive]_ for the months of March and April 2016 and on >>>> the python-dev mailing list archives [#python-dev-archive]_ during >>>> April 2016. >>>> >>>> >>>> Proposal >>>> ======== >>>> >>>> This proposal is split into two parts. One part is the proposal of a >>>> protocol for objects to declare and provide support for exposing a >>>> file system path representation. The other part deals with changes to >>>> Python's standard library to support the new protocol. These changes >>>> will also lead to the pathlib module dropping its provisional status. >>>> >>>> Protocol >>>> -------- >>>> >>>> The following abstract base class defines the protocol for an object >>>> to be considered a path object:: >>>> >>>> import abc >>>> import typing as t >>>> >>>> >>>> class PathLike(abc.ABC): >>>> >>>> """Abstract base class for implementing the file system path >>>> protocol.""" >>>> >>>> @abc.abstractmethod >>>> def __fspath__(self) -> t.Union[str, bytes]: >>>> """Return the file system path representation of the >>>> object.""" >>>> raise NotImplementedError >>>> >>>> >>>> Objects representing file system paths will implement the >>>> ``__fspath__()`` method which will return the ``str`` or ``bytes`` >>>> representation of the path. The ``str`` representation is the >>>> preferred low-level path representation as it is human-readable and >>>> what people historically represent paths as. >>>> >>>> >>>> Standard library changes >>>> ------------------------ >>>> >>>> It is expected that most APIs in Python's standard library that >>>> currently accept a file system path will be updated appropriately to >>>> accept path objects (whether that requires code or simply an update >>>> to documentation will vary). The modules mentioned below, though, >>>> deserve specific details as they have either fundamental changes that >>>> empower the ability to use path objects, or entail additions/removal >>>> of APIs. >>>> >>>> >>>> builtins >>>> '''''''' >>>> >>>> ``open()`` [#builtins-open]_ will be updated to accept path objects as >>>> well as continue to accept ``str`` and ``bytes``. >>>> >>>> >>>> os >>>> ''' >>>> >>>> The ``fspath()`` function will be added with the following semantics:: >>>> >>>> import typing as t >>>> >>>> >>>> def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, >>>> bytes]: >>>> """Return the string representation of the path. >>>> >>>> If str or bytes is passed in, it is returned unchanged. >>>> """ >>>> if isinstance(path, (str, bytes)): >>>> return path >>>> >>>> # Work from the object's type to match method resolution of >>>> other magic >>>> # methods. >>>> path_type = type(path) >>>> try: >>>> return path_type.__fspath__(path) >>>> except AttributeError: >>>> if hasattr(path_type, '__fspath__'): >>>> raise >>>> >>>> raise TypeError("expected str, bytes or os.PathLike object, >>>> not " >>>> + path_type.__name__) >>>> >>>> The ``os.fsencode()`` [#os-fsencode]_ and >>>> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept >>>> path objects. As both functions coerce their arguments to >>>> ``bytes`` and ``str``, respectively, they will be updated to call >>>> ``__fspath__()`` if present to convert the path object to a ``str`` or >>>> ``bytes`` representation, and then perform their appropriate >>>> coercion operations as if the return value from ``__fspath__()`` had >>>> been the original argument to the coercion function in question. >>>> >>>> The addition of ``os.fspath()``, the updates to >>>> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of >>>> ``pathlib.PurePath`` provide the semantics necessary to >>>> get the path representation one prefers. For a path object, >>>> ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or >>>> ``bytes`` representation without any coersion, then ``os.fspath()`` >>>> can be used. If a ``str`` is desired and the encoding of ``bytes`` >>>> should be assumed to be the default file system encoding, then >>>> ``os.fsdecode()`` should be used. If a ``bytes`` representation is >>>> desired and any strings should be encoded using the default file >>>> system encoding, then ``os.fsencode()`` is used. This PEP recommends >>>> using path objects when possible and falling back to string paths as >>>> necessary and using ``bytes`` as a last resort. >>>> >>>> Another way to view this is as a hierarchy of file system path >>>> representations (highest- to lowest-level): path ? str ? bytes. The >>>> functions and classes under discussion can all accept objects on the >>>> same level of the hierarchy, but they vary in whether they promote or >>>> demote objects to another level. The ``pathlib.PurePath`` class can >>>> promote a ``str`` to a path object. The ``os.fspath()`` function can >>>> demote a path object to a ``str`` or ``bytes`` instance, depending >>>> on what ``__fspath__()`` returns. >>>> The ``os.fsdecode()`` function will demote a path object to >>>> a string or promote a ``bytes`` object to a ``str``. The >>>> ``os.fsencode()`` function will demote a path or string object to >>>> ``bytes``. There is no function that provides a way to demote a path >>>> object directly to ``bytes`` while bypassing string demotion. >>>> >>>> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` >>>> method. It will return the same value as currently found on the >>>> ``path`` attribute of ``DirEntry`` instances. >>>> >>>> The Protocol_ ABC will be added to the ``os`` module under the name >>>> ``os.PathLike``. >>>> >>>> >>>> os.path >>>> ''''''' >>>> >>>> The various path-manipulation functions of ``os.path`` [#os-path]_ >>>> will be updated to accept path objects. For polymorphic functions that >>>> accept both bytes and strings, they will be updated to simply use >>>> ``os.fspath()``. >>>> >>>> During the discussions leading up to this PEP it was suggested that >>>> ``os.path`` not be updated using an "explicit is better than implicit" >>>> argument. The thinking was that since ``__fspath__()`` is polymorphic >>>> itself it may be better to have code working with ``os.path`` extract >>>> the path representation from path objects explicitly. There is also >>>> the consideration that adding support this deep into the low-level OS >>>> APIs will lead to code magically supporting path objects without >>>> requiring any documentation updated, leading to potential complaints >>>> when it doesn't work, unbeknownst to the project author. >>>> >>>> But it is the view of this PEP that "practicality beats purity" in >>>> this instance. To help facilitate the transition to supporting path >>>> objects, it is better to make the transition as easy as possible than >>>> to worry about unexpected/undocumented duck typing support for >>>> path objects by projects. >>>> >>>> There has also been the suggestion that ``os.path`` functions could be >>>> used in a tight loop and the overhead of checking or calling >>>> ``__fspath__()`` would be too costly. In this scenario only >>>> path-consuming APIs would be directly updated and path-manipulating >>>> APIs like the ones in ``os.path`` would go unmodified. This would >>>> require library authors to update their code to support path objects >>>> if they performed any path manipulations, but if the library code >>>> passed the path straight through then the library wouldn't need to be >>>> updated. It is the view of this PEP and Guido, though, that this is an >>>> unnecessary worry and that performance will still be acceptable. >>>> >>>> >>>> pathlib >>>> ''''''' >>>> >>>> The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be >>>> updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path`` >>>> will continue to not accept ``bytes`` path representations, and so if >>>> ``__fspath__()`` returns ``bytes`` it will raise an exception. >>>> >>>> The ``path`` attribute will be removed as this PEP makes it >>>> redundant (it has not been included in any released version of Python >>>> and so is not a backwards-compatibility concern). >>>> >>>> >>>> C API >>>> ''''' >>>> >>>> The C API will gain an equivalent function to ``os.fspath()``:: >>>> >>>> /* >>>> Return the file system path of the object. >>>> >>>> If the object is str or bytes, then allow it to pass through >>>> with >>>> an incremented refcount. If the object defines __fspath__(), >>>> then >>>> return the result of that method. All other types raise a >>>> TypeError. >>>> */ >>>> PyObject * >>>> PyOS_FSPath(PyObject *path) >>>> { >>>> if (PyUnicode_Check(path) || PyBytes_Check(path)) { >>>> Py_INCREF(path); >>>> return path; >>>> } >>>> >>>> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { >>>> return PyObject_CallMethodObjArgs(path->ob_type, >>>> "__fspath__", path, >>>> NULL); >>>> } >>>> >>>> return PyErr_Format(PyExc_TypeError, >>>> "expected a str, bytes, or os.PathLike >>>> object, not %S", >>>> path->ob_type); >>>> } >>>> >>>> >>>> >>>> Backwards compatibility >>>> ======================= >>>> >>>> There are no explicit backwards-compatibility concerns. Unless an >>>> object incidentally already defines a ``__fspath__()`` method there is >>>> no reason to expect the pre-existing code to break or expect to have >>>> its semantics implicitly changed. >>>> >>>> Libraries wishing to support path objects and a version of Python >>>> prior to Python 3.6 and the existence of ``os.fspath()`` can use the >>>> idiom of >>>> ``path.__fspath__() if hasattr(path, "__fspath__") else path``. >>>> >>>> >>>> Implementation >>>> ============== >>>> >>>> This is the task list for what this PEP proposes: >>>> >>>> #. Remove the ``path`` attribute from pathlib >>>> #. Remove the provisional status of pathlib >>>> #. Add ``os.PathLike`` >>>> #. Add ``os.fspath()`` >>>> #. Add ``PyOS_FSPath()`` >>>> #. Update ``os.fsencode()`` >>>> #. Update ``os.fsdecode()`` >>>> #. Update ``pathlib.PurePath`` and ``pathlib.Path`` >>>> #. Update ``builtins.open()`` >>>> #. Update ``os.DirEntry`` >>>> #. Update ``os.path`` >>>> #. Add a glossary entry for "path-like" >>>> >>>> >>>> Rejected Ideas >>>> ============== >>>> >>>> Other names for the protocol's method >>>> ------------------------------------- >>>> >>>> Various names were proposed during discussions leading to this PEP, >>>> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In >>>> the end people seemed to gravitate towards ``__fspath__`` for being >>>> unambiguous without being unnecessarily long. >>>> >>>> >>>> Separate str/bytes methods >>>> -------------------------- >>>> >>>> At one point it was suggested that ``__fspath__()`` only return >>>> strings and another method named ``__fspathb__()`` be introduced to >>>> return bytes. The thinking is that by making ``__fspath__()`` not be >>>> polymorphic it could make dealing with the potential string or bytes >>>> representations easier. But the general consensus was that returning >>>> bytes will more than likely be rare and that the various functions in >>>> the os module are the better abstraction to promote over direct >>>> calls to ``__fspath__()``. >>>> >>>> >>>> Providing a ``path`` attribute >>>> ------------------------------ >>>> >>>> To help deal with the issue of ``pathlib.PurePath`` not inheriting >>>> from ``str``, originally it was proposed to introduce a ``path`` >>>> attribute to mirror what ``os.DirEntry`` provides. In the end, >>>> though, it was determined that a protocol would provide the same >>>> result while not directly exposing an API that most people will never >>>> need to interact with directly. >>>> >>>> >>>> Have ``__fspath__()`` only return strings >>>> ------------------------------------------ >>>> >>>> Much of the discussion that led to this PEP revolved around whether >>>> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as >>>> ``str`` or only return ``str``. The general sentiment for this view >>>> was that ``bytes`` are difficult to work with due to their >>>> inherent lack of information about their encoding and PEP 383 makes >>>> it possible to represent all file system paths using ``str`` with the >>>> ``surrogateescape`` handler. Thus, it would be better to forcibly >>>> promote the use of ``str`` as the low-level path representation for >>>> high-level path objects. >>>> >>>> In the end, it was decided that using ``bytes`` to represent paths is >>>> simply not going to go away and thus they should be supported to some >>>> degree. The hope is that people will gravitate towards path objects >>>> like pathlib and that will move people away from operating directly >>>> with ``bytes``. >>>> >>>> >>>> A generic string encoding mechanism >>>> ----------------------------------- >>>> >>>> At one point there was a discussion of developing a generic mechanism >>>> to extract a string representation of an object that had semantic >>>> meaning (``__str__()`` does not necessarily return anything of >>>> semantic significance beyond what may be helpful for debugging). In >>>> the end, it was deemed to lack a motivating need beyond the one this >>>> PEP is trying to solve in a specific fashion. >>>> >>>> >>>> Have __fspath__ be an attribute >>>> ------------------------------- >>>> >>>> It was briefly considered to have ``__fspath__`` be an attribute >>>> instead of a method. This was rejected for two reasons. One, >>>> historically protocols have been implemented as "magic methods" and >>>> not "magic methods and attributes". Two, there is no guarantee that >>>> the lower-level representation of a path object will be pre-computed, >>>> potentially misleading users that there was no expensive computation >>>> behind the scenes in case the attribute was implemented as a property. >>>> >>>> This also indirectly ties into the idea of introducing a ``path`` >>>> attribute to accomplish the same thing. This idea has an added issue, >>>> though, of accidentally having any object with a ``path`` attribute >>>> meet the protocol's duck typing. Introducing a new magic method for >>>> the protocol helpfully avoids any accidental opting into the protocol. >>>> >>>> >>>> Provide specific type hinting support >>>> ------------------------------------- >>>> >>>> There was some consideration to provdinga generic ``typing.PathLike`` >>>> class which would allow for e.g. ``typing.PathLike[str]`` to specify >>>> a type hint for a path object which returned a string representation. >>>> While potentially beneficial, the usefulness was deemed too small to >>>> bother adding the type hint class. >>>> >>>> This also removed any desire to have a class in the ``typing`` module >>>> which represented the union of all acceptable path-representing types >>>> as that can be represented with >>>> ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope >>>> is users will slowly gravitate to path objects only. >>>> >>>> >>>> Provide ``os.fspathb()`` >>>> ------------------------ >>>> >>>> It was suggested that to mirror the structure of e.g. >>>> ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return >>>> ``str`` and that another function named ``os.fspathb()`` be >>>> introduced that only returned ``bytes``. This was rejected as the >>>> purposes of the ``*b()`` functions are tied to querying the file >>>> system where there is a need to get the raw bytes back. As this PEP >>>> does not work directly with data on a file system (but which *may* >>>> be), the view was taken this distinction is unnecessary. It's also >>>> believed that the need for only bytes will not be common enough to >>>> need to support in such a specific manner as ``os.fsencode()`` will >>>> provide similar functionality. >>>> >>>> >>>> Call ``__fspath__()`` off of the instance >>>> ----------------------------------------- >>>> >>>> An earlier draft of this PEP had ``os.fspath()`` calling >>>> ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The >>>> changed to be consistent with how other magic methods in Python are >>>> resolved. >>>> >>>> >>>> Acknowledgements >>>> ================ >>>> >>>> Thanks to everyone who participated in the various discussions related >>>> to this PEP that spanned both python-ideas and python-dev. Special >>>> thanks to Stephen Turnbull for direct feedback on early drafts of this >>>> PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not >>>> only feedback on early drafts of this PEP but also helping to drive >>>> the overall discussion on this topic across the two mailing lists. >>>> >>>> >>>> References >>>> ========== >>>> >>>> .. [#python-ideas-archive] The python-ideas mailing list archive >>>> (https://mail.python.org/pipermail/python-ideas/) >>>> >>>> .. [#python-dev-archive] The python-dev mailing list archive >>>> (https://mail.python.org/pipermail/python-dev/) >>>> >>>> .. [#libc-open] ``open()`` documention for the C standard library >>>> ( >>>> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html >>>> ) >>>> >>>> .. [#pathlib] The ``pathlib`` module >>>> (https://docs.python.org/3/library/pathlib.html#module-pathlib) >>>> >>>> .. [#builtins-open] The ``builtins.open()`` function >>>> (https://docs.python.org/3/library/functions.html#open) >>>> >>>> .. [#os-fsencode] The ``os.fsencode()`` function >>>> (https://docs.python.org/3/library/os.html#os.fsencode) >>>> >>>> .. [#os-fsdecode] The ``os.fsdecode()`` function >>>> (https://docs.python.org/3/library/os.html#os.fsdecode) >>>> >>>> .. [#os-direntry] The ``os.DirEntry`` class >>>> (https://docs.python.org/3/library/os.html#os.DirEntry) >>>> >>>> .. [#os-path] The ``os.path`` module >>>> (https://docs.python.org/3/library/os.path.html#module-os.path) >>>> >>>> >>>> Copyright >>>> ========= >>>> >>>> This document has been placed in the public domain. >>>> >>>> >>>> .. >>>> Local Variables: >>>> mode: indented-text >>>> indent-tabs-mode: nil >>>> sentence-end-double-space: t >>>> fill-column: 70 >>>> coding: utf-8 >>>> End: >>>> >>>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 16 17:32:48 2016 From: brett at python.org (Brett Cannon) Date: Mon, 16 May 2016 21:32:48 +0000 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: On Mon, 16 May 2016 at 14:03 Guido van Rossum wrote: > For those following along on the list, it's PEP 519: > https://www.python.org/dev/peps/pep-0519/ > > I've read it once more and found nothing worth bickering about, so I am > hereby approving PEP 519. Thanks Brett and Koos for getting this one over > the finish line, and congrats! It's been quite an effort (I personally did > not have the patience to keep up with the thread...). > PEP 519 is now marked as accepted! Two PEPs accepted in one day (Nick approved 518 earlier)! If pathlib doesn't get updated in 3.4, 3.5, and 3.6 before PyCon it will be the first thing I do at the sprints so that it isn't an issue for the 3.5.2 release. -Brett > > > > On Mon, May 16, 2016 at 1:26 PM, Brett Cannon wrote: > >> >> >> On Mon, 16 May 2016 at 13:12 Guido van Rossum wrote: >> >>> Once you assign yourself a PEP number I'll do one more pass and then I >>> expect to accept it -- the draft looks good to me! >>> >> >> Done: https://hg.python.org/peps/rev/b41cb718054a >> >> >>> >>> On Mon, May 16, 2016 at 1:00 PM, Brett Cannon wrote: >>> >>>> Recent discussions have been about type hints which are orthogonal to >>>> the PEP, so things have seemed to have reached a steady state. >>>> >>>> Was there anything else that needed clarification, Guido, or are you >>>> ready to pronounce? Or did you want to wait until the language summit? Or >>>> did you want to assign a BDFL delegate? >>>> >>>> >>>> On Fri, 13 May 2016 at 11:37 Brett Cannon wrote: >>>> >>>>> Biggest changes since the second draft: >>>>> >>>>> 1. Resolve __fspath__() from the type, not the instance (for Guido) >>>>> 2. Updated the TypeError messages to say "os.PathLike object" >>>>> instead of "path object" (implicitly for Steven) >>>>> 3. TODO item to define "path-like" in the glossary (for Steven) >>>>> 4. Various more things added to Rejected Ideas >>>>> 5. Added Koos as a co-author (for Koos :) >>>>> >>>>> ---------- >>>>> PEP: NNN >>>>> Title: Adding a file system path protocol >>>>> Version: $Revision$ >>>>> Last-Modified: $Date$ >>>>> Author: Brett Cannon , >>>>> Koos Zevenhoven >>>>> Status: Draft >>>>> Type: Standards Track >>>>> Content-Type: text/x-rst >>>>> Created: 11-May-2016 >>>>> Post-History: 11-May-2016, >>>>> 12-May-2016, >>>>> 13-May-2016 >>>>> >>>>> >>>>> Abstract >>>>> ======== >>>>> >>>>> This PEP proposes a protocol for classes which represent a file system >>>>> path to be able to provide a ``str`` or ``bytes`` representation. >>>>> Changes to Python's standard library are also proposed to utilize this >>>>> protocol where appropriate to facilitate the use of path objects where >>>>> historically only ``str`` and/or ``bytes`` file system paths are >>>>> accepted. The goal is to facilitate the migration of users towards >>>>> rich path objects while providing an easy way to work with code >>>>> expecting ``str`` or ``bytes``. >>>>> >>>>> >>>>> Rationale >>>>> ========= >>>>> >>>>> Historically in Python, file system paths have been represented as >>>>> strings or bytes. This choice of representation has stemmed from C's >>>>> own decision to represent file system paths as >>>>> ``const char *`` [#libc-open]_. While that is a totally serviceable >>>>> format to use for file system paths, it's not necessarily optimal. At >>>>> issue is the fact that while all file system paths can be represented >>>>> as strings or bytes, not all strings or bytes represent a file system >>>>> path. This can lead to issues where any e.g. string duck-types to a >>>>> file system path whether it actually represents a path or not. >>>>> >>>>> To help elevate the representation of file system paths from their >>>>> representation as strings and bytes to a richer object representation, >>>>> the pathlib module [#pathlib]_ was provisionally introduced in >>>>> Python 3.4 through PEP 428. While considered by some as an improvement >>>>> over strings and bytes for file system paths, it has suffered from a >>>>> lack of adoption. Typically the key issue listed for the low adoption >>>>> rate has been the lack of support in the standard library. This lack >>>>> of support required users of pathlib to manually convert path objects >>>>> to strings by calling ``str(path)`` which many found error-prone. >>>>> >>>>> One issue in converting path objects to strings comes from >>>>> the fact that the only generic way to get a string representation of >>>>> the path was to pass the object to ``str()``. This can pose a >>>>> problem when done blindly as nearly all Python objects have some >>>>> string representation whether they are a path or not, e.g. >>>>> ``str(None)`` will give a result that >>>>> ``builtins.open()`` [#builtins-open]_ will happily use to create a new >>>>> file. >>>>> >>>>> Exacerbating this whole situation is the >>>>> ``DirEntry`` object [#os-direntry]_. While path objects have a >>>>> representation that can be extracted using ``str()``, ``DirEntry`` >>>>> objects expose a ``path`` attribute instead. Having no common >>>>> interface between path objects, ``DirEntry``, and any other >>>>> third-party path library has become an issue. A solution that allows >>>>> any path-representing object to declare that it is a path and a way >>>>> to extract a low-level representation that all path objects could >>>>> support is desired. >>>>> >>>>> This PEP then proposes to introduce a new protocol to be followed by >>>>> objects which represent file system paths. Providing a protocol allows >>>>> for explicit signaling of what objects represent file system paths as >>>>> well as a way to extract a lower-level representation that can be used >>>>> with older APIs which only support strings or bytes. >>>>> >>>>> Discussions regarding path objects that led to this PEP can be found >>>>> in multiple threads on the python-ideas mailing list archive >>>>> [#python-ideas-archive]_ for the months of March and April 2016 and on >>>>> the python-dev mailing list archives [#python-dev-archive]_ during >>>>> April 2016. >>>>> >>>>> >>>>> Proposal >>>>> ======== >>>>> >>>>> This proposal is split into two parts. One part is the proposal of a >>>>> protocol for objects to declare and provide support for exposing a >>>>> file system path representation. The other part deals with changes to >>>>> Python's standard library to support the new protocol. These changes >>>>> will also lead to the pathlib module dropping its provisional status. >>>>> >>>>> Protocol >>>>> -------- >>>>> >>>>> The following abstract base class defines the protocol for an object >>>>> to be considered a path object:: >>>>> >>>>> import abc >>>>> import typing as t >>>>> >>>>> >>>>> class PathLike(abc.ABC): >>>>> >>>>> """Abstract base class for implementing the file system path >>>>> protocol.""" >>>>> >>>>> @abc.abstractmethod >>>>> def __fspath__(self) -> t.Union[str, bytes]: >>>>> """Return the file system path representation of the >>>>> object.""" >>>>> raise NotImplementedError >>>>> >>>>> >>>>> Objects representing file system paths will implement the >>>>> ``__fspath__()`` method which will return the ``str`` or ``bytes`` >>>>> representation of the path. The ``str`` representation is the >>>>> preferred low-level path representation as it is human-readable and >>>>> what people historically represent paths as. >>>>> >>>>> >>>>> Standard library changes >>>>> ------------------------ >>>>> >>>>> It is expected that most APIs in Python's standard library that >>>>> currently accept a file system path will be updated appropriately to >>>>> accept path objects (whether that requires code or simply an update >>>>> to documentation will vary). The modules mentioned below, though, >>>>> deserve specific details as they have either fundamental changes that >>>>> empower the ability to use path objects, or entail additions/removal >>>>> of APIs. >>>>> >>>>> >>>>> builtins >>>>> '''''''' >>>>> >>>>> ``open()`` [#builtins-open]_ will be updated to accept path objects as >>>>> well as continue to accept ``str`` and ``bytes``. >>>>> >>>>> >>>>> os >>>>> ''' >>>>> >>>>> The ``fspath()`` function will be added with the following semantics:: >>>>> >>>>> import typing as t >>>>> >>>>> >>>>> def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, >>>>> bytes]: >>>>> """Return the string representation of the path. >>>>> >>>>> If str or bytes is passed in, it is returned unchanged. >>>>> """ >>>>> if isinstance(path, (str, bytes)): >>>>> return path >>>>> >>>>> # Work from the object's type to match method resolution of >>>>> other magic >>>>> # methods. >>>>> path_type = type(path) >>>>> try: >>>>> return path_type.__fspath__(path) >>>>> except AttributeError: >>>>> if hasattr(path_type, '__fspath__'): >>>>> raise >>>>> >>>>> raise TypeError("expected str, bytes or os.PathLike >>>>> object, not " >>>>> + path_type.__name__) >>>>> >>>>> The ``os.fsencode()`` [#os-fsencode]_ and >>>>> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept >>>>> path objects. As both functions coerce their arguments to >>>>> ``bytes`` and ``str``, respectively, they will be updated to call >>>>> ``__fspath__()`` if present to convert the path object to a ``str`` or >>>>> ``bytes`` representation, and then perform their appropriate >>>>> coercion operations as if the return value from ``__fspath__()`` had >>>>> been the original argument to the coercion function in question. >>>>> >>>>> The addition of ``os.fspath()``, the updates to >>>>> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of >>>>> ``pathlib.PurePath`` provide the semantics necessary to >>>>> get the path representation one prefers. For a path object, >>>>> ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or >>>>> ``bytes`` representation without any coersion, then ``os.fspath()`` >>>>> can be used. If a ``str`` is desired and the encoding of ``bytes`` >>>>> should be assumed to be the default file system encoding, then >>>>> ``os.fsdecode()`` should be used. If a ``bytes`` representation is >>>>> desired and any strings should be encoded using the default file >>>>> system encoding, then ``os.fsencode()`` is used. This PEP recommends >>>>> using path objects when possible and falling back to string paths as >>>>> necessary and using ``bytes`` as a last resort. >>>>> >>>>> Another way to view this is as a hierarchy of file system path >>>>> representations (highest- to lowest-level): path ? str ? bytes. The >>>>> functions and classes under discussion can all accept objects on the >>>>> same level of the hierarchy, but they vary in whether they promote or >>>>> demote objects to another level. The ``pathlib.PurePath`` class can >>>>> promote a ``str`` to a path object. The ``os.fspath()`` function can >>>>> demote a path object to a ``str`` or ``bytes`` instance, depending >>>>> on what ``__fspath__()`` returns. >>>>> The ``os.fsdecode()`` function will demote a path object to >>>>> a string or promote a ``bytes`` object to a ``str``. The >>>>> ``os.fsencode()`` function will demote a path or string object to >>>>> ``bytes``. There is no function that provides a way to demote a path >>>>> object directly to ``bytes`` while bypassing string demotion. >>>>> >>>>> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` >>>>> method. It will return the same value as currently found on the >>>>> ``path`` attribute of ``DirEntry`` instances. >>>>> >>>>> The Protocol_ ABC will be added to the ``os`` module under the name >>>>> ``os.PathLike``. >>>>> >>>>> >>>>> os.path >>>>> ''''''' >>>>> >>>>> The various path-manipulation functions of ``os.path`` [#os-path]_ >>>>> will be updated to accept path objects. For polymorphic functions that >>>>> accept both bytes and strings, they will be updated to simply use >>>>> ``os.fspath()``. >>>>> >>>>> During the discussions leading up to this PEP it was suggested that >>>>> ``os.path`` not be updated using an "explicit is better than implicit" >>>>> argument. The thinking was that since ``__fspath__()`` is polymorphic >>>>> itself it may be better to have code working with ``os.path`` extract >>>>> the path representation from path objects explicitly. There is also >>>>> the consideration that adding support this deep into the low-level OS >>>>> APIs will lead to code magically supporting path objects without >>>>> requiring any documentation updated, leading to potential complaints >>>>> when it doesn't work, unbeknownst to the project author. >>>>> >>>>> But it is the view of this PEP that "practicality beats purity" in >>>>> this instance. To help facilitate the transition to supporting path >>>>> objects, it is better to make the transition as easy as possible than >>>>> to worry about unexpected/undocumented duck typing support for >>>>> path objects by projects. >>>>> >>>>> There has also been the suggestion that ``os.path`` functions could be >>>>> used in a tight loop and the overhead of checking or calling >>>>> ``__fspath__()`` would be too costly. In this scenario only >>>>> path-consuming APIs would be directly updated and path-manipulating >>>>> APIs like the ones in ``os.path`` would go unmodified. This would >>>>> require library authors to update their code to support path objects >>>>> if they performed any path manipulations, but if the library code >>>>> passed the path straight through then the library wouldn't need to be >>>>> updated. It is the view of this PEP and Guido, though, that this is an >>>>> unnecessary worry and that performance will still be acceptable. >>>>> >>>>> >>>>> pathlib >>>>> ''''''' >>>>> >>>>> The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be >>>>> updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path`` >>>>> will continue to not accept ``bytes`` path representations, and so if >>>>> ``__fspath__()`` returns ``bytes`` it will raise an exception. >>>>> >>>>> The ``path`` attribute will be removed as this PEP makes it >>>>> redundant (it has not been included in any released version of Python >>>>> and so is not a backwards-compatibility concern). >>>>> >>>>> >>>>> C API >>>>> ''''' >>>>> >>>>> The C API will gain an equivalent function to ``os.fspath()``:: >>>>> >>>>> /* >>>>> Return the file system path of the object. >>>>> >>>>> If the object is str or bytes, then allow it to pass through >>>>> with >>>>> an incremented refcount. If the object defines __fspath__(), >>>>> then >>>>> return the result of that method. All other types raise a >>>>> TypeError. >>>>> */ >>>>> PyObject * >>>>> PyOS_FSPath(PyObject *path) >>>>> { >>>>> if (PyUnicode_Check(path) || PyBytes_Check(path)) { >>>>> Py_INCREF(path); >>>>> return path; >>>>> } >>>>> >>>>> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { >>>>> return PyObject_CallMethodObjArgs(path->ob_type, >>>>> "__fspath__", path, >>>>> NULL); >>>>> } >>>>> >>>>> return PyErr_Format(PyExc_TypeError, >>>>> "expected a str, bytes, or os.PathLike >>>>> object, not %S", >>>>> path->ob_type); >>>>> } >>>>> >>>>> >>>>> >>>>> Backwards compatibility >>>>> ======================= >>>>> >>>>> There are no explicit backwards-compatibility concerns. Unless an >>>>> object incidentally already defines a ``__fspath__()`` method there is >>>>> no reason to expect the pre-existing code to break or expect to have >>>>> its semantics implicitly changed. >>>>> >>>>> Libraries wishing to support path objects and a version of Python >>>>> prior to Python 3.6 and the existence of ``os.fspath()`` can use the >>>>> idiom of >>>>> ``path.__fspath__() if hasattr(path, "__fspath__") else path``. >>>>> >>>>> >>>>> Implementation >>>>> ============== >>>>> >>>>> This is the task list for what this PEP proposes: >>>>> >>>>> #. Remove the ``path`` attribute from pathlib >>>>> #. Remove the provisional status of pathlib >>>>> #. Add ``os.PathLike`` >>>>> #. Add ``os.fspath()`` >>>>> #. Add ``PyOS_FSPath()`` >>>>> #. Update ``os.fsencode()`` >>>>> #. Update ``os.fsdecode()`` >>>>> #. Update ``pathlib.PurePath`` and ``pathlib.Path`` >>>>> #. Update ``builtins.open()`` >>>>> #. Update ``os.DirEntry`` >>>>> #. Update ``os.path`` >>>>> #. Add a glossary entry for "path-like" >>>>> >>>>> >>>>> Rejected Ideas >>>>> ============== >>>>> >>>>> Other names for the protocol's method >>>>> ------------------------------------- >>>>> >>>>> Various names were proposed during discussions leading to this PEP, >>>>> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In >>>>> the end people seemed to gravitate towards ``__fspath__`` for being >>>>> unambiguous without being unnecessarily long. >>>>> >>>>> >>>>> Separate str/bytes methods >>>>> -------------------------- >>>>> >>>>> At one point it was suggested that ``__fspath__()`` only return >>>>> strings and another method named ``__fspathb__()`` be introduced to >>>>> return bytes. The thinking is that by making ``__fspath__()`` not be >>>>> polymorphic it could make dealing with the potential string or bytes >>>>> representations easier. But the general consensus was that returning >>>>> bytes will more than likely be rare and that the various functions in >>>>> the os module are the better abstraction to promote over direct >>>>> calls to ``__fspath__()``. >>>>> >>>>> >>>>> Providing a ``path`` attribute >>>>> ------------------------------ >>>>> >>>>> To help deal with the issue of ``pathlib.PurePath`` not inheriting >>>>> from ``str``, originally it was proposed to introduce a ``path`` >>>>> attribute to mirror what ``os.DirEntry`` provides. In the end, >>>>> though, it was determined that a protocol would provide the same >>>>> result while not directly exposing an API that most people will never >>>>> need to interact with directly. >>>>> >>>>> >>>>> Have ``__fspath__()`` only return strings >>>>> ------------------------------------------ >>>>> >>>>> Much of the discussion that led to this PEP revolved around whether >>>>> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as >>>>> ``str`` or only return ``str``. The general sentiment for this view >>>>> was that ``bytes`` are difficult to work with due to their >>>>> inherent lack of information about their encoding and PEP 383 makes >>>>> it possible to represent all file system paths using ``str`` with the >>>>> ``surrogateescape`` handler. Thus, it would be better to forcibly >>>>> promote the use of ``str`` as the low-level path representation for >>>>> high-level path objects. >>>>> >>>>> In the end, it was decided that using ``bytes`` to represent paths is >>>>> simply not going to go away and thus they should be supported to some >>>>> degree. The hope is that people will gravitate towards path objects >>>>> like pathlib and that will move people away from operating directly >>>>> with ``bytes``. >>>>> >>>>> >>>>> A generic string encoding mechanism >>>>> ----------------------------------- >>>>> >>>>> At one point there was a discussion of developing a generic mechanism >>>>> to extract a string representation of an object that had semantic >>>>> meaning (``__str__()`` does not necessarily return anything of >>>>> semantic significance beyond what may be helpful for debugging). In >>>>> the end, it was deemed to lack a motivating need beyond the one this >>>>> PEP is trying to solve in a specific fashion. >>>>> >>>>> >>>>> Have __fspath__ be an attribute >>>>> ------------------------------- >>>>> >>>>> It was briefly considered to have ``__fspath__`` be an attribute >>>>> instead of a method. This was rejected for two reasons. One, >>>>> historically protocols have been implemented as "magic methods" and >>>>> not "magic methods and attributes". Two, there is no guarantee that >>>>> the lower-level representation of a path object will be pre-computed, >>>>> potentially misleading users that there was no expensive computation >>>>> behind the scenes in case the attribute was implemented as a property. >>>>> >>>>> This also indirectly ties into the idea of introducing a ``path`` >>>>> attribute to accomplish the same thing. This idea has an added issue, >>>>> though, of accidentally having any object with a ``path`` attribute >>>>> meet the protocol's duck typing. Introducing a new magic method for >>>>> the protocol helpfully avoids any accidental opting into the protocol. >>>>> >>>>> >>>>> Provide specific type hinting support >>>>> ------------------------------------- >>>>> >>>>> There was some consideration to provdinga generic ``typing.PathLike`` >>>>> class which would allow for e.g. ``typing.PathLike[str]`` to specify >>>>> a type hint for a path object which returned a string representation. >>>>> While potentially beneficial, the usefulness was deemed too small to >>>>> bother adding the type hint class. >>>>> >>>>> This also removed any desire to have a class in the ``typing`` module >>>>> which represented the union of all acceptable path-representing types >>>>> as that can be represented with >>>>> ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope >>>>> is users will slowly gravitate to path objects only. >>>>> >>>>> >>>>> Provide ``os.fspathb()`` >>>>> ------------------------ >>>>> >>>>> It was suggested that to mirror the structure of e.g. >>>>> ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return >>>>> ``str`` and that another function named ``os.fspathb()`` be >>>>> introduced that only returned ``bytes``. This was rejected as the >>>>> purposes of the ``*b()`` functions are tied to querying the file >>>>> system where there is a need to get the raw bytes back. As this PEP >>>>> does not work directly with data on a file system (but which *may* >>>>> be), the view was taken this distinction is unnecessary. It's also >>>>> believed that the need for only bytes will not be common enough to >>>>> need to support in such a specific manner as ``os.fsencode()`` will >>>>> provide similar functionality. >>>>> >>>>> >>>>> Call ``__fspath__()`` off of the instance >>>>> ----------------------------------------- >>>>> >>>>> An earlier draft of this PEP had ``os.fspath()`` calling >>>>> ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The >>>>> changed to be consistent with how other magic methods in Python are >>>>> resolved. >>>>> >>>>> >>>>> Acknowledgements >>>>> ================ >>>>> >>>>> Thanks to everyone who participated in the various discussions related >>>>> to this PEP that spanned both python-ideas and python-dev. Special >>>>> thanks to Stephen Turnbull for direct feedback on early drafts of this >>>>> PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not >>>>> only feedback on early drafts of this PEP but also helping to drive >>>>> the overall discussion on this topic across the two mailing lists. >>>>> >>>>> >>>>> References >>>>> ========== >>>>> >>>>> .. [#python-ideas-archive] The python-ideas mailing list archive >>>>> (https://mail.python.org/pipermail/python-ideas/) >>>>> >>>>> .. [#python-dev-archive] The python-dev mailing list archive >>>>> (https://mail.python.org/pipermail/python-dev/) >>>>> >>>>> .. [#libc-open] ``open()`` documention for the C standard library >>>>> ( >>>>> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html >>>>> ) >>>>> >>>>> .. [#pathlib] The ``pathlib`` module >>>>> (https://docs.python.org/3/library/pathlib.html#module-pathlib) >>>>> >>>>> .. [#builtins-open] The ``builtins.open()`` function >>>>> (https://docs.python.org/3/library/functions.html#open) >>>>> >>>>> .. [#os-fsencode] The ``os.fsencode()`` function >>>>> (https://docs.python.org/3/library/os.html#os.fsencode) >>>>> >>>>> .. [#os-fsdecode] The ``os.fsdecode()`` function >>>>> (https://docs.python.org/3/library/os.html#os.fsdecode) >>>>> >>>>> .. [#os-direntry] The ``os.DirEntry`` class >>>>> (https://docs.python.org/3/library/os.html#os.DirEntry) >>>>> >>>>> .. [#os-path] The ``os.path`` module >>>>> (https://docs.python.org/3/library/os.path.html#module-os.path) >>>>> >>>>> >>>>> Copyright >>>>> ========= >>>>> >>>>> This document has been placed in the public domain. >>>>> >>>>> >>>>> .. >>>>> Local Variables: >>>>> mode: indented-text >>>>> indent-tabs-mode: nil >>>>> sentence-end-double-space: t >>>>> fill-column: 70 >>>>> coding: utf-8 >>>>> End: >>>>> >>>>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon May 16 20:01:26 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 16 May 2016 17:01:26 -0700 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: Message-ID: <573A5F56.3030702@stoneleaf.us> On 05/16/2016 02:32 PM, Brett Cannon wrote: > PEP 519 is now marked as accepted! Two PEPs accepted in one day (Nick > approved 518 earlier)! Excellent! :) > If pathlib doesn't get updated in 3.4, 3.5, and 3.6 before PyCon it will > be the first thing I do at the sprints so that it isn't an issue for the > 3.5.2 release. Isn't 3.4 now in security-fix mode? -- ~Ethan~ From brett at python.org Mon May 16 20:57:42 2016 From: brett at python.org (Brett Cannon) Date: Tue, 17 May 2016 00:57:42 +0000 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: <573A5F56.3030702@stoneleaf.us> References: <573A5F56.3030702@stoneleaf.us> Message-ID: On Mon, May 16, 2016, 19:02 Ethan Furman wrote: > On 05/16/2016 02:32 PM, Brett Cannon wrote: > > > PEP 519 is now marked as accepted! Two PEPs accepted in one day (Nick > > approved 518 earlier)! > > Excellent! :) > > > If pathlib doesn't get updated in 3.4, 3.5, and 3.6 before PyCon it will > > be the first thing I do at the sprints so that it isn't an issue for the > > 3.5.2 release. > > Isn't 3.4 now in security-fix mode? > Yes, but because of pathlib's provisional status, Guido backported the path attribute from 3.4 forward. -brett > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Tue May 17 01:12:22 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Tue, 17 May 2016 07:12:22 +0200 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: <56A90B97.7090001@gmail.com> Message-ID: 2016-05-16 17:55 GMT+02:00 Meador Inge : > On Sun, May 15, 2016 at 2:23 AM, Cesare Di Mauro < > cesare.di.mauro at gmail.com> wrote: > > >> Just one thing that comes to my mind: is the stack depth calculation >> routine changed? It was suboptimal, and calculating a better number >> decreases stack allocation, and increases the frame usage. >> > > This is still a problem and came up again recently: > > http://bugs.python.org/issue26549 > > -- Meador > I saw the last two comments of the issues: this is what I was talking about (in particular the issue opened by Armin applies). However there's another case where the situation is even worse. Let me show a small reproducer: def test(self): for i in range(self.count): with self: pass The stack size reported by Python 2.7.11: >>> test.__code__.co_stacksize 6 Adding another with statement: >>> test.__code__.co_stacksize 7 But unfortunately with Python 3.5.1 the problematic is much worse: >>> test.__code__.co_stacksize 10 >>> test.__code__.co_stacksize 17 Here the situation is exacerbated by the fact that the WITH_CLEANUP instruction of Python 2.x was split into two (WITH_CLEANUP_START and WITH_CLEANUP_FINISH) in some Python 3 release. I don't know why two different instructions were introduced, but IMO it's better to have one instruction which handles all code finalization of the with statement, at least in this case. If there are other scenarios where two different instructions are needed, then ad-hoc instructions like those can be used. Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue May 17 01:28:41 2016 From: larry at hastings.org (Larry Hastings) Date: Tue, 17 May 2016 07:28:41 +0200 Subject: [Python-Dev] File system path PEP, 3rd draft In-Reply-To: References: <573A5F56.3030702@stoneleaf.us> Message-ID: <573AAC09.3040206@hastings.org> On 05/17/2016 02:57 AM, Brett Cannon wrote: > On Mon, May 16, 2016, 19:02 Ethan Furman > wrote: > > Isn't 3.4 now in security-fix mode? > > > Yes, but because of pathlib's provisional status, Guido backported the > path attribute from 3.4 forward. I think this is one reason why Raymond Hettinger says "Python is Guido's language, he just lets us use it." //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jan.jockusch at perfact.de Tue May 17 08:35:58 2016 From: jan.jockusch at perfact.de (=?UTF-8?Q?Dr._J=c3=a1n_Jockusch?=) Date: Tue, 17 May 2016 14:35:58 +0200 Subject: [Python-Dev] Python2.7.11+ as in Debian testing and Ubuntu 16.04 LTS crashes with segfault Message-ID: <573B102E.5090805@perfact.de> Hello Python Developers, I would like to draw your attention to a segmentation fault bug which concerns the packaged versions of Python 2.7.11 as seen in Ubuntu's latest 16.04 release and in Debian testing. I have reported this bug in Ubuntu's tracker here: https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1580577 The report contains a script which can reproduce the error for both distributions' python2.7 packages. The plain python.org version 2.7.11 does not exhibit this problem. Matthias Klose, Debian maintainer for the package, has asked me to report this problem upstream, because otherwise we will invariably run into this regression in 2.7.12. I'd like to help in tracking down this bug, please advise. Thank you, - J?n Jockusch From zreed at fastmail.com Tue May 17 02:25:59 2016 From: zreed at fastmail.com (zreed at fastmail.com) Date: Tue, 17 May 2016 01:25:59 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% Message-ID: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com> In the project https://github.com/zachariahreed/byteasm I mentioned on the list earlier this month, I have a pass that to computes stack usage for a given sequence of bytecodes. It seems to be a fair bit more agressive than cpython. Maybe it's more generally useful. It's pure python rather than C though. From brett at python.org Tue May 17 11:42:01 2016 From: brett at python.org (Brett Cannon) Date: Tue, 17 May 2016 15:42:01 +0000 Subject: [Python-Dev] Python2.7.11+ as in Debian testing and Ubuntu 16.04 LTS crashes with segfault In-Reply-To: <573B102E.5090805@perfact.de> References: <573B102E.5090805@perfact.de> Message-ID: On Tue, 17 May 2016 at 08:15 Dr. J?n Jockusch wrote: > Hello Python Developers, > > I would like to draw your attention to a segmentation fault bug which > concerns the packaged versions of Python 2.7.11 as seen in Ubuntu's > latest 16.04 release and in Debian testing. > > > I have reported this bug in Ubuntu's tracker here: > > https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1580577 > > > The report contains a script which can reproduce the error for both > distributions' python2.7 packages. > > The plain python.org version 2.7.11 does not exhibit this problem. > > > > Matthias Klose, Debian maintainer for the package, has asked me to > report this problem upstream, because otherwise we will invariably run > into this regression in 2.7.12. > > I'd like to help in tracking down this bug, please advise. > Please file the bug on bugs.python.org so we don't lose track of this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at python.org Tue May 17 17:15:19 2016 From: nad at python.org (Ned Deily) Date: Tue, 17 May 2016 17:15:19 -0400 Subject: [Python-Dev] Python 3.6.0a1 is now available Message-ID: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org> On behalf of the Python development community and the Python 3.6 release team, I'm happy to announce the availability of Python 3.6.0a1. 3.6.0a1 is the first of four planned alpha releases of Python 3.6, the next major release of Python. During the alpha phase, Python 3.6 remains under heavy development: additional features will be added and existing features may be modified or deleted. Please keep in mind that this is a preview release and its use is not recommended for production environments. You can find Python 3.6.0a1 here: https://www.python.org/downloads/release/python-360a1/ The next release of Python 3.6 will be 3.6.0a2, currently scheduled for 2016-06-13. Enjoy! --Ned -- Ned Deily nad at python.org -- [] From dholth at gmail.com Wed May 18 09:20:41 2016 From: dholth at gmail.com (Daniel Holth) Date: Wed, 18 May 2016 13:20:41 +0000 Subject: [Python-Dev] Adding a threadlocal to the Python interpreter Message-ID: I would like to take another stab at adding a threadlocal "str(bytes) raises an exception" to the Python interpreter, but I had a very hard time understanding both how to add a threadlocal value to either the interpreter state or the threadlocal dict that is part of that state, and then how to access the same value from both Python and CPython code. The structs were there but it was just hard to understand. Can someone explain it to me? Thanks, Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Wed May 18 12:35:30 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Wed, 18 May 2016 18:35:30 +0200 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com> References: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com> Message-ID: 2016-05-17 8:25 GMT+02:00 : > In the project https://github.com/zachariahreed/byteasm I mentioned on > the list earlier this month, I have a pass that to computes stack usage > for a given sequence of bytecodes. It seems to be a fair bit more > agressive than cpython. Maybe it's more generally useful. It's pure > python rather than C though. > IMO it's too big, resource hungry, and slower, even if you convert it in C. If you take a look at the current stackdepth_walk function which CPython uses, it's much smaller (not even 50 lines in simple C code) and quite efficient. Currently the problem is that it doesn't return the maximum depth of the tree, but it updates the intermediate/current maximum, and *then* it uses it for the subsequent calculations. So, the depth artificially grows, like in the reported cases. It doesn't require a complete rewrite, but spending some time for fine-tuning it. Regards Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From zreed at fastmail.com Wed May 18 12:50:09 2016 From: zreed at fastmail.com (zreed at fastmail.com) Date: Wed, 18 May 2016 11:50:09 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com> Message-ID: <1463590209.997754.611701105.71BF1B08@webmail.messagingengine.com> Your criticisms may very well be true. IIRC though, I wrote that pass because what was available was not general enough. The?stackdepth_walk function made assumptions that, while true of code generated by the current cpython frontend, were not universally true. If a goal is to move this calculation after any bytecode optimization, something along these lines seems like it will eventually be necessary. Anyway, just offering things already written. If you don't feel it's useful, no worries. On Wed, May 18, 2016, at 11:35 AM, Cesare Di Mauro wrote: > 2016-05-17 8:25 GMT+02:00 : >> In the project https://github.com/zachariahreed/byteasm I >> mentioned on >> the list earlier this month, I have a pass that to computes >> stack usage >> for a given sequence of bytecodes. It seems to be a fair bit more >> agressive than cpython. Maybe it's more generally useful. It's pure >> python rather than C though. >> > IMO it's too big, resource hungry, and slower, even if you convert > it in C. > > If you take a look at the current stackdepth_walk function which > CPython uses, it's much smaller (not even 50 lines in simple C code) > and quite efficient. > > Currently the problem is that it doesn't return the maximum depth of > the tree, but it updates the intermediate/current maximum, and *then* > it uses it for the subsequent calculations. So, the depth artificially > grows, like in the reported cases. > > It doesn't require a complete rewrite, but spending some time for fine- > tuning it. > > Regards > Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Wed May 18 14:06:08 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Wed, 18 May 2016 20:06:08 +0200 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: <1463590209.997754.611701105.71BF1B08@webmail.messagingengine.com> References: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com> <1463590209.997754.611701105.71BF1B08@webmail.messagingengine.com> Message-ID: If you feel like I've attacked you, I apologize: it wasn't my intention. Please, don't get it personal: I only reported my honest opinion, albeit after a re-read it looks too rude, and I'm sorry for that. Regarding the post-bytecode optimization issues, they are mainly represented by the constant folding code, which is still in the peephole stage. Once it's moved to the proper place (ASDL/AST), then such kind of issues with the stack calculations disappear, whereas the remaining ones can be addressed by a fix of the current stackdepth_walk function. And just to be clear, I've nothing against your code. I simply think that, due to my experience, it doesn't fit in CPython. Regards Cesare 2016-05-18 18:50 GMT+02:00 : > Your criticisms may very well be true. IIRC though, I wrote that pass > because what was available was not general enough. The stackdepth_walk > function made assumptions that, while true of code generated by the current > cpython frontend, were not universally true. If a goal is to move this > calculation after any bytecode optimization, something along these lines > seems like it will eventually be necessary. > > Anyway, just offering things already written. If you don't feel it's > useful, no worries. > > > On Wed, May 18, 2016, at 11:35 AM, Cesare Di Mauro wrote: > > 2016-05-17 8:25 GMT+02:00 : > > In the project https://github.com/zachariahreed/byteasm I mentioned on > the list earlier this month, I have a pass that to computes stack usage > for a given sequence of bytecodes. It seems to be a fair bit more > agressive than cpython. Maybe it's more generally useful. It's pure > python rather than C though. > > > IMO it's too big, resource hungry, and slower, even if you convert it in C. > > If you take a look at the current stackdepth_walk function which CPython > uses, it's much smaller (not even 50 lines in simple C code) and quite > efficient. > > Currently the problem is that it doesn't return the maximum depth of the > tree, but it updates the intermediate/current maximum, and *then* it uses > it for the subsequent calculations. So, the depth artificially grows, like > in the reported cases. > > It doesn't require a complete rewrite, but spending some time for > fine-tuning it. > > Regards > Cesare > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas at arctrix.com Wed May 18 14:52:40 2016 From: nas at arctrix.com (Neil Schemenauer) Date: Wed, 18 May 2016 18:52:40 +0000 (UTC) Subject: [Python-Dev] PyGC_Collect ignores state of `enabled` References: <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com> Message-ID: Benjamin Peterson wrote: > Adding PyGC_CollectIfEnabled() and calling it in Py_Finalize is probably > fine. I don't think the contract of PyGC_Collect itself (or gc.collect() > for that matter) should be changed. You might want to disable GC but > invoke it yourself. Yes, that sounds okay to me. I poked around at the calls to PyGC_Collect() and _PyGC_CollectNoFail(). The cyclic garbage collector gets invoked at least three times during shutdown. Once by Py_FinalizeEx() and two times by PyImport_Cleanup(). That seems a bit exessively expensive to me. The collection time can be significant for programs with a lot of "container" objects in memory. The whole finalize/shutdown logic of the CPython interpreter could badly use some improvement. Currently it is a set of ugly hacks piled on top of each other. Now that we have PEP 3121, Extension Module Initialization and Finalization https://www.python.org/dev/peps/pep-3121/ we should be able to cleanup this mess. PyImport_Cleanup() is the main area of trouble. I don't think we should not be clearing sys.modules and we should certainly not be clearing module dicts. If there is some whippersnapper out there who wants to get their hands dirty with Python internals, fixing PyImport_Cleanup() would be a juicy project. Neil From ethan at stoneleaf.us Wed May 18 15:04:23 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 18 May 2016 12:04:23 -0700 Subject: [Python-Dev] PyGC_Collect ignores state of `enabled` In-Reply-To: References: <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com> Message-ID: <573CBCB7.7000202@stoneleaf.us> On 05/18/2016 11:52 AM, Neil Schemenauer wrote: > Benjamin Peterson wrote: >> Adding PyGC_CollectIfEnabled() and calling it in Py_Finalize is probably >> fine. I don't think the contract of PyGC_Collect itself (or gc.collect() >> for that matter) should be changed. You might want to disable GC but >> invoke it yourself. > > Yes, that sounds okay to me. > > I poked around at the calls to PyGC_Collect() and > _PyGC_CollectNoFail(). The cyclic garbage collector gets invoked at > least three times during shutdown. Once by Py_FinalizeEx() and two > times by PyImport_Cleanup(). That seems a bit exessively expensive > to me. The collection time can be significant for programs with a > lot of "container" objects in memory. > > The whole finalize/shutdown logic of the CPython interpreter could > badly use some improvement. Currently it is a set of ugly hacks > piled on top of each other. Now that we have PEP 3121, > > Extension Module Initialization and Finalization > https://www.python.org/dev/peps/pep-3121/ > > we should be able to cleanup this mess. PyImport_Cleanup() is the > main area of trouble. I don't think we should not be clearing > sys.modules and we should certainly not be clearing module dicts. > > If there is some whippersnapper out there who wants to get their > hands dirty with Python internals, fixing PyImport_Cleanup() would > be a juicy project. Is there an issue filed for it? -- ~Ethan~ From zreed at fastmail.com Wed May 18 16:55:03 2016 From: zreed at fastmail.com (zreed at fastmail.com) Date: Wed, 18 May 2016 15:55:03 -0500 Subject: [Python-Dev] Speeding up CPython 5-10% In-Reply-To: References: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com> <1463590209.997754.611701105.71BF1B08@webmail.messagingengine.com> Message-ID: <1463604903.1810445.611934721.2D819177@webmail.messagingengine.com> No problem, I did not think you were attacking me or find your response rude. On Wed, May 18, 2016, at 01:06 PM, Cesare Di Mauro wrote: > If you feel like I've attacked you, I apologize: it wasn't my > intention. Please, don't get it personal: I only reported my honest > opinion, albeit after a re-read it looks too rude, and I'm sorry > for that. > > Regarding the post-bytecode optimization issues, they are mainly > represented by the constant folding code, which is still in the > peephole stage. Once it's moved to the proper place (ASDL/AST), then > such kind of issues with the stack calculations disappear, whereas the > remaining ones can be addressed by a fix of the current > stackdepth_walk function. > > And just to be clear, I've nothing against your code. I simply think > that, due to my experience, it doesn't fit in CPython. > > Regards > Cesare > > 2016-05-18 18:50 GMT+02:00 : >> __ >> Your criticisms may very well be true. IIRC though, I wrote that pass >> because what was available was not general enough. The >> stackdepth_walk function made assumptions that, while true of code >> generated by the current cpython frontend, were not universally true. >> If a goal is to move this calculation after any bytecode >> optimization, something along these lines seems like it will >> eventually be necessary. >> >> Anyway, just offering things already written. If you don't feel it's >> useful, no worries. >> >> >> On Wed, May 18, 2016, at 11:35 AM, Cesare Di Mauro wrote: >>> 2016-05-17 8:25 GMT+02:00 : >>>> In the project https://github.com/zachariahreed/byteasm I >>>> mentioned on >>>> the list earlier this month, I have a pass that to computes stack >>>> usage >>>> for a given sequence of bytecodes. It seems to be a fair bit more >>>> agressive than cpython. Maybe it's more generally useful. It's pure >>>> python rather than C though. >>>> >>> IMO it's too big, resource hungry, and slower, even if you convert >>> it in C. >>> >>> If you take a look at the current stackdepth_walk function which >>> CPython uses, it's much smaller (not even 50 lines in simple C code) >>> and quite efficient. >>> >>> Currently the problem is that it doesn't return the maximum depth of >>> the tree, but it updates the intermediate/current maximum, and >>> *then* it uses it for the subsequent calculations. So, the depth >>> artificially grows, like in the reported cases. >>> >>> It doesn't require a complete rewrite, but spending some time for >>> fine-tuning it. >>> >>> Regards >>> Cesare >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Wed May 18 17:36:42 2016 From: christian at python.org (Christian Heimes) Date: Wed, 18 May 2016 23:36:42 +0200 Subject: [Python-Dev] Adding a threadlocal to the Python interpreter In-Reply-To: References: Message-ID: On 2016-05-18 15:20, Daniel Holth wrote: > I would like to take another stab at adding a threadlocal "str(bytes) > raises an exception" to the Python interpreter, but I had a very hard > time understanding both how to add a threadlocal value to either the > interpreter state or the threadlocal dict that is part of that state, > and then how to access the same value from both Python and CPython code. > The structs were there but it was just hard to understand. Can someone > explain it to me? Python has a two important states related to threads. The PyInterpreterState contains the state of an interpreter instance: sys module, loaded modules and a couple of additional settings. Usually there is just one interpreter state in a Python process. Additional interpreter states are used to implement subinterpreters. Each C thread, that wants to run Python code, must have a PyThreadState. The thread state contains a reference to a PyInterpreterState. Each PyThreadState has a PyObject *dict member. You can stick Python objects into the dict. The interpreter cleans up the dict when it reaps a thread. How performance critical is your code? Does the interpreter have to check the value of the thread local frequently? In that case you should add a new member to typedef struct _ts PyThreadState in pystate.h right before /* XXX signal handlers should also be here */. Otherwise you can simply use PyThreadState_GetDict(). It returns a Python dict object that is local to the current thread. You can simply use a fixed key like in Modules/_decimal/_decimal.c. Christian From gvanrossum at gmail.com Wed May 18 18:39:02 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 18 May 2016 15:39:02 -0700 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: References: Message-ID: FYI, a few people gave useful feedback on my draft text, and I've now pushed it to PEP 484. All new text is one section: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects Next I'm going to implement it. Subscribe to this issue if you want to follow along: https://github.com/python/typing/issues/107 -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Wed May 18 22:11:51 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 May 2016 12:11:51 +1000 Subject: [Python-Dev] PyGC_Collect ignores state of `enabled` In-Reply-To: <573CBCB7.7000202@stoneleaf.us> References: <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com> <573CBCB7.7000202@stoneleaf.us> Message-ID: On 19 May 2016 at 05:04, Ethan Furman wrote: > On 05/18/2016 11:52 AM, Neil Schemenauer wrote: >> The whole finalize/shutdown logic of the CPython interpreter could >> badly use some improvement. Currently it is a set of ugly hacks >> piled on top of each other. Now that we have PEP 3121, >> >> Extension Module Initialization and Finalization >> https://www.python.org/dev/peps/pep-3121/ >> >> we should be able to cleanup this mess. PEP 3121 is insufficient, since a lot of extension modules can't (or at least haven't) adopted it in practice. https://www.python.org/dev/peps/pep-0489/ has some more background on that (since it was the first step towards tackling the problem in a different way that extension module authors may be more likely to actually adopt) >> PyImport_Cleanup() is the >> main area of trouble. I don't think we should not be clearing >> sys.modules and we should certainly not be clearing module dicts. >> >> If there is some whippersnapper out there who wants to get their >> hands dirty with Python internals, fixing PyImport_Cleanup() would >> be a juicy project. > > Is there an issue filed for it? It isn't really any one issue, since PyImport_Cleanup aims to tolerate misbehaving modules across multiple Py_Initialize/Finalize cycles within a single process, and hence tries as hard as it can to forcibly break reference cycles and clean up resource allocations. Switching it over to the suggested PyGC_CollectIfEnabled() API should be fine though - it will just need to be documented that calling Py_Initialize again in the same process is unsupported if you have the GC disabled during a previous call to Py_Finalize. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed May 18 22:30:27 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 May 2016 12:30:27 +1000 Subject: [Python-Dev] Adding a threadlocal to the Python interpreter In-Reply-To: References: Message-ID: On 18 May 2016 at 23:20, Daniel Holth wrote: > I would like to take another stab at adding a threadlocal "str(bytes) raises > an exception" to the Python interpreter, but I had a very hard time > understanding both how to add a threadlocal value to either the interpreter > state or the threadlocal dict that is part of that state, and then how to > access the same value from both Python and CPython code. The structs were > there but it was just hard to understand. Can someone explain it to me? Christian covered the C aspects of the API, while the general purpose Python aspects live in the threading module. However, the Python level thread-local API doesn't provide direct access to the thread state dict. Instead, it provides access to subdicts stored under per-object keys in that dict, keyed as "thread.local.": * Key definition in local_new: https://hg.python.org/cpython/file/tip/Modules/_threadmodule.c#l705 * Subdict creation in _ldict: https://hg.python.org/cpython/file/tip/Modules/_threadmodule.c#l810 Getting access to state stored that way from C is *possible*, but significantly less convenient than access the thread state directly. What that means is that any time we want to expose thread local state to both C and Python code, it will generally be the responsibility of the C code to both manage the key in the thread state dict (or the field in the thread state struct), and also to provide a Python API for access that state. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Thu May 19 06:34:22 2016 From: christian at python.org (Christian Heimes) Date: Thu, 19 May 2016 12:34:22 +0200 Subject: [Python-Dev] Adding a threadlocal to the Python interpreter In-Reply-To: References: Message-ID: On 2016-05-19 04:30, Nick Coghlan wrote: > On 18 May 2016 at 23:20, Daniel Holth wrote: >> I would like to take another stab at adding a threadlocal "str(bytes) raises >> an exception" to the Python interpreter, but I had a very hard time >> understanding both how to add a threadlocal value to either the interpreter >> state or the threadlocal dict that is part of that state, and then how to >> access the same value from both Python and CPython code. The structs were >> there but it was just hard to understand. Can someone explain it to me? > > Christian covered the C aspects of the API, while the general purpose > Python aspects live in the threading module. > > However, the Python level thread-local API doesn't provide direct > access to the thread state dict. Instead, it provides access to > subdicts stored under per-object keys in that dict, keyed as > "thread.local.": In case you wonder about subdicts, they are required to provide multiple thread local objects. Each thread local instance has its own key in each thread state dict. The approach enables thread local to have independent storage objects. Christian From srkunze at mail.de Fri May 20 06:56:24 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 20 May 2016 12:56:24 +0200 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: References: Message-ID: <573EED58.3040508@mail.de> On 15.05.2016 19:30, Guido van Rossum wrote: > Right. I still have no good intuition for what Type[BasicUser, > ProUser] would mean so I think you should be required to use the > Union, which is clear. Type[] should only allow one parameter. Type[A, B] reminds me of isinstance(obj, (A, B)). Sven From victor.stinner at gmail.com Fri May 20 07:53:04 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 20 May 2016 13:53:04 +0200 Subject: [Python-Dev] Buildbots now always run tests in subprocesses Message-ID: Hi, I modified regrtest to not ignore -j1 anymore (issue #25285). I also modified Tools/buildbot/test.bat to add the -j1 option (-j1 was already used on UNIX on "make buildbottest"). All buildbots will now run each test file in a fresh subprocess. It helps to restrict all kinds of side effects of unit tests to a single file. The drawback is that buildbots will be less efficient to detect side effects of unit tests. By the way, I also changed regrtest output in "multiprocess mode" (when -jN is used) to add "passed" to test names: --- $ ./python -m test -j1 ... Run tests in parallel using 1 child processes 0:00:00 [ 1/401] test_grammar passed 0:00:00 [ 2/401] test_opcodes passed 0:00:01 [ 3/401] test_dict passed --- When tests are run sequentially, the output is different: --- $ ./python -u -m test ... Run tests sequentially 0:00:00 [ 1/401] test_grammar 0:00:00 [ 2/401] test_opcodes 0:00:00 [ 3/401] test_dict ... --- In multiprocess mode, regrtest displays the name of the test which just completed. In sequential mode, it displays the name of the test which is currently running. Maybe the output of sequential mode can also be enhanced, but I made the minimum change to avoid breaking regrtest. Some parts of the code rely on the regrtest output format. I added many unit tests to test_regrtest, but not everything is tested yet. Victor From status at bugs.python.org Fri May 20 12:08:42 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 20 May 2016 18:08:42 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20160520160842.313B856A9C@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-05-13 - 2016-05-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5504 ( +3) closed 33322 (+52) total 38826 (+55) Open issues with patches: 2381 Issues opened (33) ================== #26741: subprocess.Popen should emit a ResourceWarning in destructor i http://bugs.python.org/issue26741 reopened by haypo #27015: subprocess.CalledProcessError's repr changes based on kwargs, http://bugs.python.org/issue27015 opened by Taywee #27016: BlockingIOError not raised inside function. http://bugs.python.org/issue27016 opened by pcunningham80 at gmail.com #27019: Reduce marshal stack depth for 2.7 on Windows debug build http://bugs.python.org/issue27019 opened by db3l #27020: os.writev() does not accept generators (as buffers argument) http://bugs.python.org/issue27020 opened by mmarkk #27021: It is not documented that os.writev() suffer from SC_IOV_MAX http://bugs.python.org/issue27021 opened by mmarkk #27022: expose sendmmsg() syscall in sockets API http://bugs.python.org/issue27022 opened by mmarkk #27024: IDLE shutdown glitch when started by import http://bugs.python.org/issue27024 opened by terry.reedy #27025: More human readable generated widget names http://bugs.python.org/issue27025 opened by serhiy.storchaka #27026: async/await keywords are missing from reference docs http://bugs.python.org/issue27026 opened by hongminhee #27027: add is_android in test.support to detect Android platform http://bugs.python.org/issue27027 opened by xdegaye #27029: Remove support of deprecated mode 'U' in zipfile http://bugs.python.org/issue27029 opened by serhiy.storchaka #27030: Remove deprecated re features http://bugs.python.org/issue27030 opened by serhiy.storchaka #27032: Remove deprecated html.parser.HTMLParser.unescape() http://bugs.python.org/issue27032 opened by serhiy.storchaka #27033: Change the decode_data default in smtpd to False http://bugs.python.org/issue27033 opened by serhiy.storchaka #27035: Cannot set exit code in atexit callback http://bugs.python.org/issue27035 opened by Melebius #27036: Mark up of references in different form http://bugs.python.org/issue27036 opened by serhiy.storchaka #27037: Universal newline support for zipFile.ZipExtFile.read() is not http://bugs.python.org/issue27037 opened by David Pitchford #27038: Make os.DirEntry exist http://bugs.python.org/issue27038 opened by brett.cannon #27043: Describe what ???inspect.cleandoc??? does to synopsis line. http://bugs.python.org/issue27043 opened by bignose #27048: distutils._msvccompiler._get_vc_env() fails with UnicodeDecode http://bugs.python.org/issue27048 opened by ebarry #27050: Demote run() below the high level APIs in subprocess docs http://bugs.python.org/issue27050 opened by ncoghlan #27051: Create PIP gui http://bugs.python.org/issue27051 opened by upendra-k14 #27052: Python2.7.11+ as in Debian testing and Ubuntu 16.04 LTS crashe http://bugs.python.org/issue27052 opened by J??n Jockusch #27054: Python installation problem: No module named 'encodings' http://bugs.python.org/issue27054 opened by lauer at wpi.edu #27060: Documentation of assertItemsEqual in unittest is VERY misleadi http://bugs.python.org/issue27060 opened by vitaly #27062: `inspect` doesn't have `__all__` http://bugs.python.org/issue27062 opened by leewz #27063: Some unittest loader tests are silently skipped by mistake http://bugs.python.org/issue27063 opened by serhiy.storchaka #27064: Make py.exe default to Python 3 when used interactively http://bugs.python.org/issue27064 opened by paul.moore #27066: SystemError if custom opener returns -1 http://bugs.python.org/issue27066 opened by barry #27067: Improve curses tests http://bugs.python.org/issue27067 opened by serhiy.storchaka #27068: Add a detach() method to subprocess.Popen http://bugs.python.org/issue27068 opened by haypo #27069: webbrowser creates zombi processes in the background mode http://bugs.python.org/issue27069 opened by haypo Most recent 15 issues with no replies (15) ========================================== #27067: Improve curses tests http://bugs.python.org/issue27067 #27063: Some unittest loader tests are silently skipped by mistake http://bugs.python.org/issue27063 #27060: Documentation of assertItemsEqual in unittest is VERY misleadi http://bugs.python.org/issue27060 #27051: Create PIP gui http://bugs.python.org/issue27051 #27043: Describe what ???inspect.cleandoc??? does to synopsis line. http://bugs.python.org/issue27043 #27038: Make os.DirEntry exist http://bugs.python.org/issue27038 #27032: Remove deprecated html.parser.HTMLParser.unescape() http://bugs.python.org/issue27032 #27030: Remove deprecated re features http://bugs.python.org/issue27030 #27029: Remove support of deprecated mode 'U' in zipfile http://bugs.python.org/issue27029 #27021: It is not documented that os.writev() suffer from SC_IOV_MAX http://bugs.python.org/issue27021 #27019: Reduce marshal stack depth for 2.7 on Windows debug build http://bugs.python.org/issue27019 #27006: C implementation of Decimal.from_float() bypasses __new__ and http://bugs.python.org/issue27006 #27002: Support different modes in posixpath.realpath() http://bugs.python.org/issue27002 #26997: Docs for pdb should note that __future__ magic doesn't work http://bugs.python.org/issue26997 #26990: file.tell affect decoding http://bugs.python.org/issue26990 Most recent 15 issues waiting for review (15) ============================================= #27067: Improve curses tests http://bugs.python.org/issue27067 #27066: SystemError if custom opener returns -1 http://bugs.python.org/issue27066 #27064: Make py.exe default to Python 3 when used interactively http://bugs.python.org/issue27064 #27063: Some unittest loader tests are silently skipped by mistake http://bugs.python.org/issue27063 #27048: distutils._msvccompiler._get_vc_env() fails with UnicodeDecode http://bugs.python.org/issue27048 #27036: Mark up of references in different form http://bugs.python.org/issue27036 #27033: Change the decode_data default in smtpd to False http://bugs.python.org/issue27033 #27032: Remove deprecated html.parser.HTMLParser.unescape() http://bugs.python.org/issue27032 #27030: Remove deprecated re features http://bugs.python.org/issue27030 #27029: Remove support of deprecated mode 'U' in zipfile http://bugs.python.org/issue27029 #27027: add is_android in test.support to detect Android platform http://bugs.python.org/issue27027 #27026: async/await keywords are missing from reference docs http://bugs.python.org/issue27026 #27025: More human readable generated widget names http://bugs.python.org/issue27025 #27024: IDLE shutdown glitch when started by import http://bugs.python.org/issue27024 #27019: Reduce marshal stack depth for 2.7 on Windows debug build http://bugs.python.org/issue27019 Top 10 most discussed issues (10) ================================= #26870: Unexpected call to readline's add_history in call_readline http://bugs.python.org/issue26870 11 msgs #27048: distutils._msvccompiler._get_vc_env() fails with UnicodeDecode http://bugs.python.org/issue27048 11 msgs #20371: datetime.datetime.replace bypasses a subclass's __new__ http://bugs.python.org/issue20371 9 msgs #27036: Mark up of references in different form http://bugs.python.org/issue27036 8 msgs #27054: Python installation problem: No module named 'encodings' http://bugs.python.org/issue27054 8 msgs #26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure http://bugs.python.org/issue26168 6 msgs #26331: Tokenizer: allow underscores for grouping in numeric literals http://bugs.python.org/issue26331 6 msgs #26369: unicode.decode and str.encode are unnecessarily confusing for http://bugs.python.org/issue26369 6 msgs #26814: [WIP] Add a new _PyObject_FastCall() function which avoids the http://bugs.python.org/issue26814 6 msgs #26839: Python 3.5 running on Linux kernel 3.17+ can block at startup http://bugs.python.org/issue26839 6 msgs Issues closed (48) ================== #3213: "pydoc -p" should listen to [::] if IPv6 is supported http://bugs.python.org/issue3213 closed by martin.panter #3948: readline steals sigwinch http://bugs.python.org/issue3948 closed by martin.panter #11980: zipfile.ZipFile.write should accept fp as argument http://bugs.python.org/issue11980 closed by serhiy.storchaka #14132: Redirect is not working correctly in urllib2 http://bugs.python.org/issue14132 closed by martin.panter #18410: Idle: test SearchDialog.py http://bugs.python.org/issue18410 closed by terry.reedy #20640: Idle: test configHelpSourceEdit http://bugs.python.org/issue20640 closed by terry.reedy #20792: Idle: test PathBrowser more http://bugs.python.org/issue20792 closed by terry.reedy #21676: IDLE - Test Replace Dialog http://bugs.python.org/issue21676 closed by terry.reedy #21703: IDLE: Test UndoDelegator http://bugs.python.org/issue21703 closed by terry.reedy #21939: IDLE - Test Percolator http://bugs.python.org/issue21939 closed by terry.reedy #23275: Can assign [] = (), but not () = [] http://bugs.python.org/issue23275 closed by berker.peksag #23958: compile warnings in libffi http://bugs.python.org/issue23958 closed by berker.peksag #23977: Enhancing IDLE's test_delegator.py unit test http://bugs.python.org/issue23977 closed by terry.reedy #25285: regrtest: run tests in subprocesses with -j1 on buildbots http://bugs.python.org/issue25285 closed by haypo #25650: Mismatching documentation <=> behaviour for typing.Any http://bugs.python.org/issue25650 closed by berker.peksag #25747: test_idle failure in leaks searching mode http://bugs.python.org/issue25747 closed by terry.reedy #26039: More flexibility in zipfile write interface http://bugs.python.org/issue26039 closed by takluyver #26807: mock_open()().readline() fails at EOF http://bugs.python.org/issue26807 closed by rbcollins #26848: asyncio.subprocess's communicate() method mishandles empty inp http://bugs.python.org/issue26848 closed by berker.peksag #26850: PyMem_RawMalloc(): update also sys.getallocatedblocks() in deb http://bugs.python.org/issue26850 closed by haypo #26938: android: test_concurrent_futures hangs on armv7 http://bugs.python.org/issue26938 closed by xdegaye #26948: Simplify PyImport_ImportModuleLevelObject: avoid temporary tup http://bugs.python.org/issue26948 closed by haypo #26991: Possible reference leak in MAKE_FUNCTION http://bugs.python.org/issue26991 closed by python-dev #26995: Add tests for parsing float and object arguments http://bugs.python.org/issue26995 closed by serhiy.storchaka #27014: maximum recursion depth when using typing options http://bugs.python.org/issue27014 closed by gvanrossum #27017: Python3.5.1: type().startswith() http://bugs.python.org/issue27017 closed by SilentGhost #27018: Incorrect documentation of select module http://bugs.python.org/issue27018 closed by orsenthil #27023: Documentation of tempfile.gettempdir() does not mention it onl http://bugs.python.org/issue27023 closed by r.david.murray #27028: configparser BasicInterpolation issue with %() http://bugs.python.org/issue27028 closed by r.david.murray #27031: Remove interface to obsolete Tk commands http://bugs.python.org/issue27031 closed by serhiy.storchaka #27034: Remove deprecated asynchat.fifo http://bugs.python.org/issue27034 closed by serhiy.storchaka #27039: bytearray.remove cannot remove bytes with value greater than 1 http://bugs.python.org/issue27039 closed by serhiy.storchaka #27040: asyncio: Add loop.get_exception_handler method http://bugs.python.org/issue27040 closed by yselivanov #27041: asyncio: Add loop.create_future method http://bugs.python.org/issue27041 closed by yselivanov #27042: Incorrect grammar for function definitions http://bugs.python.org/issue27042 closed by python-dev #27044: Stop test_idle memory leaks http://bugs.python.org/issue27044 closed by terry.reedy #27045: Forward slashes in Windows paths http://bugs.python.org/issue27045 closed by ebarry #27046: Emoji support in Pie Chart legend with matplotlib http://bugs.python.org/issue27046 closed by berker.peksag #27047: O(1) deque indexing http://bugs.python.org/issue27047 closed by rhettinger #27049: Typo in "The Python Language Reference" 7.11 -- "individiual" http://bugs.python.org/issue27049 closed by ned.deily #27053: Python Windows x86-64 embeddable zip file missing module encod http://bugs.python.org/issue27053 closed by steve.dower #27055: python fails to run zipapp when zip contains comments http://bugs.python.org/issue27055 closed by serhiy.storchaka #27056: pickle: constant propagation in _Unpickler_Read() http://bugs.python.org/issue27056 closed by haypo #27057: os.set_inheritable(): fall back to fcntl() if ioctl() fails wi http://bugs.python.org/issue27057 closed by haypo #27058: Tkinter's canvas' dashed lines have incorrect segment lengths http://bugs.python.org/issue27058 closed by kms70847 #27059: find_spec and find_loader ignore package/path argument http://bugs.python.org/issue27059 closed by brett.cannon #27061: 2.7.11 OS X pkg doesn't put proper version in the receipts http://bugs.python.org/issue27061 closed by ned.deily #27065: robotparser user agent considered hostile by mod_security rule http://bugs.python.org/issue27065 closed by berker.peksag From brett at python.org Fri May 20 12:27:26 2016 From: brett at python.org (Brett Cannon) Date: Fri, 20 May 2016 16:27:26 +0000 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? Message-ID: Is there a specific reason for calling out two spaces in comments after a period(I realize it's probably for consistency, but I sure don't ever think about this when I write comment)? Otherwise who actually still writes using two spaces after punctuation? :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri May 20 12:35:21 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 20 May 2016 09:35:21 -0700 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: References: Message-ID: In Emacs, the paragraph reflow commands still try to keep a period followed by only one space together on a line, under the assumption that it might be something like `Dr. Brett Cannon`. A reasonable HTML generator should ignore the second space. So all in all I still like the two-space rule -- not so much for readability but because of tooling. On Fri, May 20, 2016 at 9:27 AM, Brett Cannon wrote: > Is there a specific reason for calling out two spaces in comments after a > period(I realize it's probably for consistency, but I sure don't ever think > about this when I write comment)? Otherwise who actually still writes using > two spaces after punctuation? :) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From mafagafogigante at gmail.com Fri May 20 12:37:05 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Fri, 20 May 2016 13:37:05 -0300 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: References: Message-ID: <573F3D31.1020806@gmail.com> On 05/20/2016 01:27 PM, Brett Cannon wrote: > Is there a specific reason for calling out two spaces in comments after a > period(I realize it's probably for consistency, but I sure don't ever think > about this when I write comment)? Otherwise who actually still writes using > two spaces after punctuation? :) > I've also asked myself this question. It may have made more sense back then, but nowadays I only get double spaces after a period if Ms. Vim decides to use them there for me, making me do substitution magic to "clean" it later on. There may even be a way to disable this in Vim (without changing the sources and recompiling), but I never looked after it. For modern monospaced typefaces and screen resolutions one space after a period is as readable (arguably even better looking) than two. From facundobatista at gmail.com Fri May 20 12:40:18 2016 From: facundobatista at gmail.com (Facundo Batista) Date: Fri, 20 May 2016 13:40:18 -0300 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: References: Message-ID: On Fri, May 20, 2016 at 1:27 PM, Brett Cannon wrote: > Is there a specific reason for calling out two spaces in comments after a > period(I realize it's probably for consistency, but I sure don't ever think > about this when I write comment)? Otherwise who actually still writes using > two spaces after punctuation? :) I couldn't find it in the reference but AFAIK is something that is also done in ReStructured Text. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From brett at python.org Fri May 20 12:43:09 2016 From: brett at python.org (Brett Cannon) Date: Fri, 20 May 2016 16:43:09 +0000 Subject: [Python-Dev] Removing the provisional label from pathlib Message-ID: Three questions: 1. Should pathlib gain __fspath__() all the way back to 3.4? 2. Should pathlib's constructor support __fspath__() all the way back to 3.4? (separate question as os.fspath() will only be in 3.6; and if we backport I'm not looking forward to making Typeshed happy w/o os.PathLike being available only in 3.6 :/) 3. Should the docs from 3.4 and forward reflect the removal of the provisional status? (I assume yes, but wanted to double-check) And a quick thanks to Guido for removing `path` from pathlib for me already. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri May 20 12:45:04 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 20 May 2016 09:45:04 -0700 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: References: Message-ID: <573F3F10.4000408@stoneleaf.us> On 05/20/2016 09:27 AM, Brett Cannon wrote: > Is there a specific reason for calling out two spaces in comments after > a period(I realize it's probably for consistency, but I sure don't ever > think about this when I write comment)? Otherwise who actually still > writes using two spaces after punctuation? :) I do!! :) The period is a full-stop punctuation mark, but visually obscure [1]. The two spaces reinforces its full-stop nature [2]. white-space-matters-ly yrs, -- ~Ethan~ [1] I'm pretty sure "obscure" is not the word I want, but it's close. [2] No, I'm not happy with the decision to drop to one space after a period in modern grammar [3]. [3] At least, at the colleges my daughter has attended. From guido at python.org Fri May 20 12:46:09 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 20 May 2016 09:46:09 -0700 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: References: Message-ID: ALso, in case anyone reading this didn't realize, this is a classic internet debate that has been going on for decades (just not on python-dev). Just Google it. On Fri, May 20, 2016 at 9:40 AM, Facundo Batista wrote: > On Fri, May 20, 2016 at 1:27 PM, Brett Cannon wrote: > >> Is there a specific reason for calling out two spaces in comments after a >> period(I realize it's probably for consistency, but I sure don't ever think >> about this when I write comment)? Otherwise who actually still writes using >> two spaces after punctuation? :) > > I couldn't find it in the reference but AFAIK is something that is > also done in ReStructured Text. > > Regards, > > -- > . Facundo > > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > Twitter: @facundobatista > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri May 20 12:56:00 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 20 May 2016 09:56:00 -0700 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: Message-ID: Let's start in 3.6 with all this. I added path to 3.4 because I didn't realize it was in security-mode only. I've now undone all my work there. Let's not disturb it again, not even its docs. I don't think there's an "upstream" repo for pathlib (like there still is for asyncio) and I don't think there's much of a point in supporting __fspath__ in pathlib if there's no os.fspath(). It would only encourage hackery in apps that want to play with __fspath__. On Fri, May 20, 2016 at 9:43 AM, Brett Cannon wrote: > Three questions: > > Should pathlib gain __fspath__() all the way back to 3.4? > Should pathlib's constructor support __fspath__() all the way back to 3.4? > (separate question as os.fspath() will only be in 3.6; and if we backport > I'm not looking forward to making Typeshed happy w/o os.PathLike being > available only in 3.6 :/) > Should the docs from 3.4 and forward reflect the removal of the provisional > status? (I assume yes, but wanted to double-check) > > And a quick thanks to Guido for removing `path` from pathlib for me already. > :) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From nas at arctrix.com Fri May 20 13:00:19 2016 From: nas at arctrix.com (Neil Schemenauer) Date: Fri, 20 May 2016 17:00:19 +0000 (UTC) Subject: [Python-Dev] PyGC_Collect ignores state of `enabled` References: <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com> <573CBCB7.7000202@stoneleaf.us> Message-ID: Nick Coghlan wrote: > PEP 3121 is insufficient, since a lot of extension modules can't (or > at least haven't) adopted it in practice. > https://www.python.org/dev/peps/pep-0489/ has some more background on > that (since it was the first step towards tackling the problem in a > different way that extension module authors may be more likely to > actually adopt) My idea is that if we can cleanup the built-in extension modules then maybe that would be enough to stop doing a lot of the finalization hacks (e.g. module dict clearing). If 3rd party extension modules are not fixed, then some finalizers that used to be called might no longer get called. From mafagafogigante at gmail.com Fri May 20 13:30:08 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Fri, 20 May 2016 14:30:08 -0300 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: <573F3F10.4000408@stoneleaf.us> References: <573F3F10.4000408@stoneleaf.us> Message-ID: <573F49A0.8090404@gmail.com> > On 05/20/2016 09:27 AM, Brett Cannon wrote: > > The period is a full-stop punctuation mark, but visually obscure [1]. It is small. This does not make it hard to notice, however. Not to mention that it will usually be followed by a capitalized letter, which further highlights it. Commas usually use more pixels or ink and the tiny size of the period makes it quite distinguishable to me. This email-based "tennis" match could go on for quite a while as Guido pointed out and there are several good points for both sides. This is not a critic to the thread, which regarded PEP 7/8 and seems perfectly fine. This is also not a critic to Cannon's view on this. Who is right in the end? ~One space defenders~ Nobody. There is no single right solution to this. As there is no single right decision regarding tabs or spaces~, except for spaces, always use spaces~. Consistency is what really matters (assuming we agree that one space is enough to make it visible). From brett at python.org Fri May 20 14:40:12 2016 From: brett at python.org (Brett Cannon) Date: Fri, 20 May 2016 18:40:12 +0000 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: References: Message-ID: On Fri, 20 May 2016 at 09:46 Guido van Rossum wrote: > ALso, in case anyone reading this didn't realize, this is a classic > internet debate that has been going on for decades (just not on > python-dev). Just Google it. > Yes, I don't mean to suggest we should change the two space specification to one-space, just why specify at all to that granularity of formatting? -Brett > > On Fri, May 20, 2016 at 9:40 AM, Facundo Batista > wrote: > > On Fri, May 20, 2016 at 1:27 PM, Brett Cannon wrote: > > > >> Is there a specific reason for calling out two spaces in comments after > a > >> period(I realize it's probably for consistency, but I sure don't ever > think > >> about this when I write comment)? Otherwise who actually still writes > using > >> two spaces after punctuation? :) > > > > I couldn't find it in the reference but AFAIK is something that is > > also done in ReStructured Text. > > > > Regards, > > > > -- > > . Facundo > > > > Blog: http://www.taniquetil.com.ar/plog/ > > PyAr: http://www.python.org/ar/ > > Twitter: @facundobatista > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri May 20 14:41:09 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 20 May 2016 11:41:09 -0700 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: References: Message-ID: Because Emacs. :-) I am going to mute this thread now. On Fri, May 20, 2016 at 11:40 AM, Brett Cannon wrote: > > > On Fri, 20 May 2016 at 09:46 Guido van Rossum wrote: >> >> ALso, in case anyone reading this didn't realize, this is a classic >> internet debate that has been going on for decades (just not on >> python-dev). Just Google it. > > > Yes, I don't mean to suggest we should change the two space specification to > one-space, just why specify at all to that granularity of formatting? > > -Brett > >> >> >> On Fri, May 20, 2016 at 9:40 AM, Facundo Batista >> wrote: >> > On Fri, May 20, 2016 at 1:27 PM, Brett Cannon wrote: >> > >> >> Is there a specific reason for calling out two spaces in comments after >> >> a >> >> period(I realize it's probably for consistency, but I sure don't ever >> >> think >> >> about this when I write comment)? Otherwise who actually still writes >> >> using >> >> two spaces after punctuation? :) >> > >> > I couldn't find it in the reference but AFAIK is something that is >> > also done in ReStructured Text. >> > >> > Regards, >> > >> > -- >> > . Facundo >> > >> > Blog: http://www.taniquetil.com.ar/plog/ >> > PyAr: http://www.python.org/ar/ >> > Twitter: @facundobatista >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) -- --Guido van Rossum (python.org/~guido) From brett at python.org Fri May 20 14:42:06 2016 From: brett at python.org (Brett Cannon) Date: Fri, 20 May 2016 18:42:06 +0000 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: Message-ID: On Fri, 20 May 2016 at 09:56 Guido van Rossum wrote: > Let's start in 3.6 with all this. I added path to 3.4 because I didn't > realize it was in security-mode only. I've now undone all my work > there. Let's not disturb it again, not even its docs. > > I don't think there's an "upstream" repo for pathlib (like there still > is for asyncio) and I don't think there's much of a point in > supporting __fspath__ in pathlib if there's no os.fspath(). It would > only encourage hackery in apps that want to play with __fspath__. > WFM. I'll let 3.4 and 3.5 just stay as they are and make all PEP 519 changes a 3.6 thing. I'll update the implementation task list shortly. -Brett > > On Fri, May 20, 2016 at 9:43 AM, Brett Cannon wrote: > > Three questions: > > > > Should pathlib gain __fspath__() all the way back to 3.4? > > Should pathlib's constructor support __fspath__() all the way back to > 3.4? > > (separate question as os.fspath() will only be in 3.6; and if we backport > > I'm not looking forward to making Typeshed happy w/o os.PathLike being > > available only in 3.6 :/) > > Should the docs from 3.4 and forward reflect the removal of the > provisional > > status? (I assume yes, but wanted to double-check) > > > > And a quick thanks to Guido for removing `path` from pathlib for me > already. > > :) > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri May 20 15:15:18 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 20 May 2016 21:15:18 +0200 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: Message-ID: 2016-05-20 18:56 GMT+02:00 Guido van Rossum : > Let's start in 3.6 with all this. I added path to 3.4 because I didn't > realize it was in security-mode only. I also had to ask the question to myself about branches, that's why I wrote this table ;-) https://docs.python.org/devguide/#status-of-python-branches By the way, is it still up to date? Python 3.2 end-of-line is documented as 2016-02-20, so its status should be end-of-life, no? Georg Brandl scheduled a 3.2.7 release at the end of February 2016, but I don't see it on python.org and I don't recall its announcement. https://mail.python.org/pipermail/python-dev/2016-February/143300.html Georg: any update on the Python 3.2 last release? Victor From pjenvey at underboss.org Fri May 20 15:21:15 2016 From: pjenvey at underboss.org (Philip Jenvey) Date: Fri, 20 May 2016 12:21:15 -0700 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: References: Message-ID: <00F1D709-27D0-48C4-8CDC-DAC7D70CAD83@underboss.org> So is line breaks before or after operators . -- Philip Jenvey > On May 20, 2016, at 9:46 AM, Guido van Rossum wrote: > > ALso, in case anyone reading this didn't realize, this is a classic > internet debate that has been going on for decades (just not on > python-dev). Just Google it. > > On Fri, May 20, 2016 at 9:40 AM, Facundo Batista > wrote: >> On Fri, May 20, 2016 at 1:27 PM, Brett Cannon wrote: >> >>> Is there a specific reason for calling out two spaces in comments after a >>> period(I realize it's probably for consistency, but I sure don't ever think >>> about this when I write comment)? Otherwise who actually still writes using >>> two spaces after punctuation? :) >> >> I couldn't find it in the reference but AFAIK is something that is >> also done in ReStructured Text. >> >> Regards, >> >> -- >> . Facundo >> >> Blog: http://www.taniquetil.com.ar/plog/ >> PyAr: http://www.python.org/ar/ >> Twitter: @facundobatista >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/pjenvey%40underboss.org From g.brandl at gmx.net Fri May 20 15:18:37 2016 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 20 May 2016 21:18:37 +0200 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: Message-ID: On 05/20/2016 09:15 PM, Victor Stinner wrote: > 2016-05-20 18:56 GMT+02:00 Guido van Rossum : >> Let's start in 3.6 with all this. I added path to 3.4 because I didn't >> realize it was in security-mode only. > > I also had to ask the question to myself about branches, that's why I > wrote this table ;-) > https://docs.python.org/devguide/#status-of-python-branches > > By the way, is it still up to date? Python 3.2 end-of-line is > documented as 2016-02-20, so its status should be end-of-life, no? > > Georg Brandl scheduled a 3.2.7 release at the end of February 2016, > but I don't see it on python.org and I don't recall its announcement. > https://mail.python.org/pipermail/python-dev/2016-February/143300.html > > Georg: any update on the Python 3.2 last release? Still waiting for some last security update. I sent another mail to python-dev that announced the delay, I think. Georg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From stephen at xemacs.org Sat May 21 03:44:20 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 21 May 2016 16:44:20 +0900 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: <573F49A0.8090404@gmail.com> References: <573F3F10.4000408@stoneleaf.us> <573F49A0.8090404@gmail.com> Message-ID: <22336.4564.834078.187823@turnbull.sk.tsukuba.ac.jp> Bernardo Sulzbach writes: > > On 05/20/2016 09:27 AM, Brett Cannon wrote: > > > > The period is a full-stop punctuation mark, but visually obscure [1]. > > It is small. This does not make it hard to notice, however. That depends on fonts, and on the ability of the user to write correctly as well. "Syntax should not look like grit on Tim's screen" applies to natural language as well. I read a lot of English-as-a- second-language text, and it really does help to have the extra space, because the period often appears where English grammar suggests it doesn't belong. The extra effort to use the second space helps to emphasize the author's intent, and even subjectively make the period "more visible". But I agree with Guido. No Python contributors write poorly, so the real issues are consistency with past practice (a minor consideration in the case of comments) and tooling (since both Emacs and vim seem to come out of the box believing in this rule, I guess it's widespread enough to care). As for Brett's concern about whether this should be specified at this level, I think tooling again is an answer. Emacs reflows whole paragraphs. That means that text that is off-screen may be changed according to this rule, and that is an annoyance when you only notice it while preparing a patch for submission. I think a detail of style that triggers unwanted (but plausible[1]) behavior in a widely-used tool is worth standardizing. Sometimes, of course, you standardize *despite* the tool, but then at least the tool-users know that a decision has been taken to impose the burden on them, vs. imposing the burden on people who prefer the other style. Footnotes: [1] If the behavior is implausible, fix the tool, of course! From stephen at xemacs.org Sat May 21 03:49:11 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 21 May 2016 16:49:11 +0900 Subject: [Python-Dev] Adding Type[C] to PEP 484 In-Reply-To: <573EED58.3040508@mail.de> References: <573EED58.3040508@mail.de> Message-ID: <22336.4855.445392.833399@turnbull.sk.tsukuba.ac.jp> Sven R. Kunze writes: > Type[A, B] reminds me of isinstance(obj, (A, B)). Sure, but Guido already invoked EIBTI. For one (obscure) alternative, Type[A, B] reminded me of Fun(A,B) (the functors from category A to category B), and in Python I would tend to map that to a function type (taking argument from A, returning value from B). Another (probably more plausible to Pythonistas) would be a type that *mixes* A and B, ie, one might be reminded of class C(A, B): pass isinstance(obj, C) or even (continuing the example) class E(B, A, D): # Order of A, B irrelevant! Yikes! pass isinstance(obj, (C, E)) Regards, From pludemann at google.com Sun May 22 11:15:12 2016 From: pludemann at google.com (Peter Ludemann) Date: Sun, 22 May 2016 08:15:12 -0700 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: <22336.4564.834078.187823@turnbull.sk.tsukuba.ac.jp> References: <573F3F10.4000408@stoneleaf.us> <573F49A0.8090404@gmail.com> <22336.4564.834078.187823@turnbull.sk.tsukuba.ac.jp> Message-ID: I just tried a re-flow in emacs (M-q or fill-paragraph) and it didn't change my single space to two (it also changed 3 spaces after a period to 2; and it preserved my hanging indents). I have no idea what vim does. (OTOH, emacs annoyingly adds a blank line to my """ docstrings, which I remove manually.) I'm not going to stoke the fires of debate (especially as I've worked with typographers over the years and know that the typographical sins of programmers go far beyond the double-space-after-period), except to note that if you read the thousands of words that have been written about spacing after periods, they're usually talking about *proportional* fonts; and we still tend to use monospaced fonts with programming tools. On 21 May 2016 at 00:44, Stephen J. Turnbull wrote: > Bernardo Sulzbach writes: > > > On 05/20/2016 09:27 AM, Brett Cannon wrote: > > > > > > The period is a full-stop punctuation mark, but visually obscure [1]. > > > > It is small. This does not make it hard to notice, however. > > That depends on fonts, and on the ability of the user to write > correctly as well. "Syntax should not look like grit on Tim's screen" > applies to natural language as well. I read a lot of English-as-a- > second-language text, and it really does help to have the extra space, > because the period often appears where English grammar suggests it > doesn't belong. The extra effort to use the second space helps to > emphasize the author's intent, and even subjectively make the period > "more visible". > > But I agree with Guido. No Python contributors write poorly, so the > real issues are consistency with past practice (a minor consideration > in the case of comments) and tooling (since both Emacs and vim seem to > come out of the box believing in this rule, I guess it's widespread > enough to care). > > As for Brett's concern about whether this should be specified at this > level, I think tooling again is an answer. Emacs reflows whole > paragraphs. That means that text that is off-screen may be changed > according to this rule, and that is an annoyance when you only notice > it while preparing a patch for submission. I think a detail of style > that triggers unwanted (but plausible[1]) behavior in a widely-used > tool is worth standardizing. Sometimes, of course, you standardize > *despite* the tool, but then at least the tool-users know that a > decision has been taken to impose the burden on them, vs. imposing the > burden on people who prefer the other style. > > Footnotes: > [1] If the behavior is implausible, fix the tool, of course! > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/pludemann%40google.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Sun May 22 14:01:44 2016 From: random832 at fastmail.com (Random832) Date: Sun, 22 May 2016 14:01:44 -0400 Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a period? In-Reply-To: References: <573F3F10.4000408@stoneleaf.us> <573F49A0.8090404@gmail.com> <22336.4564.834078.187823@turnbull.sk.tsukuba.ac.jp> Message-ID: <1463940104.1833604.615256001.4820654B@webmail.messagingengine.com> On Sun, May 22, 2016, at 11:15, Peter Ludemann via Python-Dev wrote: > I just tried a re-flow in emacs (M-q or fill-paragraph) and it didn't > change my single space to two (it also changed 3 spaces after a period > to 2; and it preserved my hanging indents). It won't change a single space to two, but it will add two spaces after a sentence which was previously at the end of a line... and it *won't* break a line after a dot followed by a single space (Vim isn't clever enough for this second bit), preferring to break before the preceding word instead. A dot followed by a single space is *permitted* (hence why you observed it didn't change it to two), it is simply assumed to represent something other than the end of a sentence, such as an initial in a name. From chris.barker at noaa.gov Mon May 23 12:55:04 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 23 May 2016 09:55:04 -0700 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: Message-ID: On Fri, May 20, 2016 at 11:42 AM, Brett Cannon wrote: > > WFM. I'll let 3.4 and 3.5 just stay as they are and make all PEP 519 > changes a 3.6 thing. > I'd like to see it in 3.5 if we could do that -- that'll be available for operational code a lot sooner than 3.6, and it's still in active maintenance mode, yes? -CHB > I'll update the implementation task list shortly. > > -Brett > > >> >> On Fri, May 20, 2016 at 9:43 AM, Brett Cannon wrote: >> > Three questions: >> > >> > Should pathlib gain __fspath__() all the way back to 3.4? >> > Should pathlib's constructor support __fspath__() all the way back to >> 3.4? >> > (separate question as os.fspath() will only be in 3.6; and if we >> backport >> > I'm not looking forward to making Typeshed happy w/o os.PathLike being >> > available only in 3.6 :/) >> > Should the docs from 3.4 and forward reflect the removal of the >> provisional >> > status? (I assume yes, but wanted to double-check) >> > >> > And a quick thanks to Guido for removing `path` from pathlib for me >> already. >> > :) >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 23 13:13:51 2016 From: brett at python.org (Brett Cannon) Date: Mon, 23 May 2016 17:13:51 +0000 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: Message-ID: On Mon, May 23, 2016, 09:55 Chris Barker wrote: > On Fri, May 20, 2016 at 11:42 AM, Brett Cannon wrote: > >> >> WFM. I'll let 3.4 and 3.5 just stay as they are and make all PEP 519 >> changes a 3.6 thing. >> > > I'd like to see it in 3.5 if we could do that -- that'll be available for > operational code a lot sooner than 3.6, and it's still in active > maintenance mode, yes? > 3.5 is still getting bugfixes: https://docs.python.org/devguide/#status-of-python-branches As for backporting __fspath__() for pathlib, you can easily write your own subclass that adds it. And since the stdlib won't be updated in 3.5 for consumption of fspath you're only missing out on production which your subclass can take care of. -brett > -CHB > > > > >> I'll update the implementation task list shortly. >> >> -Brett >> >> >>> >>> On Fri, May 20, 2016 at 9:43 AM, Brett Cannon wrote: >>> > Three questions: >>> > >>> > Should pathlib gain __fspath__() all the way back to 3.4? >>> > Should pathlib's constructor support __fspath__() all the way back to >>> 3.4? >>> > (separate question as os.fspath() will only be in 3.6; and if we >>> backport >>> > I'm not looking forward to making Typeshed happy w/o os.PathLike being >>> > available only in 3.6 :/) >>> > Should the docs from 3.4 and forward reflect the removal of the >>> provisional >>> > status? (I assume yes, but wanted to double-check) >>> > >>> > And a quick thanks to Guido for removing `path` from pathlib for me >>> already. >>> > :) >>> > >>> > _______________________________________________ >>> > Python-Dev mailing list >>> > Python-Dev at python.org >>> > https://mail.python.org/mailman/listinfo/python-dev >>> > Unsubscribe: >>> > https://mail.python.org/mailman/options/python-dev/guido%40python.org >>> > >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov >> >> > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon May 23 15:38:53 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 23 May 2016 12:38:53 -0700 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: Message-ID: On Mon, May 23, 2016 at 10:13 AM, Brett Cannon wrote: > 3.5 is still getting bugfixes: > https://docs.python.org/devguide/#status-of-python-branches > > As for backporting __fspath__() for pathlib, you can easily write your own > subclass that adds it. And since the stdlib won't be updated in 3.5 for > consumption of fspath > OK -- when I said "it", I meant the whole shebang -- i.e. the stdlib stuff too. But fair enough, we can't be backporting everything, and I that would be touching a lot of the lib. -CHB > you're only missing out on production which your subclass can take care of. > > -brett > > > >> -CHB >> >> >> >> >>> I'll update the implementation task list shortly. >>> >>> -Brett >>> >>> >>>> >>>> On Fri, May 20, 2016 at 9:43 AM, Brett Cannon wrote: >>>> > Three questions: >>>> > >>>> > Should pathlib gain __fspath__() all the way back to 3.4? >>>> > Should pathlib's constructor support __fspath__() all the way back to >>>> 3.4? >>>> > (separate question as os.fspath() will only be in 3.6; and if we >>>> backport >>>> > I'm not looking forward to making Typeshed happy w/o os.PathLike being >>>> > available only in 3.6 :/) >>>> > Should the docs from 3.4 and forward reflect the removal of the >>>> provisional >>>> > status? (I assume yes, but wanted to double-check) >>>> > >>>> > And a quick thanks to Guido for removing `path` from pathlib for me >>>> already. >>>> > :) >>>> > >>>> > _______________________________________________ >>>> > Python-Dev mailing list >>>> > Python-Dev at python.org >>>> > https://mail.python.org/mailman/listinfo/python-dev >>>> > Unsubscribe: >>>> > https://mail.python.org/mailman/options/python-dev/guido%40python.org >>>> > >>>> >>>> >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >>>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> >> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov >>> >>> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon May 23 21:36:06 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 23 May 2016 18:36:06 -0700 Subject: [Python-Dev] Python 3.6.0a1 is now available In-Reply-To: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org> References: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org> Message-ID: <-7150763876064620774@unknownmsgid> Ned, It looks like you're still building for OS-X v. 10.6. I can't find an official statement, but this: http://www.computerworld.com/article/2950580/operating-systems/the-end-is-near-for-os-x-mountain-lion-support.html Indicates that 10.8 is falling off Apple's support. Maybe it's time for a newer baseline. -Chris > On May 17, 2016, at 2:18 PM, Ned Deily wrote: > > On behalf of the Python development community and the Python 3.6 release > team, I'm happy to announce the availability of Python 3.6.0a1. > 3.6.0a1 is the first of four planned alpha releases of Python 3.6, > the next major release of Python. During the alpha phase, Python 3.6 > remains under heavy development: additional features will be added > and existing features may be modified or deleted. Please keep in mind > that this is a preview release and its use is not recommended for > production environments. > > You can find Python 3.6.0a1 here: > > https://www.python.org/downloads/release/python-360a1/ > > The next release of Python 3.6 will be 3.6.0a2, currently scheduled for > 2016-06-13. > > Enjoy! > > --Ned > > -- > Ned Deily > nad at python.org -- [] > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov From k7hoven at gmail.com Tue May 24 07:03:34 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 24 May 2016 14:03:34 +0300 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: Message-ID: On Mon, May 23, 2016 at 10:38 PM, Chris Barker wrote: > On Mon, May 23, 2016 at 10:13 AM, Brett Cannon wrote: >> >> 3.5 is still getting bugfixes: >> https://docs.python.org/devguide/#status-of-python-branches >> >> As for backporting __fspath__() for pathlib, you can easily write your own >> subclass that adds it. And since the stdlib won't be updated in 3.5 for >> consumption of fspath > > > OK -- when I said "it", I meant the whole shebang -- i.e. the stdlib stuff > too. But fair enough, we can't be backporting everything, and I that would > be touching a lot of the lib. > > -CHB I guess we might consider adding __fspath__ in maintenance releases, and make open() support it? That would cover a significant share of use cases, although it might be weird if code written for 3.5.2 doesn't run on 3.5.1... So maybe just 3.6-> ? I'm not quite as busy as last week, so I might consider working on the stdlib changes if I find the time. Or is someone already working on this? -- Koos From barry at python.org Tue May 24 09:56:51 2016 From: barry at python.org (Barry Warsaw) Date: Tue, 24 May 2016 09:56:51 -0400 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: Message-ID: <20160524095651.014e37c4.barry@wooz.org> On May 24, 2016, at 02:03 PM, Koos Zevenhoven wrote: >I guess we might consider adding __fspath__ in maintenance releases, >and make open() support it? That would cover a significant share of >use cases, although it might be weird if code written for 3.5.2 >doesn't run on 3.5.1... Please, no. We learned that lesson in Python 2.2.1 with True/False. Cheers, -Barry From k7hoven at gmail.com Tue May 24 10:11:46 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 24 May 2016 17:11:46 +0300 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: <20160524095651.014e37c4.barry@wooz.org> References: <20160524095651.014e37c4.barry@wooz.org> Message-ID: On Tue, May 24, 2016 at 4:56 PM, Barry Warsaw wrote: > On May 24, 2016, at 02:03 PM, Koos Zevenhoven wrote: > >>I guess we might consider adding __fspath__ in maintenance releases, >>and make open() support it? That would cover a significant share of >>use cases, although it might be weird if code written for 3.5.2 >>doesn't run on 3.5.1... > > Please, no. We learned that lesson in Python 2.2.1 with True/False. What happened? True was included in 2.2.1 but not False?-). Anyway, I guess you are probably right, and "3.6->" is the way to go. Besides, Guido already wrote that in the first response. -- Koos > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com From p.f.moore at gmail.com Tue May 24 10:49:34 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 May 2016 15:49:34 +0100 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: <20160524095651.014e37c4.barry@wooz.org> Message-ID: On 24 May 2016 at 15:11, Koos Zevenhoven wrote: >> Please, no. We learned that lesson in Python 2.2.1 with True/False. > > What happened? True was included in 2.2.1 but not False?-). Anyway, I > guess you are probably right, and "3.6->" is the way to go. Besides, > Guido already wrote that in the first response. The history is at http://python-history.blogspot.co.uk/2013/11/the-history-of-bool-true-and-false.html but basically we added new *constants* for True/False in a point release. Some people used them, and their code only worked on 2.2.1 and above but not 2.2.0. Others were using variables called True and False, and the 2.2.1 change broke their code (you could no longer assign to those names). So essentially it ended up that people couldn't say "works on 2.2" and had to distinguish between 2.2.0 and 2.2.1+. Python's strict backward compatibility rules basically stem from the grief caused by that change. Paul From brett at snarky.ca Tue May 24 12:57:18 2016 From: brett at snarky.ca (Brett Cannon) Date: Tue, 24 May 2016 16:57:18 +0000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #26647: Python interpreter now uses 16-bit wordcode instead of bytecode. In-Reply-To: <20160524061542.28801.84595.29E5EEA7@psf.io> References: <20160524061542.28801.84595.29E5EEA7@psf.io> Message-ID: This probably deserves a "What's New" entry since it does break bytecode-inspecting code. On Mon, 23 May 2016 at 23:15 serhiy.storchaka wrote: > https://hg.python.org/cpython/rev/3a57eafd8401 > changeset: 101486:3a57eafd8401 > user: Serhiy Storchaka > date: Tue May 24 09:15:14 2016 +0300 > summary: > Issue #26647: Python interpreter now uses 16-bit wordcode instead of > bytecode. > Patch by Demur Rumed. > > files: > Doc/library/dis.rst | 9 +- > Lib/ctypes/test/test_values.py | 6 +- > Lib/dis.py | 35 +- > Lib/importlib/_bootstrap_external.py | 3 +- > Lib/test/test_dis.py | 512 +- > Misc/NEWS | 3 + > Objects/frameobject.c | 12 +- > Objects/genobject.c | 4 +- > PC/launcher.c | 2 +- > PCbuild/pythoncore.vcxproj | 1 + > PCbuild/pythoncore.vcxproj.filters | 3 + > Python/ceval.c | 133 +- > Python/compile.c | 81 +- > Python/frozen.c | 20 +- > Python/importlib.h | 3719 +++++----- > Python/importlib_external.h | 4612 ++++++------- > Python/peephole.c | 578 +- > Python/wordcode_helpers.h | 38 + > 18 files changed, 4748 insertions(+), 5023 deletions(-) > > > diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst > --- a/Doc/library/dis.rst > +++ b/Doc/library/dis.rst > @@ -31,9 +31,9 @@ > > >>> dis.dis(myfunc) > 2 0 LOAD_GLOBAL 0 (len) > - 3 LOAD_FAST 0 (alist) > - 6 CALL_FUNCTION 1 > - 9 RETURN_VALUE > + 2 LOAD_FAST 0 (alist) > + 4 CALL_FUNCTION 1 > + 6 RETURN_VALUE > > (The "2" is a line number). > > @@ -682,8 +682,7 @@ > .. XXX explain the WHY stuff! > > > -All of the following opcodes expect arguments. An argument is two bytes, > with > -the more significant byte last. > +All of the following opcodes use their arguments. > > .. opcode:: STORE_NAME (namei) > > diff --git a/Lib/ctypes/test/test_values.py > b/Lib/ctypes/test/test_values.py > --- a/Lib/ctypes/test/test_values.py > +++ b/Lib/ctypes/test/test_values.py > @@ -79,9 +79,9 @@ > continue > items.append((entry.name.decode("ascii"), entry.size)) > > - expected = [("__hello__", 161), > - ("__phello__", -161), > - ("__phello__.spam", 161), > + expected = [("__hello__", 139), > + ("__phello__", -139), > + ("__phello__.spam", 139), > ] > self.assertEqual(items, expected, "PyImport_FrozenModules example > " > "in Doc/library/ctypes.rst may be out of date") > diff --git a/Lib/dis.py b/Lib/dis.py > --- a/Lib/dis.py > +++ b/Lib/dis.py > @@ -285,7 +285,6 @@ > """ > labels = findlabels(code) > starts_line = None > - free = None > for offset, op, arg in _unpack_opargs(code): > if linestarts is not None: > starts_line = linestarts.get(offset, None) > @@ -296,7 +295,7 @@ > argrepr = '' > if arg is not None: > # Set argval to the dereferenced value of the argument when > - # availabe, and argrepr to the string representation of > argval. > + # available, and argrepr to the string representation of > argval. > # _disassemble_bytes needs the string repr of the > # raw name index for LOAD_GLOBAL, LOAD_CONST, etc. > argval = arg > @@ -305,7 +304,7 @@ > elif op in hasname: > argval, argrepr = _get_name_info(arg, names) > elif op in hasjrel: > - argval = offset + 3 + arg > + argval = offset + 2 + arg > argrepr = "to " + repr(argval) > elif op in haslocal: > argval, argrepr = _get_name_info(arg, varnames) > @@ -352,23 +351,15 @@ > disco = disassemble # XXX For backwards compatibility > > def _unpack_opargs(code): > - # enumerate() is not an option, since we sometimes process > - # multiple elements on a single pass through the loop > extended_arg = 0 > - n = len(code) > - i = 0 > - while i < n: > + for i in range(0, len(code), 2): > op = code[i] > - offset = i > - i = i+1 > - arg = None > if op >= HAVE_ARGUMENT: > - arg = code[i] + code[i+1]*256 + extended_arg > - extended_arg = 0 > - i = i+2 > - if op == EXTENDED_ARG: > - extended_arg = arg*65536 > - yield (offset, op, arg) > + arg = code[i+1] | extended_arg > + extended_arg = (arg << 8) if op == EXTENDED_ARG else 0 > + else: > + arg = None > + yield (i, op, arg) > > def findlabels(code): > """Detect all offsets in a byte code which are jump targets. > @@ -379,14 +370,14 @@ > labels = [] > for offset, op, arg in _unpack_opargs(code): > if arg is not None: > - label = -1 > if op in hasjrel: > - label = offset + 3 + arg > + label = offset + 2 + arg > elif op in hasjabs: > label = arg > - if label >= 0: > - if label not in labels: > - labels.append(label) > + else: > + continue > + if label not in labels: > + labels.append(label) > return labels > > def findlinestarts(code): > diff --git a/Lib/importlib/_bootstrap_external.py > b/Lib/importlib/_bootstrap_external.py > --- a/Lib/importlib/_bootstrap_external.py > +++ b/Lib/importlib/_bootstrap_external.py > @@ -225,6 +225,7 @@ > # Python 3.5b2 3350 (add GET_YIELD_FROM_ITER opcode #24400) > # Python 3.6a0 3360 (add FORMAT_VALUE opcode #25483 > # Python 3.6a0 3361 (lineno delta of code.co_lnotab becomes signed) > +# Python 3.6a0 3370 (16 bit wordcode) > # > # MAGIC must change whenever the bytecode emitted by the compiler may no > # longer be understood by older implementations of the eval loop (usually > @@ -233,7 +234,7 @@ > # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array > # in PC/launcher.c must also be updated. > > -MAGIC_NUMBER = (3361).to_bytes(2, 'little') + b'\r\n' > +MAGIC_NUMBER = (3370).to_bytes(2, 'little') + b'\r\n' > _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c > > _PYCACHE = '__pycache__' > diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py > --- a/Lib/test/test_dis.py > +++ b/Lib/test/test_dis.py > @@ -40,41 +40,41 @@ > > dis_c_instance_method = """\ > %3d 0 LOAD_FAST 1 (x) > - 3 LOAD_CONST 1 (1) > - 6 COMPARE_OP 2 (==) > - 9 LOAD_FAST 0 (self) > - 12 STORE_ATTR 0 (x) > - 15 LOAD_CONST 0 (None) > - 18 RETURN_VALUE > + 2 LOAD_CONST 1 (1) > + 4 COMPARE_OP 2 (==) > + 6 LOAD_FAST 0 (self) > + 8 STORE_ATTR 0 (x) > + 10 LOAD_CONST 0 (None) > + 12 RETURN_VALUE > """ % (_C.__init__.__code__.co_firstlineno + 1,) > > dis_c_instance_method_bytes = """\ > 0 LOAD_FAST 1 (1) > - 3 LOAD_CONST 1 (1) > - 6 COMPARE_OP 2 (==) > - 9 LOAD_FAST 0 (0) > - 12 STORE_ATTR 0 (0) > - 15 LOAD_CONST 0 (0) > - 18 RETURN_VALUE > + 2 LOAD_CONST 1 (1) > + 4 COMPARE_OP 2 (==) > + 6 LOAD_FAST 0 (0) > + 8 STORE_ATTR 0 (0) > + 10 LOAD_CONST 0 (0) > + 12 RETURN_VALUE > """ > > dis_c_class_method = """\ > %3d 0 LOAD_FAST 1 (x) > - 3 LOAD_CONST 1 (1) > - 6 COMPARE_OP 2 (==) > - 9 LOAD_FAST 0 (cls) > - 12 STORE_ATTR 0 (x) > - 15 LOAD_CONST 0 (None) > - 18 RETURN_VALUE > + 2 LOAD_CONST 1 (1) > + 4 COMPARE_OP 2 (==) > + 6 LOAD_FAST 0 (cls) > + 8 STORE_ATTR 0 (x) > + 10 LOAD_CONST 0 (None) > + 12 RETURN_VALUE > """ % (_C.cm.__code__.co_firstlineno + 2,) > > dis_c_static_method = """\ > %3d 0 LOAD_FAST 0 (x) > - 3 LOAD_CONST 1 (1) > - 6 COMPARE_OP 2 (==) > - 9 STORE_FAST 0 (x) > - 12 LOAD_CONST 0 (None) > - 15 RETURN_VALUE > + 2 LOAD_CONST 1 (1) > + 4 COMPARE_OP 2 (==) > + 6 STORE_FAST 0 (x) > + 8 LOAD_CONST 0 (None) > + 10 RETURN_VALUE > """ % (_C.sm.__code__.co_firstlineno + 2,) > > # Class disassembling info has an extra newline at end. > @@ -95,23 +95,23 @@ > > dis_f = """\ > %3d 0 LOAD_GLOBAL 0 (print) > - 3 LOAD_FAST 0 (a) > - 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > - 9 POP_TOP > + 2 LOAD_FAST 0 (a) > + 4 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > + 6 POP_TOP > > -%3d 10 LOAD_CONST 1 (1) > - 13 RETURN_VALUE > +%3d 8 LOAD_CONST 1 (1) > + 10 RETURN_VALUE > """ % (_f.__code__.co_firstlineno + 1, > _f.__code__.co_firstlineno + 2) > > > dis_f_co_code = """\ > 0 LOAD_GLOBAL 0 (0) > - 3 LOAD_FAST 0 (0) > - 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > - 9 POP_TOP > - 10 LOAD_CONST 1 (1) > - 13 RETURN_VALUE > + 2 LOAD_FAST 0 (0) > + 4 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > + 6 POP_TOP > + 8 LOAD_CONST 1 (1) > + 10 RETURN_VALUE > """ > > > @@ -121,20 +121,20 @@ > pass > > dis_bug708901 = """\ > -%3d 0 SETUP_LOOP 23 (to 26) > - 3 LOAD_GLOBAL 0 (range) > - 6 LOAD_CONST 1 (1) > +%3d 0 SETUP_LOOP 18 (to 20) > + 2 LOAD_GLOBAL 0 (range) > + 4 LOAD_CONST 1 (1) > > -%3d 9 LOAD_CONST 2 (10) > - 12 CALL_FUNCTION 2 (2 positional, 0 keyword pair) > - 15 GET_ITER > - >> 16 FOR_ITER 6 (to 25) > - 19 STORE_FAST 0 (res) > +%3d 6 LOAD_CONST 2 (10) > + 8 CALL_FUNCTION 2 (2 positional, 0 keyword pair) > + 10 GET_ITER > + >> 12 FOR_ITER 4 (to 18) > + 14 STORE_FAST 0 (res) > > -%3d 22 JUMP_ABSOLUTE 16 > - >> 25 POP_BLOCK > - >> 26 LOAD_CONST 0 (None) > - 29 RETURN_VALUE > +%3d 16 JUMP_ABSOLUTE 12 > + >> 18 POP_BLOCK > + >> 20 LOAD_CONST 0 (None) > + 22 RETURN_VALUE > """ % (bug708901.__code__.co_firstlineno + 1, > bug708901.__code__.co_firstlineno + 2, > bug708901.__code__.co_firstlineno + 3) > @@ -147,22 +147,22 @@ > > dis_bug1333982 = """\ > %3d 0 LOAD_CONST 1 (0) > - 3 POP_JUMP_IF_TRUE 35 > - 6 LOAD_GLOBAL 0 (AssertionError) > - 9 LOAD_CONST 2 ( at > 0x..., file "%s", line %d>) > - 12 LOAD_CONST 3 > ('bug1333982..') > - 15 MAKE_FUNCTION 0 > - 18 LOAD_FAST 0 (x) > - 21 GET_ITER > + 2 POP_JUMP_IF_TRUE 26 > + 4 LOAD_GLOBAL 0 (AssertionError) > + 6 LOAD_CONST 2 ( at > 0x..., file "%s", line %d>) > + 8 LOAD_CONST 3 > ('bug1333982..') > + 10 MAKE_FUNCTION 0 > + 12 LOAD_FAST 0 (x) > + 14 GET_ITER > + 16 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > + > +%3d 18 LOAD_CONST 4 (1) > + 20 BINARY_ADD > 22 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > + 24 RAISE_VARARGS 1 > > -%3d 25 LOAD_CONST 4 (1) > - 28 BINARY_ADD > - 29 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > - 32 RAISE_VARARGS 1 > - > -%3d >> 35 LOAD_CONST 0 (None) > - 38 RETURN_VALUE > +%3d >> 26 LOAD_CONST 0 (None) > + 28 RETURN_VALUE > """ % (bug1333982.__code__.co_firstlineno + 1, > __file__, > bug1333982.__code__.co_firstlineno + 1, > @@ -171,19 +171,19 @@ > > _BIG_LINENO_FORMAT = """\ > %3d 0 LOAD_GLOBAL 0 (spam) > - 3 POP_TOP > + 2 POP_TOP > 4 LOAD_CONST 0 (None) > - 7 RETURN_VALUE > + 6 RETURN_VALUE > """ > > dis_module_expected_results = """\ > Disassembly of f: > 4 0 LOAD_CONST 0 (None) > - 3 RETURN_VALUE > + 2 RETURN_VALUE > > Disassembly of g: > 5 0 LOAD_CONST 0 (None) > - 3 RETURN_VALUE > + 2 RETURN_VALUE > > """ > > @@ -191,20 +191,20 @@ > > dis_expr_str = """\ > 1 0 LOAD_NAME 0 (x) > - 3 LOAD_CONST 0 (1) > - 6 BINARY_ADD > - 7 RETURN_VALUE > + 2 LOAD_CONST 0 (1) > + 4 BINARY_ADD > + 6 RETURN_VALUE > """ > > simple_stmt_str = "x = x + 1" > > dis_simple_stmt_str = """\ > 1 0 LOAD_NAME 0 (x) > - 3 LOAD_CONST 0 (1) > - 6 BINARY_ADD > - 7 STORE_NAME 0 (x) > - 10 LOAD_CONST 1 (None) > - 13 RETURN_VALUE > + 2 LOAD_CONST 0 (1) > + 4 BINARY_ADD > + 6 STORE_NAME 0 (x) > + 8 LOAD_CONST 1 (None) > + 10 RETURN_VALUE > """ > > compound_stmt_str = """\ > @@ -215,54 +215,54 @@ > > dis_compound_stmt_str = """\ > 1 0 LOAD_CONST 0 (0) > - 3 STORE_NAME 0 (x) > + 2 STORE_NAME 0 (x) > > - 2 6 SETUP_LOOP 14 (to 23) > + 2 4 SETUP_LOOP 12 (to 18) > > - 3 >> 9 LOAD_NAME 0 (x) > - 12 LOAD_CONST 1 (1) > - 15 INPLACE_ADD > - 16 STORE_NAME 0 (x) > - 19 JUMP_ABSOLUTE 9 > - 22 POP_BLOCK > - >> 23 LOAD_CONST 2 (None) > - 26 RETURN_VALUE > + 3 >> 6 LOAD_NAME 0 (x) > + 8 LOAD_CONST 1 (1) > + 10 INPLACE_ADD > + 12 STORE_NAME 0 (x) > + 14 JUMP_ABSOLUTE 6 > + 16 POP_BLOCK > + >> 18 LOAD_CONST 2 (None) > + 20 RETURN_VALUE > """ > > dis_traceback = """\ > -%3d 0 SETUP_EXCEPT 12 (to 15) > +%3d 0 SETUP_EXCEPT 12 (to 14) > > -%3d 3 LOAD_CONST 1 (1) > - 6 LOAD_CONST 2 (0) > - --> 9 BINARY_TRUE_DIVIDE > - 10 POP_TOP > - 11 POP_BLOCK > - 12 JUMP_FORWARD 46 (to 61) > +%3d 2 LOAD_CONST 1 (1) > + 4 LOAD_CONST 2 (0) > + --> 6 BINARY_TRUE_DIVIDE > + 8 POP_TOP > + 10 POP_BLOCK > + 12 JUMP_FORWARD 40 (to 54) > > -%3d >> 15 DUP_TOP > +%3d >> 14 DUP_TOP > 16 LOAD_GLOBAL 0 (Exception) > - 19 COMPARE_OP 10 (exception match) > - 22 POP_JUMP_IF_FALSE 60 > - 25 POP_TOP > - 26 STORE_FAST 0 (e) > - 29 POP_TOP > - 30 SETUP_FINALLY 14 (to 47) > + 18 COMPARE_OP 10 (exception match) > + 20 POP_JUMP_IF_FALSE 52 > + 22 POP_TOP > + 24 STORE_FAST 0 (e) > + 26 POP_TOP > + 28 SETUP_FINALLY 12 (to 42) > > -%3d 33 LOAD_FAST 0 (e) > - 36 LOAD_ATTR 1 (__traceback__) > - 39 STORE_FAST 1 (tb) > - 42 POP_BLOCK > - 43 POP_EXCEPT > - 44 LOAD_CONST 0 (None) > - >> 47 LOAD_CONST 0 (None) > - 50 STORE_FAST 0 (e) > - 53 DELETE_FAST 0 (e) > - 56 END_FINALLY > - 57 JUMP_FORWARD 1 (to 61) > - >> 60 END_FINALLY > +%3d 30 LOAD_FAST 0 (e) > + 32 LOAD_ATTR 1 (__traceback__) > + 34 STORE_FAST 1 (tb) > + 36 POP_BLOCK > + 38 POP_EXCEPT > + 40 LOAD_CONST 0 (None) > + >> 42 LOAD_CONST 0 (None) > + 44 STORE_FAST 0 (e) > + 46 DELETE_FAST 0 (e) > + 48 END_FINALLY > + 50 JUMP_FORWARD 2 (to 54) > + >> 52 END_FINALLY > > -%3d >> 61 LOAD_FAST 1 (tb) > - 64 RETURN_VALUE > +%3d >> 54 LOAD_FAST 1 (tb) > + 56 RETURN_VALUE > """ % (TRACEBACK_CODE.co_firstlineno + 1, > TRACEBACK_CODE.co_firstlineno + 2, > TRACEBACK_CODE.co_firstlineno + 3, > @@ -650,170 +650,170 @@ > Instruction = dis.Instruction > expected_opinfo_outer = [ > Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=3, > argrepr='3', offset=0, starts_line=2, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, > argrepr='4', offset=3, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CLOSURE', opcode=135, arg=0, argval='a', > argrepr='a', offset=6, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CLOSURE', opcode=135, arg=1, argval='b', > argrepr='b', offset=9, starts_line=None, is_jump_target=False), > - Instruction(opname='BUILD_TUPLE', opcode=102, arg=2, argval=2, > argrepr='', offset=12, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=3, > argval=code_object_f, argrepr=repr(code_object_f), offset=15, > starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=4, > argval='outer..f', argrepr="'outer..f'", offset=18, > starts_line=None, is_jump_target=False), > - Instruction(opname='MAKE_CLOSURE', opcode=134, arg=2, argval=2, > argrepr='', offset=21, starts_line=None, is_jump_target=False), > - Instruction(opname='STORE_FAST', opcode=125, arg=2, argval='f', > argrepr='f', offset=24, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print', > argrepr='print', offset=27, starts_line=7, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='a', > argrepr='a', offset=30, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='b', > argrepr='b', offset=33, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval='', > argrepr="''", offset=36, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval=1, > argrepr='1', offset=39, starts_line=None, is_jump_target=False), > - Instruction(opname='BUILD_LIST', opcode=103, arg=0, argval=0, > argrepr='', offset=42, starts_line=None, is_jump_target=False), > - Instruction(opname='BUILD_MAP', opcode=105, arg=0, argval=0, > argrepr='', offset=45, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval='Hello > world!', argrepr="'Hello world!'", offset=48, starts_line=None, > is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=7, argval=7, > argrepr='7 positional, 0 keyword pair', offset=51, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=54, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=2, argval='f', > argrepr='f', offset=55, starts_line=8, is_jump_target=False), > - Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=58, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, > argrepr='4', offset=2, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CLOSURE', opcode=135, arg=0, argval='a', > argrepr='a', offset=4, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CLOSURE', opcode=135, arg=1, argval='b', > argrepr='b', offset=6, starts_line=None, is_jump_target=False), > + Instruction(opname='BUILD_TUPLE', opcode=102, arg=2, argval=2, > argrepr='', offset=8, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=3, > argval=code_object_f, argrepr=repr(code_object_f), offset=10, > starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=4, > argval='outer..f', argrepr="'outer..f'", offset=12, > starts_line=None, is_jump_target=False), > + Instruction(opname='MAKE_CLOSURE', opcode=134, arg=2, argval=2, > argrepr='', offset=14, starts_line=None, is_jump_target=False), > + Instruction(opname='STORE_FAST', opcode=125, arg=2, argval='f', > argrepr='f', offset=16, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print', > argrepr='print', offset=18, starts_line=7, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='a', > argrepr='a', offset=20, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='b', > argrepr='b', offset=22, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval='', > argrepr="''", offset=24, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval=1, > argrepr='1', offset=26, starts_line=None, is_jump_target=False), > + Instruction(opname='BUILD_LIST', opcode=103, arg=0, argval=0, > argrepr='', offset=28, starts_line=None, is_jump_target=False), > + Instruction(opname='BUILD_MAP', opcode=105, arg=0, argval=0, > argrepr='', offset=30, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval='Hello > world!', argrepr="'Hello world!'", offset=32, starts_line=None, > is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=7, argval=7, > argrepr='7 positional, 0 keyword pair', offset=34, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=36, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=2, argval='f', > argrepr='f', offset=38, starts_line=8, is_jump_target=False), > + Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=40, starts_line=None, is_jump_target=False), > ] > > expected_opinfo_f = [ > Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=5, > argrepr='5', offset=0, starts_line=3, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=6, > argrepr='6', offset=3, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CLOSURE', opcode=135, arg=2, argval='a', > argrepr='a', offset=6, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CLOSURE', opcode=135, arg=3, argval='b', > argrepr='b', offset=9, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CLOSURE', opcode=135, arg=0, argval='c', > argrepr='c', offset=12, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CLOSURE', opcode=135, arg=1, argval='d', > argrepr='d', offset=15, starts_line=None, is_jump_target=False), > - Instruction(opname='BUILD_TUPLE', opcode=102, arg=4, argval=4, > argrepr='', offset=18, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=3, > argval=code_object_inner, argrepr=repr(code_object_inner), offset=21, > starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=4, > argval='outer..f..inner', > argrepr="'outer..f..inner'", offset=24, starts_line=None, > is_jump_target=False), > - Instruction(opname='MAKE_CLOSURE', opcode=134, arg=2, argval=2, > argrepr='', offset=27, starts_line=None, is_jump_target=False), > - Instruction(opname='STORE_FAST', opcode=125, arg=2, argval='inner', > argrepr='inner', offset=30, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print', > argrepr='print', offset=33, starts_line=5, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=2, argval='a', > argrepr='a', offset=36, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=3, argval='b', > argrepr='b', offset=39, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='c', > argrepr='c', offset=42, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='d', > argrepr='d', offset=45, starts_line=None, is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=4, argval=4, > argrepr='4 positional, 0 keyword pair', offset=48, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=51, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=2, argval='inner', > argrepr='inner', offset=52, starts_line=6, is_jump_target=False), > - Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=55, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=6, > argrepr='6', offset=2, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CLOSURE', opcode=135, arg=2, argval='a', > argrepr='a', offset=4, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CLOSURE', opcode=135, arg=3, argval='b', > argrepr='b', offset=6, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CLOSURE', opcode=135, arg=0, argval='c', > argrepr='c', offset=8, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CLOSURE', opcode=135, arg=1, argval='d', > argrepr='d', offset=10, starts_line=None, is_jump_target=False), > + Instruction(opname='BUILD_TUPLE', opcode=102, arg=4, argval=4, > argrepr='', offset=12, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=3, > argval=code_object_inner, argrepr=repr(code_object_inner), offset=14, > starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=4, > argval='outer..f..inner', > argrepr="'outer..f..inner'", offset=16, starts_line=None, > is_jump_target=False), > + Instruction(opname='MAKE_CLOSURE', opcode=134, arg=2, argval=2, > argrepr='', offset=18, starts_line=None, is_jump_target=False), > + Instruction(opname='STORE_FAST', opcode=125, arg=2, argval='inner', > argrepr='inner', offset=20, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print', > argrepr='print', offset=22, starts_line=5, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=2, argval='a', > argrepr='a', offset=24, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=3, argval='b', > argrepr='b', offset=26, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='c', > argrepr='c', offset=28, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='d', > argrepr='d', offset=30, starts_line=None, is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=4, argval=4, > argrepr='4 positional, 0 keyword pair', offset=32, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=34, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=2, argval='inner', > argrepr='inner', offset=36, starts_line=6, is_jump_target=False), > + Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=38, starts_line=None, is_jump_target=False), > ] > > expected_opinfo_inner = [ > Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print', > argrepr='print', offset=0, starts_line=4, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='a', > argrepr='a', offset=3, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='b', > argrepr='b', offset=6, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=2, argval='c', > argrepr='c', offset=9, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_DEREF', opcode=136, arg=3, argval='d', > argrepr='d', offset=12, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='e', > argrepr='e', offset=15, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=1, argval='f', > argrepr='f', offset=18, starts_line=None, is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=6, argval=6, > argrepr='6 positional, 0 keyword pair', offset=21, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=24, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, > argrepr='None', offset=25, starts_line=None, is_jump_target=False), > - Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=28, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='a', > argrepr='a', offset=2, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='b', > argrepr='b', offset=4, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=2, argval='c', > argrepr='c', offset=6, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_DEREF', opcode=136, arg=3, argval='d', > argrepr='d', offset=8, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='e', > argrepr='e', offset=10, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=1, argval='f', > argrepr='f', offset=12, starts_line=None, is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=6, argval=6, > argrepr='6 positional, 0 keyword pair', offset=14, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=16, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, > argrepr='None', offset=18, starts_line=None, is_jump_target=False), > + Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=20, starts_line=None, is_jump_target=False), > ] > > expected_opinfo_jumpy = [ > - Instruction(opname='SETUP_LOOP', opcode=120, arg=68, argval=71, > argrepr='to 71', offset=0, starts_line=3, is_jump_target=False), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='range', > argrepr='range', offset=3, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=10, > argrepr='10', offset=6, starts_line=None, is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=9, starts_line=None, > is_jump_target=False), > - Instruction(opname='GET_ITER', opcode=68, arg=None, argval=None, > argrepr='', offset=12, starts_line=None, is_jump_target=False), > - Instruction(opname='FOR_ITER', opcode=93, arg=44, argval=60, > argrepr='to 60', offset=13, starts_line=None, is_jump_target=True), > - Instruction(opname='STORE_FAST', opcode=125, arg=0, argval='i', > argrepr='i', offset=16, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=19, starts_line=4, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=22, starts_line=None, is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=25, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=28, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=29, starts_line=5, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, > argrepr='4', offset=32, starts_line=None, is_jump_target=False), > - Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<', > argrepr='<', offset=35, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=44, argval=44, > argrepr='', offset=38, starts_line=None, is_jump_target=False), > - Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=13, argval=13, > argrepr='', offset=41, starts_line=6, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=44, starts_line=7, is_jump_target=True), > - Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6, > argrepr='6', offset=47, starts_line=None, is_jump_target=False), > - Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>', > argrepr='>', offset=50, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=13, argval=13, > argrepr='', offset=53, starts_line=None, is_jump_target=False), > - Instruction(opname='BREAK_LOOP', opcode=80, arg=None, argval=None, > argrepr='', offset=56, starts_line=8, is_jump_target=False), > - Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=13, argval=13, > argrepr='', offset=57, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=60, starts_line=None, is_jump_target=True), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=61, starts_line=10, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=4, argval='I can haz > else clause?', argrepr="'I can haz else clause?'", offset=64, > starts_line=None, is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=67, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=70, starts_line=None, is_jump_target=False), > - Instruction(opname='SETUP_LOOP', opcode=120, arg=68, argval=142, > argrepr='to 142', offset=71, starts_line=11, is_jump_target=True), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=74, starts_line=None, is_jump_target=True), > - Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=131, > argval=131, argrepr='', offset=77, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=80, starts_line=12, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=83, starts_line=None, is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=86, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=89, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=90, starts_line=13, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1, > argrepr='1', offset=93, starts_line=None, is_jump_target=False), > - Instruction(opname='INPLACE_SUBTRACT', opcode=56, arg=None, > argval=None, argrepr='', offset=96, starts_line=None, is_jump_target=False), > - Instruction(opname='STORE_FAST', opcode=125, arg=0, argval='i', > argrepr='i', offset=97, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=100, starts_line=14, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6, > argrepr='6', offset=103, starts_line=None, is_jump_target=False), > - Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>', > argrepr='>', offset=106, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=115, > argval=115, argrepr='', offset=109, starts_line=None, is_jump_target=False), > - Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=74, argval=74, > argrepr='', offset=112, starts_line=15, is_jump_target=False), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=115, starts_line=16, is_jump_target=True), > - Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, > argrepr='4', offset=118, starts_line=None, is_jump_target=False), > - Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<', > argrepr='<', offset=121, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=74, argval=74, > argrepr='', offset=124, starts_line=None, is_jump_target=False), > - Instruction(opname='BREAK_LOOP', opcode=80, arg=None, argval=None, > argrepr='', offset=127, starts_line=17, is_jump_target=False), > - Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=74, argval=74, > argrepr='', offset=128, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=131, starts_line=None, is_jump_target=True), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=132, starts_line=19, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval='Who let > lolcatz into this test suite?', argrepr="'Who let lolcatz into this test > suite?'", offset=135, starts_line=None, is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=138, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=141, starts_line=None, is_jump_target=False), > - Instruction(opname='SETUP_FINALLY', opcode=122, arg=73, argval=218, > argrepr='to 218', offset=142, starts_line=20, is_jump_target=True), > - Instruction(opname='SETUP_EXCEPT', opcode=121, arg=12, argval=160, > argrepr='to 160', offset=145, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1, > argrepr='1', offset=148, starts_line=21, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval=0, > argrepr='0', offset=151, starts_line=None, is_jump_target=False), > - Instruction(opname='BINARY_TRUE_DIVIDE', opcode=27, arg=None, > argval=None, argrepr='', offset=154, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=155, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=156, starts_line=None, is_jump_target=False), > - Instruction(opname='JUMP_FORWARD', opcode=110, arg=28, argval=188, > argrepr='to 188', offset=157, starts_line=None, is_jump_target=False), > - Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None, > argrepr='', offset=160, starts_line=22, is_jump_target=True), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=2, > argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=161, > starts_line=None, is_jump_target=False), > - Instruction(opname='COMPARE_OP', opcode=107, arg=10, argval='exception > match', argrepr='exception match', offset=164, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=187, > argval=187, argrepr='', offset=167, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=170, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=171, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=172, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=173, starts_line=23, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=8, argval='Here we go, > here we go, here we go...', argrepr="'Here we go, here we go, here we > go...'", offset=176, starts_line=None, is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=179, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=182, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, > argrepr='', offset=183, starts_line=None, is_jump_target=False), > - Instruction(opname='JUMP_FORWARD', opcode=110, arg=27, argval=214, > argrepr='to 214', offset=184, starts_line=None, is_jump_target=False), > - Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None, > argrepr='', offset=187, starts_line=None, is_jump_target=True), > - Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=188, starts_line=25, is_jump_target=True), > - Instruction(opname='SETUP_WITH', opcode=143, arg=17, argval=211, > argrepr='to 211', offset=191, starts_line=None, is_jump_target=False), > - Instruction(opname='STORE_FAST', opcode=125, arg=1, argval='dodgy', > argrepr='dodgy', offset=194, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=197, starts_line=26, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=9, argval='Never reach > this', argrepr="'Never reach this'", offset=200, starts_line=None, > is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=203, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=206, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=207, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, > argrepr='None', offset=208, starts_line=None, is_jump_target=False), > - Instruction(opname='WITH_CLEANUP_START', opcode=81, arg=None, > argval=None, argrepr='', offset=211, starts_line=None, is_jump_target=True), > - Instruction(opname='WITH_CLEANUP_FINISH', opcode=82, arg=None, > argval=None, argrepr='', offset=212, starts_line=None, > is_jump_target=False), > - Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None, > argrepr='', offset=213, starts_line=None, is_jump_target=False), > - Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=214, starts_line=None, is_jump_target=True), > - Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, > argrepr='None', offset=215, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=218, starts_line=28, is_jump_target=True), > - Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now > we're done", argrepr='"OK, now we\'re done"', offset=221, starts_line=None, > is_jump_target=False), > - Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=224, starts_line=None, > is_jump_target=False), > - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=227, starts_line=None, is_jump_target=False), > - Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None, > argrepr='', offset=228, starts_line=None, is_jump_target=False), > - Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, > argrepr='None', offset=229, starts_line=None, is_jump_target=False), > - Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=232, starts_line=None, is_jump_target=False), > + Instruction(opname='SETUP_LOOP', opcode=120, arg=52, argval=54, > argrepr='to 54', offset=0, starts_line=3, is_jump_target=False), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='range', > argrepr='range', offset=2, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=10, > argrepr='10', offset=4, starts_line=None, is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=6, starts_line=None, > is_jump_target=False), > + Instruction(opname='GET_ITER', opcode=68, arg=None, argval=None, > argrepr='', offset=8, starts_line=None, is_jump_target=False), > + Instruction(opname='FOR_ITER', opcode=93, arg=32, argval=44, > argrepr='to 44', offset=10, starts_line=None, is_jump_target=True), > + Instruction(opname='STORE_FAST', opcode=125, arg=0, argval='i', > argrepr='i', offset=12, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=14, starts_line=4, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=16, starts_line=None, is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=18, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=20, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=22, starts_line=5, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, > argrepr='4', offset=24, starts_line=None, is_jump_target=False), > + Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<', > argrepr='<', offset=26, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=32, argval=32, > argrepr='', offset=28, starts_line=None, is_jump_target=False), > + Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=10, argval=10, > argrepr='', offset=30, starts_line=6, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=32, starts_line=7, is_jump_target=True), > + Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6, > argrepr='6', offset=34, starts_line=None, is_jump_target=False), > + Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>', > argrepr='>', offset=36, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=10, argval=10, > argrepr='', offset=38, starts_line=None, is_jump_target=False), > + Instruction(opname='BREAK_LOOP', opcode=80, arg=None, argval=None, > argrepr='', offset=40, starts_line=8, is_jump_target=False), > + Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=10, argval=10, > argrepr='', offset=42, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=44, starts_line=None, is_jump_target=True), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=46, starts_line=10, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=4, argval='I can haz > else clause?', argrepr="'I can haz else clause?'", offset=48, > starts_line=None, is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=50, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=52, starts_line=None, is_jump_target=False), > + Instruction(opname='SETUP_LOOP', opcode=120, arg=52, argval=108, > argrepr='to 108', offset=54, starts_line=11, is_jump_target=True), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=56, starts_line=None, is_jump_target=True), > + Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=98, argval=98, > argrepr='', offset=58, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=60, starts_line=12, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=62, starts_line=None, is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=64, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=66, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=68, starts_line=13, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1, > argrepr='1', offset=70, starts_line=None, is_jump_target=False), > + Instruction(opname='INPLACE_SUBTRACT', opcode=56, arg=None, > argval=None, argrepr='', offset=72, starts_line=None, is_jump_target=False), > + Instruction(opname='STORE_FAST', opcode=125, arg=0, argval='i', > argrepr='i', offset=74, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=76, starts_line=14, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6, > argrepr='6', offset=78, starts_line=None, is_jump_target=False), > + Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>', > argrepr='>', offset=80, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=86, argval=86, > argrepr='', offset=82, starts_line=None, is_jump_target=False), > + Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=56, argval=56, > argrepr='', offset=84, starts_line=15, is_jump_target=False), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=86, starts_line=16, is_jump_target=True), > + Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, > argrepr='4', offset=88, starts_line=None, is_jump_target=False), > + Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<', > argrepr='<', offset=90, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=56, argval=56, > argrepr='', offset=92, starts_line=None, is_jump_target=False), > + Instruction(opname='BREAK_LOOP', opcode=80, arg=None, argval=None, > argrepr='', offset=94, starts_line=17, is_jump_target=False), > + Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=56, argval=56, > argrepr='', offset=96, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=98, starts_line=None, is_jump_target=True), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=100, starts_line=19, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval='Who let > lolcatz into this test suite?', argrepr="'Who let lolcatz into this test > suite?'", offset=102, starts_line=None, is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=104, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=106, starts_line=None, is_jump_target=False), > + Instruction(opname='SETUP_FINALLY', opcode=122, arg=70, argval=180, > argrepr='to 180', offset=108, starts_line=20, is_jump_target=True), > + Instruction(opname='SETUP_EXCEPT', opcode=121, arg=12, argval=124, > argrepr='to 124', offset=110, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1, > argrepr='1', offset=112, starts_line=21, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval=0, > argrepr='0', offset=114, starts_line=None, is_jump_target=False), > + Instruction(opname='BINARY_TRUE_DIVIDE', opcode=27, arg=None, > argval=None, argrepr='', offset=116, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=118, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=120, starts_line=None, is_jump_target=False), > + Instruction(opname='JUMP_FORWARD', opcode=110, arg=28, argval=152, > argrepr='to 152', offset=122, starts_line=None, is_jump_target=False), > + Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None, > argrepr='', offset=124, starts_line=22, is_jump_target=True), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=2, > argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=126, > starts_line=None, is_jump_target=False), > + Instruction(opname='COMPARE_OP', opcode=107, arg=10, argval='exception > match', argrepr='exception match', offset=128, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=150, > argval=150, argrepr='', offset=130, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=132, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=134, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=136, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=138, starts_line=23, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=8, argval='Here we go, > here we go, here we go...', argrepr="'Here we go, here we go, here we > go...'", offset=140, starts_line=None, is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=142, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=144, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, > argrepr='', offset=146, starts_line=None, is_jump_target=False), > + Instruction(opname='JUMP_FORWARD', opcode=110, arg=26, argval=176, > argrepr='to 176', offset=148, starts_line=None, is_jump_target=False), > + Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None, > argrepr='', offset=150, starts_line=None, is_jump_target=True), > + Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', > argrepr='i', offset=152, starts_line=25, is_jump_target=True), > + Instruction(opname='SETUP_WITH', opcode=143, arg=14, argval=170, > argrepr='to 170', offset=154, starts_line=None, is_jump_target=False), > + Instruction(opname='STORE_FAST', opcode=125, arg=1, argval='dodgy', > argrepr='dodgy', offset=156, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=158, starts_line=26, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=9, argval='Never reach > this', argrepr="'Never reach this'", offset=160, starts_line=None, > is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=162, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=164, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=166, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, > argrepr='None', offset=168, starts_line=None, is_jump_target=False), > + Instruction(opname='WITH_CLEANUP_START', opcode=81, arg=None, > argval=None, argrepr='', offset=170, starts_line=None, is_jump_target=True), > + Instruction(opname='WITH_CLEANUP_FINISH', opcode=82, arg=None, > argval=None, argrepr='', offset=172, starts_line=None, > is_jump_target=False), > + Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None, > argrepr='', offset=174, starts_line=None, is_jump_target=False), > + Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, > argrepr='', offset=176, starts_line=None, is_jump_target=True), > + Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, > argrepr='None', offset=178, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', > argrepr='print', offset=180, starts_line=28, is_jump_target=True), > + Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now > we're done", argrepr='"OK, now we\'re done"', offset=182, starts_line=None, > is_jump_target=False), > + Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, > argrepr='1 positional, 0 keyword pair', offset=184, starts_line=None, > is_jump_target=False), > + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, > argrepr='', offset=186, starts_line=None, is_jump_target=False), > + Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None, > argrepr='', offset=188, starts_line=None, is_jump_target=False), > + Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, > argrepr='None', offset=190, starts_line=None, is_jump_target=False), > + Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=192, starts_line=None, is_jump_target=False), > ] > > # One last piece of inspect fodder to check the default line number > handling > def simple(): pass > expected_opinfo_simple = [ > Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, > argrepr='None', offset=0, starts_line=simple.__code__.co_firstlineno, > is_jump_target=False), > - Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=3, starts_line=None, is_jump_target=False) > + Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, > argrepr='', offset=2, starts_line=None, is_jump_target=False) > ] > > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -10,6 +10,9 @@ > Core and Builtins > ----------------- > > +- Issue #26647: Python interpreter now uses 16-bit wordcode instead of > bytecode. > + Patch by Demur Rumed. > + > - Issue #23275: Allow assigning to an empty target list in round brackets: > () = iterable. > > diff --git a/Objects/frameobject.c b/Objects/frameobject.c > --- a/Objects/frameobject.c > +++ b/Objects/frameobject.c > @@ -189,7 +189,7 @@ > memset(blockstack, '\0', sizeof(blockstack)); > memset(in_finally, '\0', sizeof(in_finally)); > blockstack_top = 0; > - for (addr = 0; addr < code_len; addr++) { > + for (addr = 0; addr < code_len; addr += 2) { > unsigned char op = code[addr]; > switch (op) { > case SETUP_LOOP: > @@ -251,10 +251,6 @@ > } > } > } > - > - if (op >= HAVE_ARGUMENT) { > - addr += 2; > - } > } > > /* Verify that the blockstack tracking code didn't get lost. */ > @@ -277,7 +273,7 @@ > * can tell whether the jump goes into any blocks without coming out > * again - in that case we raise an exception below. */ > delta_iblock = 0; > - for (addr = min_addr; addr < max_addr; addr++) { > + for (addr = min_addr; addr < max_addr; addr += 2) { > unsigned char op = code[addr]; > switch (op) { > case SETUP_LOOP: > @@ -294,10 +290,6 @@ > } > > min_delta_iblock = Py_MIN(min_delta_iblock, delta_iblock); > - > - if (op >= HAVE_ARGUMENT) { > - addr += 2; > - } > } > > /* Derive the absolute iblock values from the deltas. */ > diff --git a/Objects/genobject.c b/Objects/genobject.c > --- a/Objects/genobject.c > +++ b/Objects/genobject.c > @@ -277,7 +277,7 @@ > PyObject *bytecode = f->f_code->co_code; > unsigned char *code = (unsigned char > *)PyBytes_AS_STRING(bytecode); > > - if (code[f->f_lasti + 1] != YIELD_FROM) > + if (code[f->f_lasti + 2] != YIELD_FROM) > return NULL; > yf = f->f_stacktop[-1]; > Py_INCREF(yf); > @@ -376,7 +376,7 @@ > assert(ret == yf); > Py_DECREF(ret); > /* Termination repetition of YIELD_FROM */ > - gen->gi_frame->f_lasti++; > + gen->gi_frame->f_lasti += 2; > if (_PyGen_FetchStopIterationValue(&val) == 0) { > ret = gen_send_ex(gen, val, 0, 0); > Py_DECREF(val); > diff --git a/PC/launcher.c b/PC/launcher.c > --- a/PC/launcher.c > +++ b/PC/launcher.c > @@ -1089,7 +1089,7 @@ > { 3190, 3230, L"3.3" }, > { 3250, 3310, L"3.4" }, > { 3320, 3350, L"3.5" }, > - { 3360, 3361, L"3.6" }, > + { 3360, 3370, L"3.6" }, > { 0 } > }; > > diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj > --- a/PCbuild/pythoncore.vcxproj > +++ b/PCbuild/pythoncore.vcxproj > @@ -209,6 +209,7 @@ > > > > + > > > > diff --git a/PCbuild/pythoncore.vcxproj.filters > b/PCbuild/pythoncore.vcxproj.filters > --- a/PCbuild/pythoncore.vcxproj.filters > +++ b/PCbuild/pythoncore.vcxproj.filters > @@ -420,6 +420,9 @@ > > Python > > + > + Python > + > > Python > > diff --git a/Python/ceval.c b/Python/ceval.c > --- a/Python/ceval.c > +++ b/Python/ceval.c > @@ -886,24 +886,10 @@ > /* Import the static jump table */ > #include "opcode_targets.h" > > -/* This macro is used when several opcodes defer to the same > implementation > - (e.g. SETUP_LOOP, SETUP_FINALLY) */ > -#define TARGET_WITH_IMPL(op, impl) \ > - TARGET_##op: \ > - opcode = op; \ > - if (HAS_ARG(op)) \ > - oparg = NEXTARG(); \ > - case op: \ > - goto impl; \ > - > #define TARGET(op) \ > TARGET_##op: \ > - opcode = op; \ > - if (HAS_ARG(op)) \ > - oparg = NEXTARG(); \ > case op: > > - > #define DISPATCH() \ > { \ > if (!_Py_atomic_load_relaxed(&eval_breaker)) { \ > @@ -917,7 +903,9 @@ > { \ > if (!lltrace && !_Py_TracingPossible) { \ > f->f_lasti = INSTR_OFFSET(); \ > - goto *opcode_targets[*next_instr++]; \ > + opcode = NEXTOP(); \ > + oparg = NEXTARG(); \ > + goto *opcode_targets[opcode]; \ > } \ > goto fast_next_opcode; \ > } > @@ -926,7 +914,9 @@ > { \ > if (!_Py_TracingPossible) { \ > f->f_lasti = INSTR_OFFSET(); \ > - goto *opcode_targets[*next_instr++]; \ > + opcode = NEXTOP(); \ > + oparg = NEXTARG(); \ > + goto *opcode_targets[opcode]; \ > } \ > goto fast_next_opcode; \ > } > @@ -935,10 +925,7 @@ > #else > #define TARGET(op) \ > case op: > -#define TARGET_WITH_IMPL(op, impl) \ > - /* silence compiler warnings about `impl` unused */ \ > - if (0) goto impl; \ > - case op: > + > #define DISPATCH() continue > #define FAST_DISPATCH() goto fast_next_opcod -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue May 24 13:06:00 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 24 May 2016 13:06:00 -0400 Subject: [Python-Dev] Removing the provisional label from pathlib In-Reply-To: References: <20160524095651.014e37c4.barry@wooz.org> Message-ID: On 5/24/2016 10:49 AM, Paul Moore wrote: > On 24 May 2016 at 15:11, Koos Zevenhoven wrote: >>> Please, no. We learned that lesson in Python 2.2.1 with True/False. >> >> What happened? True was included in 2.2.1 but not False?-). Anyway, I >> guess you are probably right, and "3.6->" is the way to go. Besides, >> Guido already wrote that in the first response. He did so because our policy rightly forbids changing os. > The history is at > http://python-history.blogspot.co.uk/2013/11/the-history-of-bool-true-and-false.html > but basically we added new *constants* for True/False in a point > release. Some people used them, and their code only worked on 2.2.1 > and above but not 2.2.0. Others were using variables called True and > False, and the 2.2.1 change broke their code (you could no longer > assign to those names). > > So essentially it ended up that people couldn't say "works on 2.2" and > had to distinguish between 2.2.0 and 2.2.1+. > > Python's strict backward compatibility rules basically stem from the > grief caused by that change. The policy was reinforced by its violation in changing the behavior of a StringIO method in 3.1.2 or something like that. For people affected (like me), the change resulted in a subtle bug that only manifested later. -- Terry Jan Reedy From christian at python.org Wed May 25 06:29:12 2016 From: christian at python.org (Christian Heimes) Date: Wed, 25 May 2016 12:29:12 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 Message-ID: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> Hi everybody, I have three hashing-related patches for Python 3.6 that are waiting for review. Altogether the three patches add ten new hash algorithms to the hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256), BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256). SHA-3 / SHAKE: https://bugs.python.org/issue16113 BLAKE2: https://bugs.python.org/issue26798 SHA512/224 / SHA512/256: https://bugs.python.org/issue26834 I like to push the patches during the sprints at PyCon. Please assist with reviews. Regards, Christian From leewangzhong+python at gmail.com Wed May 25 13:23:42 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 25 May 2016 13:23:42 -0400 Subject: [Python-Dev] devinabox has moved to GitHub In-Reply-To: References: Message-ID: Should these notes come with version requirements/minimums? "OS X users should be told to download XCode from the Apple App Store ahead of time." "If new contributors think they may be doing C development, suggest the use of LLVM + clang as this provides better error reporting than gcc." "For Windows users, ask them to download and install Visual Studio Community edition ahead of time." On Sun, May 8, 2016 at 4:41 PM, Brett Cannon wrote: > https://github.com/python/devinabox > > The single issue for devinabox has moved to its own issue tracker, so > there's no need to worry about those issues cluttering b.p.o in the future. > I have made the Python core team I created on GitHub last week have write > privileges and Nick and I as admins on the repository. I have also turned on > the CLA bot for the repository (FYI > https://github.com/python/the-knights-who-say-ni has that bot's code). > > I've asked Georg, Antoine, and Benjamin to tell me what I need to do to shut > off -- either by making it read-only or just deleting -- > hg.python.org/devinabox. > > Now that the migration has seriously begun, the next repos will be the peps > and devguide (slightly more complicated thanks to needing to update commands > for building their online versions). There's also the benchmarks repo, but > that might not get migrated if we start from scratch (see the speed@ ML > about that). > > As for cpython, I've been asked to talk about it at the language summit > where I will start a conversation about what the minimal feature set will > need to be to migrate. Once we have settled on what has to be in place to > migrate the cpython repo then we can start working on that TODO list. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/leewangzhong%2Bpython%40gmail.com > From brett at python.org Wed May 25 13:37:52 2016 From: brett at python.org (Brett Cannon) Date: Wed, 25 May 2016 17:37:52 +0000 Subject: [Python-Dev] devinabox has moved to GitHub In-Reply-To: References: Message-ID: On Wed, 25 May 2016 at 10:24 Franklin? Lee wrote: > Should these notes come with version requirements/minimums? > > "OS X users should be told to download XCode from the Apple App Store > ahead of time." > "If new contributors think they may be doing C development, suggest > the use of LLVM + clang as this provides better error reporting than > gcc." > "For Windows users, ask them to download and install Visual Studio > Community edition ahead of time." > If you want to submit a PR to say "the latest" that's fine, but I don't know if any of those tools really promote downloading older versions such that one has to worry about it. > > On Sun, May 8, 2016 at 4:41 PM, Brett Cannon wrote: > > https://github.com/python/devinabox > > > > The single issue for devinabox has moved to its own issue tracker, so > > there's no need to worry about those issues cluttering b.p.o in the > future. > > I have made the Python core team I created on GitHub last week have write > > privileges and Nick and I as admins on the repository. I have also > turned on > > the CLA bot for the repository (FYI > > https://github.com/python/the-knights-who-say-ni has that bot's code). > > > > I've asked Georg, Antoine, and Benjamin to tell me what I need to do to > shut > > off -- either by making it read-only or just deleting -- > > hg.python.org/devinabox. > > > > Now that the migration has seriously begun, the next repos will be the > peps > > and devguide (slightly more complicated thanks to needing to update > commands > > for building their online versions). There's also the benchmarks repo, > but > > that might not get migrated if we start from scratch (see the speed@ ML > > about that). > > > > As for cpython, I've been asked to talk about it at the language summit > > where I will start a conversation about what the minimal feature set will > > need to be to migrate. Once we have settled on what has to be in place > to > > migrate the cpython repo then we can start working on that TODO list. > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > > https://mail.python.org/mailman/options/python-dev/leewangzhong%2Bpython%40gmail.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Wed May 25 14:54:01 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 25 May 2016 14:54:01 -0400 Subject: [Python-Dev] devinabox has moved to GitHub In-Reply-To: References: Message-ID: It's just that I don't know whether any of them require particular versions. If you say the latest is fine, then okay. On Wed, May 25, 2016 at 1:37 PM, Brett Cannon wrote: > > > On Wed, 25 May 2016 at 10:24 Franklin? Lee > wrote: >> >> Should these notes come with version requirements/minimums? >> >> "OS X users should be told to download XCode from the Apple App Store >> ahead of time." >> "If new contributors think they may be doing C development, suggest >> the use of LLVM + clang as this provides better error reporting than >> gcc." >> "For Windows users, ask them to download and install Visual Studio >> Community edition ahead of time." > > > If you want to submit a PR to say "the latest" that's fine, but I don't know > if any of those tools really promote downloading older versions such that > one has to worry about it. > >> >> >> On Sun, May 8, 2016 at 4:41 PM, Brett Cannon wrote: >> > https://github.com/python/devinabox >> > >> > The single issue for devinabox has moved to its own issue tracker, so >> > there's no need to worry about those issues cluttering b.p.o in the >> > future. >> > I have made the Python core team I created on GitHub last week have >> > write >> > privileges and Nick and I as admins on the repository. I have also >> > turned on >> > the CLA bot for the repository (FYI >> > https://github.com/python/the-knights-who-say-ni has that bot's code). >> > >> > I've asked Georg, Antoine, and Benjamin to tell me what I need to do to >> > shut >> > off -- either by making it read-only or just deleting -- >> > hg.python.org/devinabox. >> > >> > Now that the migration has seriously begun, the next repos will be the >> > peps >> > and devguide (slightly more complicated thanks to needing to update >> > commands >> > for building their online versions). There's also the benchmarks repo, >> > but >> > that might not get migrated if we start from scratch (see the speed@ ML >> > about that). >> > >> > As for cpython, I've been asked to talk about it at the language summit >> > where I will start a conversation about what the minimal feature set >> > will >> > need to be to migrate. Once we have settled on what has to be in place >> > to >> > migrate the cpython repo then we can start working on that TODO list. >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > >> > https://mail.python.org/mailman/options/python-dev/leewangzhong%2Bpython%40gmail.com >> > From brett at snarky.ca Wed May 25 13:45:55 2016 From: brett at snarky.ca (Brett Cannon) Date: Wed, 25 May 2016 17:45:55 +0000 Subject: [Python-Dev] [Python-checkins] peps: Short subsection on annotating coroutines (Ivan L, #225). In-Reply-To: <20160524192004.14798.31017.1CC1E33A@psf.io> References: <20160524192004.14798.31017.1CC1E33A@psf.io> Message-ID: On Tue, 24 May 2016 at 12:20 guido.van.rossum wrote: > https://hg.python.org/peps/rev/50c3f5aefbb7 > changeset: 6341:50c3f5aefbb7 > user: Guido van Rossum > date: Tue May 24 12:18:54 2016 -0700 > summary: > Short subsection on annotating coroutines (Ivan L, #225). > > files: > pep-0484.txt | 34 ++++++++++++++++++++++++++++++++++ > 1 files changed, 34 insertions(+), 0 deletions(-) > > > diff --git a/pep-0484.txt b/pep-0484.txt > --- a/pep-0484.txt > +++ b/pep-0484.txt > @@ -1015,6 +1015,40 @@ > ellipsis, i.e. the above example is literally what you would write. > > > +Annotating generator functions and coroutines > +--------------------------------------------- > + > +The return type of generator functions can be annotated by > +the generic type ``Generator[yield_type, send_type, > +return_type]`` provided by ``typing.py`` module:: > + > + def echo_round() -> Generator[int, float, str]: > + res = yield > + while res: > + res = yield round(res) > + return 'OK' > + > +Coroutines introduced in PEP 492 are annotated with the same syntax as > +ordinary functions. However, the return type annotation corresponds to the > +type of ``await`` expression, not to the coroutine type:: > + > + async def spam(ignored: int) -> str: > + return 'spam' > + > + async def foo(): > + bar = await spam(42) # type: str > If the coroutine has multiple await expressions that return different types then do you simply use a Union to express that? -Brett > + > +The ``typing.py`` module also provides generic ABCs ``Awaitable``, > +``AsyncIterable``, and ``AsyncIterator`` for situations where more precise > +types cannot be specified:: > + > + def op() -> typing.Awaitable[str]: > + if cond: > + return spam(42) > + else: > + return asyncio.Future(...) > + > + > Compatibility with other uses of function annotations > ===================================================== > > > -- > Repository URL: https://hg.python.org/peps > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed May 25 15:29:48 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 25 May 2016 12:29:48 -0700 Subject: [Python-Dev] runtime dlls on Windows Message-ID: Hi folks, The standard build of Py3.5 for Windows is built with VS2015 (correct??) And it includes the runtime dlls it needs. However, we've found that wxPython wheels for win32 (not sure about win64) also need: MSVCP140.DLL So: wxPython could include that of course, But it looks like it's getting included with the Matplotlib wheels already (so folks with Matplotlib can run wx....). I'm just guessing, but this looks like the standard run time for C++ with that compiler. Python itself doesn't use C++, of course, but maybe we should include that dll with Python anyway -- that way folks can build wheels of packages with C++ extensions in the normal way, and those wheels will "just work", and we don't have to have every individual package ship the same dll. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed May 25 15:37:48 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 25 May 2016 14:37:48 -0500 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: Wouldn't downloading the Microsoft C++ Runtime 2015 also work? Many recent computers already have it pre-installed. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On May 25, 2016 2:31 PM, "Chris Barker" wrote: > Hi folks, > > The standard build of Py3.5 for Windows is built with VS2015 (correct??) > And it includes the runtime dlls it needs. > > However, we've found that wxPython wheels for win32 (not sure about win64) > also need: > > MSVCP140.DLL > > So: wxPython could include that of course, But it looks like it's getting > included with the Matplotlib wheels already (so folks with Matplotlib can > run wx....). I'm just guessing, but this looks like the standard run time > for C++ with that compiler. > > Python itself doesn't use C++, of course, but maybe we should include that > dll with Python anyway -- that way folks can build wheels of packages with > C++ extensions in the normal way, and those wheels will "just work", and > we don't have to have every individual package ship the same dll. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed May 25 16:26:54 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 25 May 2016 13:26:54 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On Wed, May 25, 2016 at 12:37 PM, Ryan Gonzalez wrote: > Wouldn't downloading the Microsoft C++ Runtime 2015 also work? > I'm sure it would -- I know that installing the entire MSVC2015 Community Edition does... but the point here is that end users should be able to: pip install something and if there is a binary wheel for something, it should work without them having to install something else. (why MS doesn't ship ALL their runtimes with eh OS is beyond me...) As it stands now, there are two options: 1) users need to install something else themselves: the runtime, that particular dll, the compiler... 2) every package maintainer that uses C++ needs to ship that dll with the binary wheels. If we put the dll into the python binary, then it would "just work" -CHB > Many recent computers already have it pre-installed. > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > On May 25, 2016 2:31 PM, "Chris Barker" wrote: > >> Hi folks, >> >> The standard build of Py3.5 for Windows is built with VS2015 (correct??) >> And it includes the runtime dlls it needs. >> >> However, we've found that wxPython wheels for win32 (not sure about >> win64) also need: >> >> MSVCP140.DLL >> >> So: wxPython could include that of course, But it looks like it's getting >> included with the Matplotlib wheels already (so folks with Matplotlib can >> run wx....). I'm just guessing, but this looks like the standard run time >> for C++ with that compiler. >> >> Python itself doesn't use C++, of course, but maybe we should include >> that dll with Python anyway -- that way folks can build wheels of packages >> with C++ extensions in the normal way, and those wheels will "just work", >> and we don't have to have every individual package ship the same dll. >> >> -CHB >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com >> >> -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed May 25 16:40:19 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 May 2016 06:40:19 +1000 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On Thu, May 26, 2016 at 6:26 AM, Chris Barker wrote: > the point here is that end users should be able to: > > pip install something > > and if there is a binary wheel for something, it should work without them > having to install something else. (why MS doesn't ship ALL their runtimes > with eh OS is beyond me...) Agreed with that, but meh. > As it stands now, there are two options: > > 1) users need to install something else themselves: the runtime, that > particular dll, the compiler... > 2) every package maintainer that uses C++ needs to ship that dll with the > binary wheels. > > If we put the dll into the python binary, then it would "just work" Counting your conclusion as option 3, you're offering the status quo and two competing solutions to it. I agree that the status quo is unideal; a binary wheel should either include or be able to fetch everything necessary for a supported platform. But why should CPython package a runtime that it doesn't use? Which is more common - someone uses two C++ modules, or someone uses none of them? I don't know how hard it is for the wheels to ship the DLL ("hard" here including any licensing or versioning issues, as well as the actual effort involved), but from a purely logical standpoint, it seems the most sane option. So I'm in favour of your option 2. ChrisA From mafagafogigante at gmail.com Wed May 25 17:05:52 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Wed, 25 May 2016 18:05:52 -0300 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: <4c3c0dcb-1050-86ec-a229-3a4fbe79ec41@gmail.com> On 05/25/2016 04:37 PM, Ryan Gonzalez wrote: > Wouldn't downloading the Microsoft C++ Runtime 2015 also work? Many recent > computers already have it pre-installed. Even though the download seems to be only 14 MB (I don't have a Windows machine, so I cannot assure that just the file on MS website is enough), it is an inconvenience. Additionally, I don't know how clear it will be for the average Python user on Windows that a C++ runtime is missing. From njs at pobox.com Wed May 25 17:52:58 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 25 May 2016 14:52:58 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On Wed, May 25, 2016 at 12:29 PM, Chris Barker wrote: > > Hi folks, > > The standard build of Py3.5 for Windows is built with VS2015 (correct??) And it includes the runtime dlls it needs. > > However, we've found that wxPython wheels for win32 (not sure about win64) also need: > > MSVCP140.DLL > > So: wxPython could include that of course, But it looks like it's getting included with the Matplotlib wheels already (so folks with Matplotlib can run wx....). I'm just guessing, but this looks like the standard run time for C++ with that compiler. > > Python itself doesn't use C++, of course, but maybe we should include that dll with Python anyway -- that way folks can build wheels of packages with C++ extensions in the normal way, and those wheels will "just work", and we don't have to have every individual package ship the same dll. The other challenge with this proposal is that Python 3.5.0 and 3.5.1 have already shipped without this .dll. So the most we could hope for is for it to be included in 3.5.2, and then wxPython and Matplotlib would either have to continue shipping it anyway, or else accept that their wheels actually require 3.5.2+ and will be broken if installed onto 3.5.0 or 3.5.1. And unfortunately there's no way in wheel metadata to express this kind of requirement -- wheels assume that all 3.5.x are both forwards and backwards compatible. So it would just be a weird unexplained breakage that some users would see. An alternative approach would be to stick MSVCP140.DLL into a tiny shim wheel and upload that to PyPI, and then wxPython and matplotlib's windows wheels could declare a dependency on this msvcp410 wheel. Basically this is the idea of my pynativelib proposal, though in this case you would only be using a small piece of the full proposal: https://mail.python.org/pipermail/wheel-builders/2016-April/000090.html (I really need to get back to that... if anyone wants to help sprint on this at PyCon let me know :-)) -n -- Nathaniel J. Smith -- https://vorpus.org From steve.dower at python.org Wed May 25 17:20:55 2016 From: steve.dower at python.org (Steve Dower) Date: Wed, 25 May 2016 14:20:55 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: <43d238d5-c5a5-057f-b509-dcadfb34bc60@python.org> On 25May2016 1229, Chris Barker wrote: > Hi folks, > > The standard build of Py3.5 for Windows is built with VS2015 (correct??) > And it includes the runtime dlls it needs. > > However, we've found that wxPython wheels for win32 (not sure about > win64) also need: > > MSVCP140.DLL There are two different versions of this DLL for each architecture (same name). > So: wxPython could include that of course, But it looks like it's > getting included with the Matplotlib wheels already (so folks with > Matplotlib can run wx....). I'm just guessing, but this looks like the > standard run time for C++ with that compiler. > > Python itself doesn't use C++, of course, but maybe we should include > that dll with Python anyway -- that way folks can build wheels of > packages with C++ extensions in the normal way, and those wheels will > "just work", and we don't have to have every individual package ship the > same dll. Unfortunately, it won't "just work". More often than not, it will break in weird and very difficult to diagnose ways, as the version of msvcp140.dll changes on a regular basis (there are at least four different version of it "in the wild" since VS 2015 released, not counting the preview releases before mid-last year). Importantly, it will break forward compatibility. We are already on the hook to keep shipping vcruntime140.dll for all 3.5 (and probably 3.6) releases, and if binary packages are built with later versions of VS then they'll need to include a (hypothetical) vcruntime150.dll (which distutils will already do, because I added that functionality). Other than that, every dependency of Python 3.5+ is forwards-compatible with new operating systems and compilers. There's also a slippery slope argument that would legitimately justify shipping all sorts of "common" dependencies with the core product. I chose to draw the line at "needed by the core Python product", rather than "created by Microsoft" or "used by many packages". (Hence vcruntime140.dll is included, despite not having forwards-compatibility, and I tried real hard to avoid including that one too.) Finally, unless we demand users are administrators for every install, we will quickly have many Python installs using versions of msvcp140.dll (or any other dependency) with security vulnerabilities, with no way to update them other than a full Python release. Installing the regular runtime (which is patched automatically) or as part of a package (which can be independently updated) is far more feasible. I really don't want to be doing security updates for 15 years just to keep an unused DLL secure. Hopefully that helps explain the position somewhat. I'm happy to go into more detail on any of these issues if anyone would like (potentially just linking to where I've previously discussed it either here, on bugs.p.o or on my blog). Cheers, Steve From ncoghlan at gmail.com Thu May 26 01:34:55 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 May 2016 15:34:55 +1000 Subject: [Python-Dev] devinabox has moved to GitHub In-Reply-To: References: Message-ID: On 26 May 2016 at 04:54, Franklin? Lee wrote: > It's just that I don't know whether any of them require particular > versions. If you say the latest is fine, then okay. For working on CPython trunk, the latest is fine. Things only have the potential to get trickier when building extensions for older Python versions (since you need to make sure the binaries are compatible), but that's not a topic devinabox concerns itself with. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From Artyom.Skrobov at arm.com Thu May 26 06:19:05 2016 From: Artyom.Skrobov at arm.com (Artyom Skrobov) Date: Thu, 26 May 2016 10:19:05 +0000 Subject: [Python-Dev] Python parser performance optimizations Message-ID: Hello, Back in March, I've posted a patch at http://bugs.python.org/issue26526 -- "In parsermodule.c, replace over 2KLOC of hand-crafted validation code, with a DFA". The motivation for this patch was to enable a memory footprint optimization, discussed at http://bugs.python.org/issue26415 My proposed optimization reduces the memory footprint by up to 30% on the standard benchmarks, and by 200% on a degenerate case which sparked the discussion. The run time stays unaffected by this optimization. Python Developer's Guide says: "If you don't get a response within a few days after pinging the issue, then you can try emailing python-dev at python.org asking for someone to review your patch." So, here I am. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu May 26 12:42:56 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 26 May 2016 09:42:56 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On Wed, May 25, 2016 at 1:40 PM, Chris Angelico wrote: > But why should CPython > package a runtime that it doesn't use? Because it IS part of the "standard runtime environment" that cPython is providing. The cPython Windows builds are built with a particular compiler that expects a particular runtime. C++ is part of that runtime, even if the cPython code itself doesn't use that part. I'm sure there are features in the C runtime DLL that cPython doesn't happen to use, either. The fact that it's a separate dll is kind of an implementation detail. > Which is more common - someone > uses two C++ modules, or someone uses none of them? > Irrelevant -- which is the bigger problem for users? a lot of folks getting an extra dll as part of a large install that they aren't using, or folks getting cryptic error messages and/or every package builder of C++ packages having to identify and solve the problem? BUT -- Steve Dower seems to have identified that the wonders of dll hell never cease, and this isn't possible anyway. Oh well. I don't know how hard it is for the wheels to ship the DLL ("hard" > here including any licensing or versioning issues, as well as the > actual effort involved), It's not all that hard. but one more step that has to be done. I wonder if setuptools / wheel could automate this somehow? -- hmm. On Wed, May 25, 2016 at 2:05 PM, Bernardo Sulzbach < mafagafogigante at gmail.com> wrote: > I don't know how clear it will be for the average Python user on Windows > that a C++ runtime is missing. That's the key problem -- the error message is NOT helpful -- it makes it look like the extension dll is missing, when it's actually that the extension dll can't link to the runtime dll -- but the message doesn't say what dll it's trying to link to. sigh. You need to be a pretty sophisticated user to figure it out -- or use Google :-) On Wed, May 25, 2016 at 2:52 PM, Nathaniel Smith wrote: > The other challenge with this proposal is that Python 3.5.0 and 3.5.1 > have already shipped without this .dll. yeah, maybe only 3.6 then, if possible at all. > wxPython and Matplotlib > would either have to continue shipping it anyway, or else accept that > their wheels actually require 3.5.2+ and will be broken if installed > onto 3.5.0 or 3.5.1. (unless the user has that dll ;-) ) And to be clear, this is going to effect any package that used C++ -- those two are two I happen to be familiar with. There can't be a small number of such packages. (hmm -- GDAL, anything Boost-based, anything else that uses AGG, .....) > An alternative approach would be to stick MSVCP140.DLL into a tiny > shim wheel and upload that to PyPI, and then wxPython and matplotlib's > windows wheels could declare a dependency on this msvcp410 wheel. > > Basically this is the idea of my pynativelib proposal, Yes -- if pynativelib becomes standard practice, then yes, that would be a good way to handle this issue -- jsut ike any other lib multiple packages might need. (I really need to get back to that... if anyone wants to help sprint > on this at PyCon let me know :-)) > I might -- haven't decided what to sprint on yet -- let's talk. (let's talk anyway) On Wed, May 25, 2016 at 2:20 PM, Steve Dower wrote: > MSVCP140.DLL >> > > There are two different versions of this DLL for each architecture (same > name). got to love Windows! Does the OS do the right thing if they are both installed? ouch! > Unfortunately, it won't "just work". More often than not, it will break in >> weird and very difficult to diagnose ways, as the version of msvcp140.dll >> changes on a regular basis (there are at least four different version of it >> "in the wild" since VS 2015 released, not counting the preview releases >> before mid-last year). >> > oh God. Well, that will pretty much kill it. So essentially every single app or lib written in C++ has to ship this dll? And IIUC, folks that have the compiler installed have SOME version on the system paths, and who knows what version other arbitrary stuff is linked to? How can this possibly be considered "Enterprise" software? Wow! Importantly, it will break forward compatibility. We are already on the > hook to keep shipping vcruntime140.dll for all 3.5 (and probably 3.6) > releases, and if binary packages are built with later versions of VS then > they'll need to include a (hypothetical) vcruntime150.dll (which distutils > will already do, because I added that functionality). wow. So could that be extended to do the same thing with MSVCP140.DLL? then each package would be shipping it -- but it would happen automagically, rather than everyone having to figure out that they have to do it, and how to do it. > There's also a slippery slope argument that would legitimately justify > shipping all sorts of "common" dependencies with the core product. I chose > to draw the line at "needed by the core Python product", rather than > "created by Microsoft" or "used by many packages". (Hence vcruntime140.dll > is included, despite not having forwards-compatibility, and I tried real > hard to avoid including that one too.) > yeah, a line has to be drawn, and "created by Microsoft" wouldn't make any sense either. but "standard runtime expected by the compiler" would -- but, as you say above, you have may have to ship a new version with your extension anyway -- so it seems adding the feature for distutils to do that for C++ extensions would both solve the problem and not ship anything extra with the core install. > Finally, unless we demand users are administrators for every install, we > will quickly have many Python installs using versions of msvcp140.dll (or > any other dependency) with security vulnerabilities, with no way to update > them other than a full Python release. Installing the regular runtime > (which is patched automatically) or as part of a package (which can be > independently updated) is far more feasible. I really don't want to be > doing security updates for 15 years just to keep an unused DLL secure. > Though we have to do that for msvcp140.dll anyway, yes? god, what a pain! (and why MS should ship the damn things themselves!) Hopefully that helps explain the position somewhat. I'm happy to go into > more detail on any of these issues if anyone would like (potentially just > linking to where I've previously discussed it either here, on bugs.p.o or > on my blog). Thanks Steve. Will you be at PyCon? if Nathaniel and I look at this during the sprints, maybe you could coach us a bit. Anyway, proposal withdrawn. Replaced with possible proposal to have distutils include the C++ runtime dll like it does now for the C one. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu May 26 12:53:27 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 May 2016 02:53:27 +1000 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On Fri, May 27, 2016 at 2:42 AM, Chris Barker wrote: > On Wed, May 25, 2016 at 1:40 PM, Chris Angelico wrote: >> >> But why should CPython >> package a runtime that it doesn't use? > > > Because it IS part of the "standard runtime environment" that cPython is > providing. The cPython Windows builds are built with a particular compiler > that expects a particular runtime. C++ is part of that runtime, even if the > cPython code itself doesn't use that part. I'm sure there are features in > the C runtime DLL that cPython doesn't happen to use, either. The fact that > it's a separate dll is kind of an implementation detail. I'm not sure that it is an implementation detail; it's indicative that the compiler/runtime designers intended for the C runtime to be able to be distributed without the C++ runtime. You wouldn't expect to have to ship a Fortran runtime and a Cobol runtime, even if MSVC happened to package up compilers for all of the above, would you? >> Which is more common - someone >> uses two C++ modules, or someone uses none of them? > > > Irrelevant -- which is the bigger problem for users? a lot of folks getting > an extra dll as part of a large install that they aren't using, or folks > getting cryptic error messages and/or every package builder of C++ packages > having to identify and solve the problem? I agree with you that end users getting cryptic error messages is a problem. But the choice of solutions is: 1) CPython ships the C++ runtime that it doesn't use, in case someone pip-installs something that needs it 2) Package authors who choose to use C++ must ship the C++ runtime. Even apart from the DLL Hell that Steve mentioned, the second option appears more sane. Package authors get the freedom to choose any runtime dependencies they want, and then they package them or make it possible to obtain them. > BUT -- Steve Dower seems to have identified that the wonders of dll hell > never cease, and this isn't possible anyway. Oh well. > I'm not entirely grasping what's happening here. There are multiple versions of msvcp140.dll floating around out there; what happens if one app requires one of them and another requires another? Can you simply say "take the newer one"? Is there a way to instantly recognize that there's a newer one available, and ignore the older one? Versioning is a *pain*. ChrisA From chris.barker at noaa.gov Thu May 26 13:12:53 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 26 May 2016 10:12:53 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On Thu, May 26, 2016 at 9:53 AM, Chris Angelico wrote: > > BUT -- Steve Dower seems to have identified that the wonders of dll hell > > never cease, and this isn't possible anyway. Oh well. > > I'm not entirely grasping what's happening here. me neither :-( > There are multiple > versions of msvcp140.dll floating around out there; what happens if > one app requires one of them and another requires another? That's pretty much the dll hell that has plagued Windows FOREVER! why they don't do some versioning is totally beyond me, though 8.3 filenames have not been a restriction for a long time. > Can you simply say "take the newer one"? I *think* that standard Windows policy/practice is that newer versions of a dll with the same name should always be backward compatible -- so something built against an older version should work with the newer one -- so you can ship a dll with a security patch, and it will all work. I presume that MS itself is careful about this with their runtimes. > Is there a way to instantly recognize > that there's a newer one available, and ignore the older one? > I don't think so -- the linker will pick the first one it finds on the search path :-( -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu May 26 14:02:12 2016 From: brett at python.org (Brett Cannon) Date: Thu, 26 May 2016 18:02:12 +0000 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On Thu, 26 May 2016 at 09:44 Chris Barker wrote: > [SNIP] > Thanks Steve. Will you be at PyCon? if Nathaniel and I look at this > during the sprints, maybe you could coach us a bit. > Steve will be at PyCon but I don't think he will be around for the sprints as he has to catch a flight out for PyCon Taiwan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu May 26 15:13:10 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 26 May 2016 12:13:10 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: <1feb49e3-f2e0-11c0-640f-ca4ef75bf8bc@python.org> On 26May2016 0942, Chris Barker wrote: > An alternative approach would be to stick MSVCP140.DLL into a tiny > shim wheel and upload that to PyPI, and then wxPython and matplotlib's > windows wheels could declare a dependency on this msvcp410 wheel. > > Basically this is the idea of my pynativelib proposal, > > > Yes -- if pynativelib becomes standard practice, then yes, that would be > a good way to handle this issue -- jsut ike any other lib multiple > packages might need. > > (I really need to get back to that... if anyone wants to help sprint > on this at PyCon let me know :-)) > > > I might -- haven't decided what to sprint on yet -- let's talk. (let's > talk anyway) This seems like a good solution. I won't be around for sprints, but we can chat about it before then. > > MSVCP140.DLL > > > There are two different versions of this DLL for each architecture > (same name). > > > got to love Windows! Does the OS do the right thing if they are both > installed? ouch! If they're installed properly into System32/SysWOW64 (using the official installer), then yes. If you simply drop them into your Python install directory, then no, unless you drop the right one - it has to match the python.exe architecture. > > Unfortunately, it won't "just work". More often than not, it > will break in weird and very difficult to diagnose ways, as the > version of msvcp140.dll changes on a regular basis (there are at > least four different version of it "in the wild" since VS 2015 > released, not counting the preview releases before mid-last year). > > > oh God. Well, that will pretty much kill it. So essentially every > single app or lib written in C++ has to ship this dll? And IIUC, folks > that have the compiler installed have SOME version on the system paths, > and who knows what version other arbitrary stuff is linked to? How can > this possibly be considered "Enterprise" software? Wow! To make it worse, you can only load it once per process. So the first extension to load their private copy wins. The DLL is designed to be installed using the official installer, statically linked, or embedded as part of an entire application. It's not meant to be injected into someone else's application (which is essentially what Python extension modules are doing). > There's also a slippery slope argument that would legitimately > justify shipping all sorts of "common" dependencies with the core > product. I chose to draw the line at "needed by the core Python > product", rather than "created by Microsoft" or "used by many > packages". (Hence vcruntime140.dll is included, despite not having > forwards-compatibility, and I tried real hard to avoid including > that one too.) > > > yeah, a line has to be drawn, and "created by Microsoft" wouldn't make > any sense either. but "standard runtime expected by the compiler" would > -- but, as you say above, you have may have to ship a new version with > your extension anyway -- so it seems adding the feature for distutils to > do that for C++ extensions would both solve the problem and not ship > anything extra with the core install. It would be possible. It might be simpler to update the default compiler settings to statically link the C++ runtime while still dynamically linking the C runtime (and any project is already capable of doing this on their own - I'm 99% sure it works). > Finally, unless we demand users are administrators for every > install, we will quickly have many Python installs using versions of > msvcp140.dll (or any other dependency) with security > vulnerabilities, with no way to update them other than a full Python > release. Installing the regular runtime (which is patched > automatically) or as part of a package (which can be independently > updated) is far more feasible. I really don't want to be doing > security updates for 15 years just to keep an unused DLL secure. > > > Though we have to do that for msvcp140.dll anyway, yes? god, what a > pain! (and why MS should ship the damn things themselves!) Maybe, but it's also the same reason why MS doesn't want to ship it themselves either. In fact, all the reasons why we don't want to include it in Python equally apply to Windows - until it's a direct dependency of the product itself, it has a really high bar to be included (SQLite is included in Windows because it is a dependency, not just as a convenience to developers). > Anyway, proposal withdrawn. > > Replaced with possible proposal to have distutils include the C++ > runtime dll like it does now for the C one. The big difference here is that msvcp140.dll contains significant functionality while vcruntime140.dll is a very small amount of shared state and initialization. Changes/additions/fixes to msvcp140.dll are far more likely as a result. I don't recall whether pip extensions are a thing yet, but if they are this would be a great candidate for that (trigger the runtime installer if necessary). Otherwise, having a package that only includes this DLL would probably be fine too, though it really only defers the problem to the next DLL you depend on. Cheers, Steve From breamoreboy at yahoo.co.uk Thu May 26 17:01:51 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Thu, 26 May 2016 22:01:51 +0100 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On 26/05/2016 19:02, Brett Cannon wrote: > > > On Thu, 26 May 2016 at 09:44 Chris Barker > wrote: > > [SNIP] > Thanks Steve. Will you be at PyCon? if Nathaniel and I look at this > during the sprints, maybe you could coach us a bit. > > Steve will be at PyCon but I don't think he will be around for the > sprints as he has to catch a flight out for PyCon Taiwan. > Steve writes such an excellent blog that when and if he has time I'm certain that he'll put something together. With people such as him following on from Martin Loewis, Tim Golden, Mark "I've forgotten more about Python on Windows than you'll ever know" Hammond and Andy "ditto" Robinson I can't say that I'll lose too much sleep over it. As I suffer insomnia on top of chronic fatigue syndrome, that is quite a statement for me to make. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From sebastian at realpath.org Thu May 26 17:53:12 2016 From: sebastian at realpath.org (Sebastian Krause) Date: Thu, 26 May 2016 23:53:12 +0200 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: (Chris Angelico's message of "Fri, 27 May 2016 02:53:27 +1000") References: Message-ID: Chris Angelico wrote: >> BUT -- Steve Dower seems to have identified that the wonders of dll hell >> never cease, and this isn't possible anyway. Oh well. > > I'm not entirely grasping what's happening here. There are multiple > versions of msvcp140.dll floating around out there; what happens if > one app requires one of them and another requires another? Can you > simply say "take the newer one"? Is there a way to instantly recognize > that there's a newer one available, and ignore the older one? > Versioning is a *pain*. Windows executables can and should ask for specific versions of the DLLs in the manifest (an XML file embedded in the EXE). Details here: https://en.wikipedia.org/wiki/Side-by-side_assembly Sebastian From steve.dower at python.org Thu May 26 18:18:35 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 26 May 2016 15:18:35 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On 26May2016 1453, Sebastian Krause wrote: > Chris Angelico wrote: >>> BUT -- Steve Dower seems to have identified that the wonders of dll hell >>> never cease, and this isn't possible anyway. Oh well. >> >> I'm not entirely grasping what's happening here. There are multiple >> versions of msvcp140.dll floating around out there; what happens if >> one app requires one of them and another requires another? Can you >> simply say "take the newer one"? Is there a way to instantly recognize >> that there's a newer one available, and ignore the older one? >> Versioning is a *pain*. > > Windows executables can and should ask for specific versions of the > DLLs in the manifest (an XML file embedded in the EXE). Details > here: https://en.wikipedia.org/wiki/Side-by-side_assembly This has been deprecated. It sounded like a great idea at the time (~8 years ago) but caused more problems than it solved. Python 2.7 and 3.2 (IIRC) were the last versions to use this feature. The way to get the newest version is to run the installer, which will upgrade if it has a newer version or leave the old one alone. Cheers, Steve From v+python at g.nevcal.com Thu May 26 19:01:37 2016 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 26 May 2016 16:01:37 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: On 5/26/2016 3:18 PM, Steve Dower wrote: > This has been deprecated. It sounded like a great idea at the time (~8 > years ago) but caused more problems than it solved. Somehow I missed the announcement of the deprecation. When I first heard of Windows manifests, though, my first reaction was that it sounded like a solution that was more complex than the problem it was trying to solve. Glad to hear they've been deprecated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu May 26 19:42:07 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 26 May 2016 16:42:07 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: Message-ID: <6e86988e-ca93-22a7-1bdf-3cde408e8c2c@python.org> On 26May2016 1601, Glenn Linderman wrote: > On 5/26/2016 3:18 PM, Steve Dower wrote: >> This has been deprecated. It sounded like a great idea at the time (~8 >> years ago) but caused more problems than it solved. > > Somehow I missed the announcement of the deprecation. The feature itself probably hasn't been officially deprecated, but using it for the C and C++ runtime libraries has not been required for a long time now. (I need to be more careful that I don't accidentally announce things that may not be true :) ) > When I first heard of Windows manifests, though, my first reaction was > that it sounded like a solution that was more complex than the problem > it was trying to solve. Indeed. Cheers, Steve From rosuav at gmail.com Thu May 26 19:58:56 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 May 2016 09:58:56 +1000 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: <1feb49e3-f2e0-11c0-640f-ca4ef75bf8bc@python.org> References: <1feb49e3-f2e0-11c0-640f-ca4ef75bf8bc@python.org> Message-ID: On Fri, May 27, 2016 at 5:13 AM, Steve Dower wrote: > If they're installed properly into System32/SysWOW64 (using the official > installer), then yes. If you simply drop them into your Python install > directory, then no, unless you drop the right one - it has to match the > python.exe architecture. > What if the Python installation isn't being run as admin? Or is that deprecated too? ChrisA From dholth at gmail.com Thu May 26 22:41:38 2016 From: dholth at gmail.com (Daniel Holth) Date: Fri, 27 May 2016 02:41:38 +0000 Subject: [Python-Dev] Adding a threadlocal to the Python interpreter In-Reply-To: References: Message-ID: Thank you. I started my patch http://bugs.python.org/issue27134 to allow Python code to set a flag causing the evil str(b'bytes') to raise an exception. I wasn't sure exactly which module to put it in, so it's in _string. Please let me know the best place to put the feature and what I should polish to get it in. Thanks, Daniel Holth On Thu, May 19, 2016 at 6:34 AM Christian Heimes wrote: > On 2016-05-19 04:30, Nick Coghlan wrote: > > On 18 May 2016 at 23:20, Daniel Holth wrote: > >> I would like to take another stab at adding a threadlocal "str(bytes) > raises > >> an exception" to the Python interpreter, but I had a very hard time > >> understanding both how to add a threadlocal value to either the > interpreter > >> state or the threadlocal dict that is part of that state, and then how > to > >> access the same value from both Python and CPython code. The structs > were > >> there but it was just hard to understand. Can someone explain it to me? > > > > Christian covered the C aspects of the API, while the general purpose > > Python aspects live in the threading module. > > > > However, the Python level thread-local API doesn't provide direct > > access to the thread state dict. Instead, it provides access to > > subdicts stored under per-object keys in that dict, keyed as > > "thread.local.": > > In case you wonder about subdicts, they are required to provide multiple > thread local objects. Each thread local instance has its own key in each > thread state dict. The approach enables thread local to have independent > storage objects. > > Christian > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu May 26 23:45:43 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 26 May 2016 20:45:43 -0700 Subject: [Python-Dev] runtime dlls on Windows In-Reply-To: References: <1feb49e3-f2e0-11c0-640f-ca4ef75bf8bc@python.org> Message-ID: Another reason why we don't want to install this DLL with Python :) Per-user install is properly supported in 3.5 for I believe the first time ever. But yes, installing them properly requires admin privileges. An unfortunately messy situation, one that I'm happy to not have to resolve for Python's incredibly diverse user base. Top-posted from my Windows Phone -----Original Message----- From: "Chris Angelico" Sent: ?5/?26/?2016 17:04 Cc: "python-dev" Subject: Re: [Python-Dev] runtime dlls on Windows On Fri, May 27, 2016 at 5:13 AM, Steve Dower wrote: > If they're installed properly into System32/SysWOW64 (using the official > installer), then yes. If you simply drop them into your Python install > directory, then no, unless you drop the right one - it has to match the > python.exe architecture. > What if the Python installation isn't being run as admin? Or is that deprecated too? ChrisA _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Fri May 27 00:54:33 2016 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 26 May 2016 21:54:33 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> Message-ID: > On May 25, 2016, at 3:29 AM, Christian Heimes wrote: > > I have three hashing-related patches for Python 3.6 that are waiting for > review. Altogether the three patches add ten new hash algorithms to the > hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256), > BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256). Do we really need ten? I don't think the standard library is the place to offer all variants of hashing. And we should avoid getting in a cycle of "this was just released by NIST" and "nobody uses that one anymore". Is any one of them an emergent best practice (i.e. starting to be commonly used in network protocols because it is better, faster, stronger, etc)? Your last message on https://bugs.python.org/issue16113 suggests that these aren't essential and that there is room for debate about whether some of them are standard-library worthy (i.e. we will have them around forever). Raymond From donald at stufft.io Fri May 27 06:01:40 2016 From: donald at stufft.io (Donald Stufft) Date: Fri, 27 May 2016 06:01:40 -0400 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> Message-ID: > On May 27, 2016, at 12:54 AM, Raymond Hettinger wrote: > > >> On May 25, 2016, at 3:29 AM, Christian Heimes wrote: >> >> I have three hashing-related patches for Python 3.6 that are waiting for >> review. Altogether the three patches add ten new hash algorithms to the >> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256), >> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256). > > Do we really need ten? I don't think the standard library is the place to offer all variants of hashing. And we should avoid getting in a cycle of "this was just released by NIST" and "nobody uses that one anymore". Is any one of them an emergent best practice (i.e. starting to be commonly used in network protocols because it is better, faster, stronger, etc)? > > Your last message on https://bugs.python.org/issue16113 suggests that these aren't essential and that there is room for debate about whether some of them are standard-library worthy (i.e. we will have them around forever). > I think that adding sha3 here is a net positive. While there isn?t a huge amount of things using it today, that?s largely because it?s fairly new? It?s a NIST standard so it won?t be long until things are using it. It would be surprising to me to be able to use sha1 and sha2 from the standard library, but not sha3. SHAKE is really just SHA3 with some additional tweaks to the parameters. I think if you?re adding SHA3 it?s pretty easy to also add these, though I don?t think that it?s as important as adding SHA3 itself. BLAKE2 is an interesting one, because while SHA3 is a NIST standard (so it?s going to gain adoption because of that), BLAKE2 is at least as strong as SHA3 but is better in many ways, particularly in speed? it?s actually faster than MD5 while being as secure as SHA3. This one I think is a good one to have in the standard library as well because it is all around a really great hash and a lot of things are starting to be built on top of it. In particularly I?d like to use this in PyPI and pip- but I can?t unless it?s in the standard library. ? Donald Stufft From victor.stinner at gmail.com Fri May 27 06:44:00 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 27 May 2016 12:44:00 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> Message-ID: Le 27 mai 2016 12:05 PM, "Donald Stufft" a ?crit : > BLAKE2 is an interesting one, because while SHA3 is a NIST standard (so it?s going to gain adoption because of that), BLAKE2 is at least as strong as SHA3 but is better in many ways, particularly in speed? it?s actually faster than MD5 while being as secure as SHA3. BLAKE2 was part of the SHA3 competition and it was in finalists. The SHA3 competition is interesting because each algorithm is deeply tested and analyzed by many teams all around the world. Obvious vulnerabilities are quickly found. The advantage of putting SHA3 and BLAKE2 in the stdlib is that they have a different design. I don't expect that two designs have the same vulnerabilities, but I'm not ax expert :-) SHA3 (Keccak) is based on a new sponge construction: https://en.m.wikipedia.org/wiki/SHA-3 BLAKE is based on ChaCha: https://en.m.wikipedia.org/wiki/BLAKE_(hash_function) https://en.m.wikipedia.org/wiki/Salsa20#ChaCha_variant Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri May 27 06:54:49 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 27 May 2016 12:54:49 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> Message-ID: <57482779.6030100@egenix.com> On 27.05.2016 06:54, Raymond Hettinger wrote: > >> On May 25, 2016, at 3:29 AM, Christian Heimes wrote: >> >> I have three hashing-related patches for Python 3.6 that are waiting for >> review. Altogether the three patches add ten new hash algorithms to the >> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256), >> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256). > > Do we really need ten? I don't think the standard library is the place to offer all variants of hashing. And we should avoid getting in a cycle of "this was just released by NIST" and "nobody uses that one anymore". Is any one of them an emergent best practice (i.e. starting to be commonly used in network protocols because it is better, faster, stronger, etc)? > > Your last message on https://bugs.python.org/issue16113 suggests that these aren't essential and that there is room for debate about whether some of them are standard-library worthy (i.e. we will have them around forever). I can understand your eagerness to get this landed, since it's been 4 years since work started, but I think we should wait with the addition until OpenSSL has them: https://github.com/openssl/openssl/issues/439 The current patch is 1.2MB for SHA-3 - that's pretty heavy for just a few hash functions, which aren't in any wide spread use yet and probably won't be for quite a few years ahead. IMO, relying on OpenSSL is a better strategy than providing (and maintaining) our own compatibility versions. Until OpenSSL has them, people can use Bj?rn's package: https://github.com/bjornedstrom/python-sha3 Perhaps you could join forces with Bj?rn to create a standard SHA-3 standalone package on PyPI based on your two variants which we could recommend to people in the docs ?! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 27 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From donald at stufft.io Fri May 27 07:03:46 2016 From: donald at stufft.io (Donald Stufft) Date: Fri, 27 May 2016 07:03:46 -0400 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <57482779.6030100@egenix.com> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> Message-ID: <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io> > On May 27, 2016, at 6:54 AM, M.-A. Lemburg wrote: > > IMO, relying on OpenSSL is a better strategy than providing > (and maintaining) our own compatibility versions. Until OpenSSL > has them, people can use Bj?rn's package: Even now, hashlib doesn?t rely on OpenSSL if I recall, I mean it will use it if OpenSSL is available but otherwise it has internal implementations too. ? Donald Stufft From mal at egenix.com Fri May 27 07:13:05 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 27 May 2016 13:13:05 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io> Message-ID: <57482BC1.6000200@egenix.com> On 27.05.2016 13:03, Donald Stufft wrote: > >> On May 27, 2016, at 6:54 AM, M.-A. Lemburg wrote: >> >> IMO, relying on OpenSSL is a better strategy than providing >> (and maintaining) our own compatibility versions. Until OpenSSL >> has them, people can use Bj?rn's package: > > Even now, hashlib doesn?t rely on OpenSSL if I recall, I mean it will > use it if OpenSSL is available but otherwise it has internal implementations > too. I know, but still don't think that's a good idea. It makes sense in case you don't want to carry around OpenSSL all the time, but how often does that happen nowadays ? BTW: If I recall correctly, those hash implementations predate the deeper support for OpenSSL we now have in Python. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 27 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From dholth at gmail.com Fri May 27 10:31:45 2016 From: dholth at gmail.com (Daniel Holth) Date: Fri, 27 May 2016 14:31:45 +0000 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <57482BC1.6000200@egenix.com> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io> <57482BC1.6000200@egenix.com> Message-ID: OpenSSL sucks. Python would only have to bundle a reference implementation of the new hash algorithm(s), and unlike TLS suites they tend to just work. BLAKE2 is important, since it removes the last objection to replacing MD5 - speed - that has made it hard for cryptography fans to convince MD5 users to upgrade. On Fri, May 27, 2016 at 7:13 AM M.-A. Lemburg wrote: > On 27.05.2016 13:03, Donald Stufft wrote: > > > >> On May 27, 2016, at 6:54 AM, M.-A. Lemburg wrote: > >> > >> IMO, relying on OpenSSL is a better strategy than providing > >> (and maintaining) our own compatibility versions. Until OpenSSL > >> has them, people can use Bj?rn's package: > > > > Even now, hashlib doesn?t rely on OpenSSL if I recall, I mean it will > > use it if OpenSSL is available but otherwise it has internal > implementations > > too. > > I know, but still don't think that's a good idea. It makes > sense in case you don't want to carry around OpenSSL all the > time, but how often does that happen nowadays ? > > BTW: If I recall correctly, those hash implementations predate > the deeper support for OpenSSL we now have in Python. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, May 27 2016) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitaly.murashev at gmail.com Fri May 27 05:49:08 2016 From: vitaly.murashev at gmail.com (Vitaly Murashev) Date: Fri, 27 May 2016 12:49:08 +0300 Subject: [Python-Dev] embedding - PyImport_AppendInittab() after Py_Initialize() Message-ID: Could anyone please clarify whether it is correct in python3.5 to call PyImport_AppendInittab() after Py_Initialize() I found this case among the tests for boost.python https://github.com/boostorg/python/blob/develop/test/exec.cpp And this test doesn't work for me with python3.5.An error looks like: exec.cpp(137): Python Error detected in function 'void __cdecl check_pyerr(bool)' Traceback (most recent call last): File "", line 1, in ImportError: 'embedded_hello' is not a built-in module 1 error detected. After debugging I found out that root cause is in importlib/_bootstrap.py which do the following: if fullname not in sys.builtin_module_names: raise ImportError('{!r} is not a built-in module'.format(fullname), name=fullname) but sys.builtin_module_names is the constant which initialized once in Py_Initialize() --- To summarize: Is it a bug in python3.5 or original test inside boost.python should be fixed ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mafagafogigante at gmail.com Fri May 27 11:37:48 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Fri, 27 May 2016 12:37:48 -0300 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io> <57482BC1.6000200@egenix.com> Message-ID: On 05/27/2016 11:31 AM, Daniel Holth wrote: > > BLAKE2 is important, since it removes the last objection to replacing MD5 - > speed - that has made it hard for cryptography fans to convince MD5 users > to upgrade. > I have had to stick to MD5 for performance reasons (2 seconds in MD5 or 9.6 seconds in SHA256, IIRC) in scenarios that did not require an SHA*. Having BLAKE2 around wouldn't be a necessity, but if it shipped with newer versions of Python eventually there would be a commit switching the underlying hash function. From chris.barker at noaa.gov Fri May 27 11:44:45 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 27 May 2016 08:44:45 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <57482779.6030100@egenix.com> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> Message-ID: <7067504675934721722@unknownmsgid> >> , which aren't in any wide spread use yet and > probably won't be for quite a few years ahead. Anything added to the stdlib now will be in py3.6+, yes? Which won't be in widespread use for quite a few years yet, either. So if ( and that's a big if) it's possible to anticipate what will be in widespread use in a couple years, getting it in now would be a good thing. -CHB > > IMO, relying on OpenSSL is a better strategy than providing > (and maintaining) our own compatibility versions. Until OpenSSL > has them, people can use Bj?rn's package: > > https://github.com/bjornedstrom/python-sha3 > > Perhaps you could join forces with Bj?rn to create a standard > SHA-3 standalone package on PyPI based on your two variants > which we could recommend to people in the docs ?! > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, May 27 2016) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> Python Database Interfaces ... http://products.egenix.com/ >>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov From status at bugs.python.org Fri May 27 12:08:42 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 27 May 2016 18:08:42 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20160527160842.70BFF56A1E@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-05-20 - 2016-05-27) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5529 (+25) closed 33364 (+42) total 38893 (+67) Open issues with patches: 2400 Issues opened (46) ================== #16858: tarfile silently hides errors http://bugs.python.org/issue16858 reopened by mmarkk #26915: Test identity first in membership operation of ItemsView, Valu http://bugs.python.org/issue26915 reopened by xiang.zhang #27067: Improve curses tests http://bugs.python.org/issue27067 reopened by martin.panter #27072: random.getrandbits is limited to 2**31-1 bits on 64-bit Window http://bugs.python.org/issue27072 opened by Steven.Barker #27073: redundant checks in long_add and long_sub http://bugs.python.org/issue27073 opened by Oren Milman #27074: Confusing text about __all__ in __init__.py in tutorial http://bugs.python.org/issue27074 opened by ztane #27078: Make f'' strings faster than .format: BUILD_STRING opcode? http://bugs.python.org/issue27078 opened by ztane #27079: Bugs in curses.ascii predicates http://bugs.python.org/issue27079 opened by serhiy.storchaka #27080: Implement the formatting part of PEP 515, '_' in numeric liter http://bugs.python.org/issue27080 opened by eric.smith #27081: Multiprocessing is not robust against sys.stderr changes invol http://bugs.python.org/issue27081 opened by ppperry #27083: PYTHONCASEOK is ignored on Windows http://bugs.python.org/issue27083 opened by eryksun #27084: Add dir_fd and follow_symlinks kwargs to os.listdir and os.sca http://bugs.python.org/issue27084 opened by abacabadabacaba #27085: Make it possible to select return type for os.listdir http://bugs.python.org/issue27085 opened by abacabadabacaba #27086: Add closefd argument to os.listdir http://bugs.python.org/issue27086 opened by abacabadabacaba #27088: doc: select: epoll.poll: incorrect timeout units, missing maxe http://bugs.python.org/issue27088 opened by fordsfords #27095: Simplify MAKE_FUNCTION http://bugs.python.org/issue27095 opened by Demur Rumed #27099: IDLE: turn builting extensions into regular modules http://bugs.python.org/issue27099 opened by terry.reedy #27100: Attempting to use class with both __enter__ & __exit__ undefin http://bugs.python.org/issue27100 opened by ellingtonjp #27101: Compilation of python (modules) for foreign target platform pr http://bugs.python.org/issue27101 opened by complement #27103: regrtest: capture stdout (-W) option is incompatible with refl http://bugs.python.org/issue27103 opened by haypo #27105: cgi.__all__ is incomplete http://bugs.python.org/issue27105 opened by Unit03 #27106: configparser.__all__ is incomplete http://bugs.python.org/issue27106 opened by Unit03 #27107: mailbox.__all__ list is incomplete http://bugs.python.org/issue27107 opened by Unit03 #27108: mimetypes.__all__ list is incomplete http://bugs.python.org/issue27108 opened by Unit03 #27109: plistlib.__all__ list is incomplete http://bugs.python.org/issue27109 opened by Unit03 #27110: smtpd.__all__ list is incomplete http://bugs.python.org/issue27110 opened by Unit03 #27111: redundant variables in long_add and long_sub http://bugs.python.org/issue27111 opened by Oren Milman #27112: tokenize.__all__ list is incomplete http://bugs.python.org/issue27112 opened by Unit03 #27113: sqlite3 connect parameter "check_same_thread" not documented http://bugs.python.org/issue27113 opened by Dave Sawyer #27115: IDLE/tkinter: in simpledialog, != [OK] click http://bugs.python.org/issue27115 opened by terry.reedy #27117: turtledemo does not work with IDLE's new dark theme. http://bugs.python.org/issue27117 opened by terry.reedy #27119: `compile` doesn't compile into an AST object as specified http://bugs.python.org/issue27119 opened by leewz #27121: imghdr does not support jpg files with Lavc bytes http://bugs.python.org/issue27121 opened by Ren?? L??we Jacobsen #27122: Hang with contextlib.ExitStack and subprocess.Popen (regressio http://bugs.python.org/issue27122 opened by Valentin David #27123: Allow `install_headers` command to follow specific directory s http://bugs.python.org/issue27123 opened by sylvain.corlay #27124: binascii.a2b_hex raises binascii.Error and ValueError, not Typ http://bugs.python.org/issue27124 opened by Lennart Grahl #27125: Typo in Python 2 multiprocessing documentation http://bugs.python.org/issue27125 opened by phx #27126: Apple-supplied libsqlite3 on OS X is not fork safe; can cause http://bugs.python.org/issue27126 opened by evan.jones at bluecore.com #27127: Never have GET_ITER not followed by FOR_ITER http://bugs.python.org/issue27127 opened by Demur Rumed #27128: Add _PyObject_FastCall() http://bugs.python.org/issue27128 opened by haypo #27129: Wordcode, part 2 http://bugs.python.org/issue27129 opened by serhiy.storchaka #27130: zlib: OverflowError while trying to compress 2^32 bytes or mor http://bugs.python.org/issue27130 opened by Klamann #27131: Unit test random shuffle http://bugs.python.org/issue27131 opened by jonathan.kross #27132: New assert method that checks an error message for a list of s http://bugs.python.org/issue27132 opened by maciej.szulik #27133: python 3.5.1 will not compile because libffi module uses wrong http://bugs.python.org/issue27133 opened by blastwave #27136: sock_connect fails for bluetooth (and probably others) http://bugs.python.org/issue27136 opened by pyptr2 Most recent 15 issues with no replies (15) ========================================== #27123: Allow `install_headers` command to follow specific directory s http://bugs.python.org/issue27123 #27121: imghdr does not support jpg files with Lavc bytes http://bugs.python.org/issue27121 #27117: turtledemo does not work with IDLE's new dark theme. http://bugs.python.org/issue27117 #27115: IDLE/tkinter: in simpledialog, != [OK] click http://bugs.python.org/issue27115 #27111: redundant variables in long_add and long_sub http://bugs.python.org/issue27111 #27109: plistlib.__all__ list is incomplete http://bugs.python.org/issue27109 #27103: regrtest: capture stdout (-W) option is incompatible with refl http://bugs.python.org/issue27103 #27088: doc: select: epoll.poll: incorrect timeout units, missing maxe http://bugs.python.org/issue27088 #27080: Implement the formatting part of PEP 515, '_' in numeric liter http://bugs.python.org/issue27080 #27079: Bugs in curses.ascii predicates http://bugs.python.org/issue27079 #27074: Confusing text about __all__ in __init__.py in tutorial http://bugs.python.org/issue27074 #27038: Make os.DirEntry exist http://bugs.python.org/issue27038 #27030: Remove deprecated re features http://bugs.python.org/issue27030 #27021: It is not documented that os.writev() suffer from SC_IOV_MAX http://bugs.python.org/issue27021 #27019: Reduce marshal stack depth for 2.7 on Windows debug build http://bugs.python.org/issue27019 Most recent 15 issues waiting for review (15) ============================================= #27136: sock_connect fails for bluetooth (and probably others) http://bugs.python.org/issue27136 #27131: Unit test random shuffle http://bugs.python.org/issue27131 #27129: Wordcode, part 2 http://bugs.python.org/issue27129 #27128: Add _PyObject_FastCall() http://bugs.python.org/issue27128 #27127: Never have GET_ITER not followed by FOR_ITER http://bugs.python.org/issue27127 #27125: Typo in Python 2 multiprocessing documentation http://bugs.python.org/issue27125 #27124: binascii.a2b_hex raises binascii.Error and ValueError, not Typ http://bugs.python.org/issue27124 #27113: sqlite3 connect parameter "check_same_thread" not documented http://bugs.python.org/issue27113 #27112: tokenize.__all__ list is incomplete http://bugs.python.org/issue27112 #27111: redundant variables in long_add and long_sub http://bugs.python.org/issue27111 #27110: smtpd.__all__ list is incomplete http://bugs.python.org/issue27110 #27109: plistlib.__all__ list is incomplete http://bugs.python.org/issue27109 #27108: mimetypes.__all__ list is incomplete http://bugs.python.org/issue27108 #27107: mailbox.__all__ list is incomplete http://bugs.python.org/issue27107 #27106: configparser.__all__ is incomplete http://bugs.python.org/issue27106 Top 10 most discussed issues (10) ================================= #26647: ceval: use Wordcode, 16-bit bytecode http://bugs.python.org/issue26647 23 msgs #27067: Improve curses tests http://bugs.python.org/issue27067 13 msgs #26632: __all__ decorator http://bugs.python.org/issue26632 9 msgs #27051: Create PIP gui http://bugs.python.org/issue27051 9 msgs #24225: Idlelib: changing file names http://bugs.python.org/issue24225 8 msgs #27122: Hang with contextlib.ExitStack and subprocess.Popen (regressio http://bugs.python.org/issue27122 8 msgs #27085: Make it possible to select return type for os.listdir http://bugs.python.org/issue27085 6 msgs #18373: let code force str(bytes) to raise an exception http://bugs.python.org/issue18373 5 msgs #27032: Remove deprecated html.parser.HTMLParser.unescape() http://bugs.python.org/issue27032 5 msgs #27072: random.getrandbits is limited to 2**31-1 bits on 64-bit Window http://bugs.python.org/issue27072 5 msgs Issues closed (41) ================== #5187: distutils upload should prompt for the user/password too http://bugs.python.org/issue5187 closed by berker.peksag #5784: raw deflate format and zlib module http://bugs.python.org/issue5784 closed by martin.panter #7159: Urllib2 authentication memory. http://bugs.python.org/issue7159 closed by terry.reedy #13615: setup.py register fails with -r argument http://bugs.python.org/issue13615 closed by berker.peksag #16509: sqlite3 docs do not explain check_same_thread http://bugs.python.org/issue16509 closed by berker.peksag #18383: test_warnings modifies warnings.filters when running with "-W http://bugs.python.org/issue18383 closed by martin.panter #23026: Winreg module doesn't support REG_QWORD, small DWORD doc updat http://bugs.python.org/issue23026 closed by python-dev #25823: Speed-up oparg decoding on little-endian machines http://bugs.python.org/issue25823 closed by serhiy.storchaka #26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure http://bugs.python.org/issue26168 closed by serhiy.storchaka #26306: Can't create abstract tuple http://bugs.python.org/issue26306 closed by r.david.murray #26742: imports in test_warnings changes warnings.filters http://bugs.python.org/issue26742 closed by martin.panter #26816: Make concurrent.futures.Executor an abc http://bugs.python.org/issue26816 closed by xiang.zhang #26877: tarfile use wrong code when read from fileobj http://bugs.python.org/issue26877 closed by mmarkk #27054: Python installation problem: No module named 'encodings' http://bugs.python.org/issue27054 closed by steve.dower #27060: Documentation of assertItemsEqual in unittest is VERY misleadi http://bugs.python.org/issue27060 closed by terry.reedy #27063: Some unittest loader tests are silently skipped by mistake http://bugs.python.org/issue27063 closed by serhiy.storchaka #27064: Make py.exe default to Python 3 when used interactively http://bugs.python.org/issue27064 closed by paul.moore #27070: Add ability to freeze (seal) mutable objects http://bugs.python.org/issue27070 closed by eric.smith #27071: unittest.TestCase.assertCountEqual is a very misleading name http://bugs.python.org/issue27071 closed by r.david.murray #27075: Link to StreamReader/Writer from codecs.getreader/writer http://bugs.python.org/issue27075 closed by berker.peksag #27076: Doc and comment spelling fixes http://bugs.python.org/issue27076 closed by martin.panter #27077: test_port_parameter_types fails with BytesWarning http://bugs.python.org/issue27077 closed by berker.peksag #27082: IDLE seriously degrades during and after printing large single http://bugs.python.org/issue27082 closed by terry.reedy #27087: unable to use socket send and sendall due to type error http://bugs.python.org/issue27087 closed by eric.smith #27089: I think this is a small bug in urlparse.py http://bugs.python.org/issue27089 closed by berker.peksag #27090: Python 3 import error after installation http://bugs.python.org/issue27090 closed by zach.ware #27091: Python IDLE doesn't like ???? http://bugs.python.org/issue27091 closed by eryksun #27092: ord() raises TypeError on string/bytes input http://bugs.python.org/issue27092 closed by cwr #27093: Silence warning in cjkcodecs.h http://bugs.python.org/issue27093 closed by python-dev #27094: Floating Point Printing Error with 2+.8553 or 3+.8553 http://bugs.python.org/issue27094 closed by r.david.murray #27096: Ability to get random bytes from random.Random (as with os.ura http://bugs.python.org/issue27096 closed by rhettinger #27097: ceval: Wordcode follow up, explicit unsigned short read http://bugs.python.org/issue27097 closed by serhiy.storchaka #27098: Pickle of memoryview not raising error http://bugs.python.org/issue27098 closed by serhiy.storchaka #27102: Spam http://bugs.python.org/issue27102 closed by zach.ware #27104: enable-framework http://bugs.python.org/issue27104 closed by alexandra.beebe at gmail.com #27114: SSLContext._load_windows_store_certs fails with PermissionErro http://bugs.python.org/issue27114 closed by steve.dower #27116: Documentation for regular expression alphanumeric matching are http://bugs.python.org/issue27116 closed by zach.ware #27118: Several Py_XDE/INCREFs in typeobject.c are not necessary http://bugs.python.org/issue27118 closed by serhiy.storchaka #27120: xmllib unable to parse in UTF8 format http://bugs.python.org/issue27120 closed by serhiy.storchaka #27134: allow str(bytes) raises an exception to be controlled programm http://bugs.python.org/issue27134 closed by dholth #27135: nested list produced with multiplication is linked to the same http://bugs.python.org/issue27135 closed by benjamin.peterson From mal at egenix.com Fri May 27 12:35:17 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 27 May 2016 18:35:17 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <7067504675934721722@unknownmsgid> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid> Message-ID: <57487745.2080900@egenix.com> On 27.05.2016 17:44, Chris Barker - NOAA Federal wrote: >>> , which aren't in any wide spread use yet and >> probably won't be for quite a few years ahead. > > Anything added to the stdlib now will be in py3.6+, yes? > > Which won't be in widespread use for quite a few years yet, either. > > So if ( and that's a big if) it's possible to anticipate what will be > in widespread use in a couple years, getting it in now would be a good > thing. You cut away the important part of what I said: "The current patch is 1.2MB for SHA-3 - that's pretty heavy for just a few hash functions, ..." If people want to use the hashes earlier, this is already possible via a separate package, so we're not delaying their use. It is clear that SHA-3 will get more traction in coming years (*), but I'm pretty sure that OpenSSL will have good implementations by the time people will actively start using the new hash algorithm and then hashlib will automatically make that available (hashlib uses the OpenSSL EVP abstraction, so will be able to use any new algorithms added to OpenSSL). However, if we add the reference implementation now, we'd then be left with 1.2MB unnecessary code in the stdlib. The question is not so much: is SHA-3 useful or not, it's whether we want to maintain this forever going forward or not. (*) People are just now starting to move from SHA-1 to SHA-2 and SHA-2 was standardized in 2001. Python received SHA-2 support in 2006. So there's plenty of time to decide :-) > -CHB > > > >> >> IMO, relying on OpenSSL is a better strategy than providing >> (and maintaining) our own compatibility versions. Until OpenSSL >> has them, people can use Bj?rn's package: >> >> https://github.com/bjornedstrom/python-sha3 >> >> Perhaps you could join forces with Bj?rn to create a standard >> SHA-3 standalone package on PyPI based on your two variants >> which we could recommend to people in the docs ?! >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Experts (#1, May 27 2016) >>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>>> Python Database Interfaces ... http://products.egenix.com/ >>>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ >> ________________________________________________________________________ >> >> ::: We implement business ideas - efficiently in both time and costs ::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> http://www.malemburg.com/ >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 27 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From chris.barker at noaa.gov Fri May 27 12:41:11 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 27 May 2016 09:41:11 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <57487745.2080900@egenix.com> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid> <57487745.2080900@egenix.com> Message-ID: On Fri, May 27, 2016 at 9:35 AM, M.-A. Lemburg wrote: > > So if ( and that's a big if) it's possible to anticipate what will be > > in widespread use in a couple years, getting it in now would be a good > > thing. > > You cut away the important part of what I said: > "The current patch is 1.2MB for SHA-3 - that's pretty heavy for > just a few hash functions, ..." > > If people want to use the hashes earlier, this is already possible > via a separate package, so we're not delaying their use. > That's true for ANY addition to the stdlib -- it could always be made available in a third party lib. (unless you want to use it in another part of the stdlib...) > It is clear that SHA-3 will get more traction in coming years (*), > but I'm pretty sure that OpenSSL will have good implementations by > the time people will actively start using the new hash algorithm > and then hashlib will automatically make that available (hashlib > uses the OpenSSL EVP abstraction, so will be able to use any > new algorithms added to OpenSSL). > > However, if we add the reference implementation now, we'd then be > left with 1.2MB unnecessary code in the stdlib. > I'm probably showing my ignorance here, but couldn't we swap in the OpenSSL implementation when that becomes available? -CHB (*) People are just now starting to move from SHA-1 to SHA-2 > and SHA-2 was standardized in 2001. Python received SHA-2 support > in 2006. So there's plenty of time to decide :-) can't deny the history, nor the inertia -- but that doesn't make it a good thing... -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri May 27 12:56:21 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 27 May 2016 18:56:21 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid> <57487745.2080900@egenix.com> Message-ID: <57487C35.9050102@egenix.com> On 27.05.2016 18:41, Chris Barker wrote: > On Fri, May 27, 2016 at 9:35 AM, M.-A. Lemburg wrote: > >>> So if ( and that's a big if) it's possible to anticipate what will be >>> in widespread use in a couple years, getting it in now would be a good >>> thing. >> >> You cut away the important part of what I said: >> "The current patch is 1.2MB for SHA-3 - that's pretty heavy for >> just a few hash functions, ..." >> >> If people want to use the hashes earlier, this is already possible >> via a separate package, so we're not delaying their use. >> > > That's true for ANY addition to the stdlib -- it could always be made > available in a third party lib. (unless you want to use it in another part > of the stdlib...) Well, any addition for which someone already wrote a package, but yes... >> It is clear that SHA-3 will get more traction in coming years (*), >> but I'm pretty sure that OpenSSL will have good implementations by >> the time people will actively start using the new hash algorithm >> and then hashlib will automatically make that available (hashlib >> uses the OpenSSL EVP abstraction, so will be able to use any >> new algorithms added to OpenSSL). >> >> However, if we add the reference implementation now, we'd then be >> left with 1.2MB unnecessary code in the stdlib. >> > > I'm probably showing my ignorance here, but couldn't we swap in the OpenSSL > implementation when that becomes available? We could, but only if we don't expose separate interfaces for the hashes and not add them to hashlib. hashlib.algorithms hashlib.algorithms_guaranteed > -CHB > > > (*) People are just now starting to move from SHA-1 to SHA-2 >> and SHA-2 was standardized in 2001. Python received SHA-2 support >> in 2006. So there's plenty of time to decide :-) > > > can't deny the history, nor the inertia -- but that doesn't make it a good > thing... > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 27 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From levkivskyi at gmail.com Fri May 27 14:58:26 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Fri, 27 May 2016 20:58:26 +0200 Subject: [Python-Dev] Adding NewType() to PEP 484 Message-ID: Hi, It has been proposed to enhance the typing module with a NewType function that allows to define simple unique types with almost zero runtime overhead. The PR containing actual implementation and PEP 484 update is here: https://github.com/python/typing/pull/226 Review comments are very welcome. Best regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 27 15:50:40 2016 From: brett at python.org (Brett Cannon) Date: Fri, 27 May 2016 19:50:40 +0000 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: Patches to Python's stdlib should go through bugs.python.org so it isn't lost in email. On Fri, May 27, 2016, 12:00 Ivan Levkivskyi wrote: > Hi, > > It has been proposed to enhance the typing module with a NewType function > that allows to define simple unique types with almost zero runtime > overhead. > > The PR containing actual implementation and PEP 484 update is here: > https://github.com/python/typing/pull/226 > > Review comments are very welcome. > > Best regards, > Ivan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Fri May 27 15:54:33 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Fri, 27 May 2016 12:54:33 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: Sorry, this is more meant to be the start of a discussion about the proposed feature. And typing.py has its own upstream repo (like asyncio). --Guido (mobile) On May 27, 2016 12:52 PM, "Brett Cannon" wrote: > Patches to Python's stdlib should go through bugs.python.org so it isn't > lost in email. > > On Fri, May 27, 2016, 12:00 Ivan Levkivskyi wrote: > >> Hi, >> >> It has been proposed to enhance the typing module with a NewType function >> that allows to define simple unique types with almost zero runtime >> overhead. >> >> The PR containing actual implementation and PEP 484 update is here: >> https://github.com/python/typing/pull/226 >> >> Review comments are very welcome. >> >> Best regards, >> Ivan >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri May 27 16:03:36 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 27 May 2016 22:03:36 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <57482779.6030100@egenix.com> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> Message-ID: Le vendredi 27 mai 2016, M.-A. Lemburg a ?crit : > > The curent patch is 1.2MB for SHA-3 - that's pretty heavy for just > a few hash functions, which aren't in any wide spread use yet and > probably won't be for quite a few years ahead. Oh wow, it's so fat? Why is it so big? Can't we use a lighter version? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Fri May 27 16:58:23 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 27 May 2016 15:58:23 -0500 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> Message-ID: On May 27, 2016 3:04 PM, "Victor Stinner" wrote: > > Le vendredi 27 mai 2016, M.-A. Lemburg a ?crit : >> >> The curent patch is 1.2MB for SHA-3 - that's pretty heavy for just >> >> a few hash functions, which aren't in any wide spread use yet and >> probably won't be for quite a few years ahead. > > > Oh wow, it's so fat? Why is it so big? Can't we use a lighter version? > The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which seems to be (as the file path implies) just test data. A whopping >1k LOC of really long hashes. > Victor > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri May 27 17:41:26 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 27 May 2016 23:41:26 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> Message-ID: <5748BF06.50606@egenix.com> On 27.05.2016 22:58, Ryan Gonzalez wrote: > On May 27, 2016 3:04 PM, "Victor Stinner" wrote: >> >> Le vendredi 27 mai 2016, M.-A. Lemburg a ?crit : >>> >>> The current patch is 1.2MB for SHA-3 - that's pretty heavy for just >>> a few hash functions, which aren't in any wide spread use yet and >>> probably won't be for quite a few years ahead. >> >> >> Oh wow, it's so fat? Why is it so big? Can't we use a lighter version? >> > > The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which > seems to be (as the file path implies) just test data. A whopping >1k LOC > of really long hashes. Right. There's about 1MB test data in the patch, but even without that data, the patch adds more than 6400 lines of code. If we add this now, there should at least be an exit strategy to remove the code again, when OpenSSL ships with the same code, IMO. Aside: BLAKE2 has already landed in OpenSSL 1.1.0: https://github.com/openssl/openssl/tree/master/crypto/blake2 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 27 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From donald at stufft.io Fri May 27 17:46:27 2016 From: donald at stufft.io (Donald Stufft) Date: Fri, 27 May 2016 17:46:27 -0400 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <5748BF06.50606@egenix.com> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <5748BF06.50606@egenix.com> Message-ID: <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io> > On May 27, 2016, at 5:41 PM, M.-A. Lemburg wrote: > > If we add this now, there should at least be an exit strategy > to remove the code again, when OpenSSL ships with the same > code, IMO. I think it is a clear win to have the fallback implementations in cases where people either don?t have OpenSSL or don?t have a new enough OpenSSL for those implementations. Not having the fallback just makes it more difficult for people to rely on those hash functions. ? Donald Stufft From mal at egenix.com Fri May 27 18:08:38 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 28 May 2016 00:08:38 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io> Message-ID: <5748C566.1070904@egenix.com> On 27.05.2016 23:46, Donald Stufft wrote: > >> On May 27, 2016, at 5:41 PM, M.-A. Lemburg wrote: >> >> If we add this now, there should at least be an exit strategy >> to remove the code again, when OpenSSL ships with the same >> code, IMO. > > I think it is a clear win to have the fallback implementations in cases where people either don?t have OpenSSL or don?t have a new enough OpenSSL for those implementations. Not having the fallback just makes it more difficult for people to rely on those hash functions. This will only be needed once the stdlib itself starts requiring support for some of these hashes and for that we could add a pure Python implementation, eg. https://github.com/coruus/py-keccak In all other cases, you can simply add the support via a package such as Bj?rn's or Christian's. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 27 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From njs at pobox.com Fri May 27 18:52:00 2016 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 27 May 2016 15:52:00 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <5748C566.1070904@egenix.com> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io> <5748C566.1070904@egenix.com> Message-ID: On Fri, May 27, 2016 at 3:08 PM, M.-A. Lemburg wrote: > On 27.05.2016 23:46, Donald Stufft wrote: >> >>> On May 27, 2016, at 5:41 PM, M.-A. Lemburg wrote: >>> >>> If we add this now, there should at least be an exit strategy >>> to remove the code again, when OpenSSL ships with the same >>> code, IMO. >> >> I think it is a clear win to have the fallback implementations in cases where people either don?t have OpenSSL or don?t have a new enough OpenSSL for those implementations. Not having the fallback just makes it more difficult for people to rely on those hash functions. > > This will only be needed once the stdlib itself starts requiring > support for some of these hashes and for that we could add > a pure Python implementation, eg. > > https://github.com/coruus/py-keccak > > In all other cases, you can simply add the support via a > package such as Bj?rn's or Christian's. SHA-3 and BLAKE are extremely widely accepted standards, our users will expect them, and they're significant improvements over all the current hashes in the algorithms_guaranteed list. If we demote them to second-class support (by making them only available in some builds, or using a slow pure Python implementation), then we'll be encouraging users to use inferior hashes. We shouldn't do this without a very good reason, and I don't see anything very convincing here... by all means drop the megabyte of test data, but why does it matter how many lines of code the algorithm is? No python developer will ever have to look at it -- hash code by its nature is *very* low maintenance (it either computes the right function or it doesn't, and the right answer never changes). And in unlikely case where some terrible unexpected bug is discovered then the only maintenance needed will be to delete the current impl and drop-in whatever the new fixed one is. So +1 to adding SHA-3 and BLAKE to algorithms_guaranteed. -n -- Nathaniel J. Smith -- https://vorpus.org From gvanrossum at gmail.com Fri May 27 19:01:11 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Fri, 27 May 2016 16:01:11 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: Also -- the most important thing. :-) What to call these things? We're pretty much settled on the semantics and how to create them (A = NewType('A', int)) but what should we call types like A when we're talking about them? "New types" sounds awkward. On Fri, May 27, 2016 at 12:54 PM, Guido van Rossum wrote: > Sorry, this is more meant to be the start of a discussion about the proposed > feature. And typing.py has its own upstream repo (like asyncio). > > --Guido (mobile) > > On May 27, 2016 12:52 PM, "Brett Cannon" wrote: >> >> Patches to Python's stdlib should go through bugs.python.org so it isn't >> lost in email. >> >> >> On Fri, May 27, 2016, 12:00 Ivan Levkivskyi wrote: >>> >>> Hi, >>> >>> It has been proposed to enhance the typing module with a NewType function >>> that allows to define simple unique types with almost zero runtime >>> overhead. >>> >>> The PR containing actual implementation and PEP 484 update is here: >>> https://github.com/python/typing/pull/226 >>> >>> Review comments are very welcome. >>> >>> Best regards, >>> Ivan >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > -- --Guido van Rossum (python.org/~guido) From mafagafogigante at gmail.com Fri May 27 19:19:15 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Fri, 27 May 2016 20:19:15 -0300 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io> <5748C566.1070904@egenix.com> Message-ID: <11e202b5-c129-b587-6594-45470591ebdf@gmail.com> On 05/27/2016 07:52 PM, Nathaniel Smith wrote: > If we demote them to second-class support (by making them only > available in some builds, or using a slow pure Python implementation), > then we'll be encouraging users to use inferior hashes. We shouldn't > do this without a very good reason. I agree. And I really think we shouldn't even ship pure Python implementations of these hashing algorithms. I am fairly confident that these algorithms would be prohibitively slow if written in pure Python. From steve at pearwood.info Fri May 27 20:58:31 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 28 May 2016 10:58:31 +1000 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: <20160528005826.GJ12028@ando.pearwood.info> On Fri, May 27, 2016 at 04:01:11PM -0700, Guido van Rossum wrote: > Also -- the most important thing. :-) What to call these things? We're > pretty much settled on the semantics and how to create them (A = > NewType('A', int)) but what should we call types like A when we're > talking about them? "New types" sounds awkward. TypeAlias? Because A is an alias for int? -- Steve From greg.ewing at canterbury.ac.nz Fri May 27 21:51:46 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 28 May 2016 13:51:46 +1200 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: <5748F9B2.5000604@canterbury.ac.nz> Guido van Rossum wrote: > Also -- the most important thing. :-) What to call these things? We're > pretty much settled on the semantics and how to create them (A = > NewType('A', int)) but what should we call types like A when we're > talking about them? "New types" sounds awkward. Fake types? Virtual types? Pseudo-types? -- Greg From greg.ewing at canterbury.ac.nz Fri May 27 21:59:02 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 28 May 2016 13:59:02 +1200 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: <20160528005826.GJ12028@ando.pearwood.info> References: <20160528005826.GJ12028@ando.pearwood.info> Message-ID: <5748FB66.7090107@canterbury.ac.nz> Steven D'Aprano wrote: > TypeAlias? Because A is an alias for int? That suggests it's just another name for the same type, but it's not. It's a distinct type as far as the static type checker is concerned. -- Greg From guido at python.org Sat May 28 00:26:29 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 27 May 2016 21:26:29 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: <5748FB66.7090107@canterbury.ac.nz> References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> Message-ID: On Fri, May 27, 2016 at 6:59 PM, Greg Ewing wrote: > Steven D'Aprano wrote: > >> TypeAlias? Because A is an alias for int? > > > That suggests it's just another name for the same type, > but it's not. It's a distinct type as far as the static > type checker is concerned. We discussed this over dinner at PyCon, some ideas we came up with: - Dependent types, harking back to a similar concept in Ada (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types) which in that language is also spelled with "new". - New type - Distinguished Type - Distinguished Subtype - Distinguished Type Alias - Distinguished Alias - BoatyMcBoatType The nice thing about "distinguished" is that it's a relatively rare word so it is easy to remember or look up. Personally I'm still in favor of Derived type (but I'm more into ancient programming languages than most folks here). I could also live with Distinguished Type. The problem with Alias is that these types are *not* aliases (at least not to the type checker, which is where we need the terminology -- so we can talk about what the type checker does to these). > Fake types? Virtual types? Pseudo-types? I'm not keen on any of these. -- --Guido van Rossum (python.org/~guido) From nd at perlig.de Sat May 28 07:26:57 2016 From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=) Date: Sat, 28 May 2016 13:26:57 +0200 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: <20160528005826.GJ12028@ando.pearwood.info> References: <20160528005826.GJ12028@ando.pearwood.info> Message-ID: <201605281326.57929@news.perlig.de> * Steven D'Aprano wrote: > On Fri, May 27, 2016 at 04:01:11PM -0700, Guido van Rossum wrote: > > Also -- the most important thing. :-) What to call these things? We're > > pretty much settled on the semantics and how to create them (A = > > NewType('A', int)) but what should we call types like A when we're > > talking about them? "New types" sounds awkward. > > TypeAlias? Because A is an alias for int? I like the view C takes on this: typedef. Cheers, -- Wer sein Wissen nicht teilen will, besitzt wahrscheinlich zu wenig davon. -- Unbekannt From steve at pearwood.info Sat May 28 08:24:45 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 28 May 2016 22:24:45 +1000 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> Message-ID: <20160528122445.GK12028@ando.pearwood.info> On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote: > We discussed this over dinner at PyCon, some ideas we came up with: > > - Dependent types, harking back to a similar concept in Ada > (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types) > which in that language is also spelled with "new". I started to explain this to my non-programmer wife, I got as far as explaining types, and that we need a name for this thing, and she stopped me and said "Please don't tell me this is leading to TypyMcTypeCheck." [...] > - BoatyMcBoatType > The nice thing about "distinguished" is that it's a relatively rare > word so it is easy to remember or look up. I would have thought that being rare, it would be *harder* to remember. > Personally I'm still in favor of Derived type (but I'm more into > ancient programming languages than most folks here). I could also live > with Distinguished Type. I think Derived Type is the nicest of the options. It accurately describes what it is: a type derived from another. And its shorter and easy to both say and write than "Distinguished type" (which sounds like "distinguished gentlemen" -- is it wearing a monocle and a top hat?). "Distinguished" is too vague for my tastes, it might as well be "flibblegubble type". *All* types are distinguished, the type checker has to distinguish int from float from list from str, so to call NewType("userid", int) a "distinguished type" is only to call it a type. -- Steve From guido at python.org Sat May 28 10:35:27 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 28 May 2016 07:35:27 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: <20160528122445.GK12028@ando.pearwood.info> References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> <20160528122445.GK12028@ando.pearwood.info> Message-ID: Just to add to the list of options, Twitter also came up with - invention - DomainType - TypedAlias But seriously I think we should just decide between Derived Type and Distinguished Type [Alias]. The latter comes from the idea that when you write e.g. UserId = int then UserId is a type alias (that's existing PEP 484 terminology) and the type checker doesn't distinguish it from int -- you can use it in places where you logically expect a UserId but to the type checker those variables have the type int. There is even a neat potential "origin story" that would explain why we'd call it Distinguished Type Alias. The story is about gradually converting a large code base to being consistent: initially you make UserId a regular type alias and you start putting it in your code incrementally, making sure it has no mypy errors as you go (but this just treats it as an int). After days, when you think you are done, you change UserId to a distinguished type alias and then mypy will point out the places where you've missed something or you're doing something questionable with user IDs. And yes, in the wider context of subclassing, Derived Type is probably confusing because a subclass is also called a derived class. On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano wrote: > On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote: > >> We discussed this over dinner at PyCon, some ideas we came up with: >> >> - Dependent types, harking back to a similar concept in Ada >> (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types) >> which in that language is also spelled with "new". > > I started to explain this to my non-programmer wife, I got as far as > explaining types, and that we need a name for this thing, and she > stopped me and said > > "Please don't tell me this is leading to TypyMcTypeCheck." > > [...] >> - BoatyMcBoatType > >> The nice thing about "distinguished" is that it's a relatively rare >> word so it is easy to remember or look up. > > I would have thought that being rare, it would be *harder* to remember. > > >> Personally I'm still in favor of Derived type (but I'm more into >> ancient programming languages than most folks here). I could also live >> with Distinguished Type. > > I think Derived Type is the nicest of the options. It accurately > describes what it is: a type derived from another. And its shorter and > easy to both say and write than "Distinguished type" (which sounds like > "distinguished gentlemen" -- is it wearing a monocle and a top hat?). > > "Distinguished" is too vague for my tastes, it might as well be > "flibblegubble type". *All* types are distinguished, the type checker > has to distinguish int from float from list from str, so to call > NewType("userid", int) a "distinguished type" is only to call it a type. > > > > -- > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From steve.dower at python.org Sat May 28 11:19:35 2016 From: steve.dower at python.org (Steve Dower) Date: Sat, 28 May 2016 08:19:35 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> <20160528122445.GK12028@ando.pearwood.info> Message-ID: Did anyone suggest "distinct type alias"? Regardless of what name, I'm fairly sure people will call it whatever the function to create it is called. So if the function is typings.distinguish_type(...), then distinguished will stick. Top-posted from my Windows Phone -----Original Message----- From: "Guido van Rossum" Sent: ?5/?28/?2016 7:38 To: "Steven D'Aprano" Cc: "Python-Dev" Subject: Re: [Python-Dev] Adding NewType() to PEP 484 Just to add to the list of options, Twitter also came up with - invention - DomainType - TypedAlias But seriously I think we should just decide between Derived Type and Distinguished Type [Alias]. The latter comes from the idea that when you write e.g. UserId = int then UserId is a type alias (that's existing PEP 484 terminology) and the type checker doesn't distinguish it from int -- you can use it in places where you logically expect a UserId but to the type checker those variables have the type int. There is even a neat potential "origin story" that would explain why we'd call it Distinguished Type Alias. The story is about gradually converting a large code base to being consistent: initially you make UserId a regular type alias and you start putting it in your code incrementally, making sure it has no mypy errors as you go (but this just treats it as an int). After days, when you think you are done, you change UserId to a distinguished type alias and then mypy will point out the places where you've missed something or you're doing something questionable with user IDs. And yes, in the wider context of subclassing, Derived Type is probably confusing because a subclass is also called a derived class. On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano wrote: > On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote: > >> We discussed this over dinner at PyCon, some ideas we came up with: >> >> - Dependent types, harking back to a similar concept in Ada >> (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types) >> which in that language is also spelled with "new". > > I started to explain this to my non-programmer wife, I got as far as > explaining types, and that we need a name for this thing, and she > stopped me and said > > "Please don't tell me this is leading to TypyMcTypeCheck." > > [...] >> - BoatyMcBoatType > >> The nice thing about "distinguished" is that it's a relatively rare >> word so it is easy to remember or look up. > > I would have thought that being rare, it would be *harder* to remember. > > >> Personally I'm still in favor of Derived type (but I'm more into >> ancient programming languages than most folks here). I could also live >> with Distinguished Type. > > I think Derived Type is the nicest of the options. It accurately > describes what it is: a type derived from another. And its shorter and > easy to both say and write than "Distinguished type" (which sounds like > "distinguished gentlemen" -- is it wearing a monocle and a top hat?). > > "Distinguished" is too vague for my tastes, it might as well be > "flibblegubble type". *All* types are distinguished, the type checker > has to distinguish int from float from list from str, so to call > NewType("userid", int) a "distinguished type" is only to call it a type. > > > > -- > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Sat May 28 05:01:03 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sat, 28 May 2016 11:01:03 +0200 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> Message-ID: On 28 May 2016 at 06:26, Guido van Rossum wrote: > Personally I'm still in favor of Derived type (but I'm more into > ancient programming languages than most folks here). I could also live > with Distinguished Type. I think both "derived" and "distinguished" are OK, but I am more in favour of "distinguished". The reason is sometimes people might talk about derived types while meaning normal derived classes. Also, indeed "distinguished" is a rare word, so that it will not cause confusions and easy to look up. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Sat May 28 06:46:55 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sat, 28 May 2016 12:46:55 +0200 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> Message-ID: There is another interesting idea from Twitter: call them type wrappers. -- Ivan On 28 May 2016 at 11:01, Ivan Levkivskyi wrote: > > On 28 May 2016 at 06:26, Guido van Rossum wrote: > > > Personally I'm still in favor of Derived type (but I'm more into > > ancient programming languages than most folks here). I could also live > > with Distinguished Type. > > I think both "derived" and "distinguished" are OK, but I am more in favour of "distinguished". > The reason is sometimes people might talk about derived types while meaning normal derived classes. > Also, indeed "distinguished" is a rare word, so that it will not cause confusions and easy to look up. > > -- > Ivan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat May 28 13:11:54 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 28 May 2016 10:11:54 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> <20160528122445.GK12028@ando.pearwood.info> Message-ID: Ohhhhh, another D-word! I really like distinct. On Sat, May 28, 2016 at 8:19 AM, Steve Dower wrote: > Did anyone suggest "distinct type alias"? > > Regardless of what name, I'm fairly sure people will call it whatever the > function to create it is called. So if the function is > typings.distinguish_type(...), then distinguished will stick. > > Top-posted from my Windows Phone > ________________________________ > From: Guido van Rossum > Sent: ?5/?28/?2016 7:38 > To: Steven D'Aprano > Cc: Python-Dev > Subject: Re: [Python-Dev] Adding NewType() to PEP 484 > > Just to add to the list of options, Twitter also came up with > > - invention > > - DomainType > > - TypedAlias > > But seriously I think we should just decide between Derived Type and > Distinguished Type [Alias]. > > The latter comes from the idea that when you write e.g. > > UserId = int > > then UserId is a type alias (that's existing PEP 484 terminology) and > the type checker doesn't distinguish it from int -- you can use it in > places where you logically expect a UserId but to the type checker > those variables have the type int. > > There is even a neat potential "origin story" that would explain why > we'd call it Distinguished Type Alias. > > The story is about gradually converting a large code base to being > consistent: initially you make UserId a regular type alias and you > start putting it in your code incrementally, making sure it has no > mypy errors as you go (but this just treats it as an int). After days, > when you think you are done, you change UserId to a distinguished type > alias and then mypy will point out the places where you've missed > something or you're doing something questionable with user IDs. > > And yes, in the wider context of subclassing, Derived Type is probably > confusing because a subclass is also called a derived class. > > > On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano > wrote: >> On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote: >> >>> We discussed this over dinner at PyCon, some ideas we came up with: >>> >>> - Dependent types, harking back to a similar concept in Ada >>> (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types) >>> which in that language is also spelled with "new". >> >> I started to explain this to my non-programmer wife, I got as far as >> explaining types, and that we need a name for this thing, and she >> stopped me and said >> >> "Please don't tell me this is leading to TypyMcTypeCheck." >> >> [...] >>> - BoatyMcBoatType >> >>> The nice thing about "distinguished" is that it's a relatively rare >>> word so it is easy to remember or look up. >> >> I would have thought that being rare, it would be *harder* to remember. >> >> >>> Personally I'm still in favor of Derived type (but I'm more into >>> ancient programming languages than most folks here). I could also live >>> with Distinguished Type. >> >> I think Derived Type is the nicest of the options. It accurately >> describes what it is: a type derived from another. And its shorter and >> easy to both say and write than "Distinguished type" (which sounds like >> "distinguished gentlemen" -- is it wearing a monocle and a top hat?). >> >> "Distinguished" is too vague for my tastes, it might as well be >> "flibblegubble type". *All* types are distinguished, the type checker >> has to distinguish int from float from list from str, so to call >> NewType("userid", int) a "distinguished type" is only to call it a type. >> >> >> >> -- >> Steve >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -- --Guido van Rossum (python.org/~guido) From levkivskyi at gmail.com Sat May 28 13:17:33 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sat, 28 May 2016 19:17:33 +0200 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> <20160528122445.GK12028@ando.pearwood.info> Message-ID: My final vote goes to "distinct type alias". But how should we call the function? NewType? Or should we change it to DistinctAlias or something? -- Ivan 28 ??? 2016 19:14 "Guido van Rossum" ????: > Ohhhhh, another D-word! I really like distinct. > > On Sat, May 28, 2016 at 8:19 AM, Steve Dower > wrote: > > Did anyone suggest "distinct type alias"? > > > > Regardless of what name, I'm fairly sure people will call it whatever the > > function to create it is called. So if the function is > > typings.distinguish_type(...), then distinguished will stick. > > > > Top-posted from my Windows Phone > > ________________________________ > > From: Guido van Rossum > > Sent: ?5/?28/?2016 7:38 > > To: Steven D'Aprano > > Cc: Python-Dev > > Subject: Re: [Python-Dev] Adding NewType() to PEP 484 > > > > Just to add to the list of options, Twitter also came up with > > > > - invention > > > > - DomainType > > > > - TypedAlias > > > > But seriously I think we should just decide between Derived Type and > > Distinguished Type [Alias]. > > > > The latter comes from the idea that when you write e.g. > > > > UserId = int > > > > then UserId is a type alias (that's existing PEP 484 terminology) and > > the type checker doesn't distinguish it from int -- you can use it in > > places where you logically expect a UserId but to the type checker > > those variables have the type int. > > > > There is even a neat potential "origin story" that would explain why > > we'd call it Distinguished Type Alias. > > > > The story is about gradually converting a large code base to being > > consistent: initially you make UserId a regular type alias and you > > start putting it in your code incrementally, making sure it has no > > mypy errors as you go (but this just treats it as an int). After days, > > when you think you are done, you change UserId to a distinguished type > > alias and then mypy will point out the places where you've missed > > something or you're doing something questionable with user IDs. > > > > And yes, in the wider context of subclassing, Derived Type is probably > > confusing because a subclass is also called a derived class. > > > > > > On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano > > wrote: > >> On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote: > >> > >>> We discussed this over dinner at PyCon, some ideas we came up with: > >>> > >>> - Dependent types, harking back to a similar concept in Ada > >>> ( > https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types) > >>> which in that language is also spelled with "new". > >> > >> I started to explain this to my non-programmer wife, I got as far as > >> explaining types, and that we need a name for this thing, and she > >> stopped me and said > >> > >> "Please don't tell me this is leading to TypyMcTypeCheck." > >> > >> [...] > >>> - BoatyMcBoatType > >> > >>> The nice thing about "distinguished" is that it's a relatively rare > >>> word so it is easy to remember or look up. > >> > >> I would have thought that being rare, it would be *harder* to remember. > >> > >> > >>> Personally I'm still in favor of Derived type (but I'm more into > >>> ancient programming languages than most folks here). I could also live > >>> with Distinguished Type. > >> > >> I think Derived Type is the nicest of the options. It accurately > >> describes what it is: a type derived from another. And its shorter and > >> easy to both say and write than "Distinguished type" (which sounds like > >> "distinguished gentlemen" -- is it wearing a monocle and a top hat?). > >> > >> "Distinguished" is too vague for my tastes, it might as well be > >> "flibblegubble type". *All* types are distinguished, the type checker > >> has to distinguish int from float from list from str, so to call > >> NewType("userid", int) a "distinguished type" is only to call it a type. > >> > >> > >> > >> -- > >> Steve > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > > -- > > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > > https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pludemann at google.com Sat May 28 13:43:22 2016 From: pludemann at google.com (Peter Ludemann) Date: Sat, 28 May 2016 10:43:22 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> <20160528122445.GK12028@ando.pearwood.info> Message-ID: I'm surprised that nobody has suggested UniqueType. ;) On 28 May 2016 at 10:17, Ivan Levkivskyi wrote: > My final vote goes to "distinct type alias". > But how should we call the function? NewType? Or should we change it to > DistinctAlias or something? > > -- > Ivan > 28 ??? 2016 19:14 "Guido van Rossum" ????: > >> Ohhhhh, another D-word! I really like distinct. >> >> On Sat, May 28, 2016 at 8:19 AM, Steve Dower >> wrote: >> > Did anyone suggest "distinct type alias"? >> > >> > Regardless of what name, I'm fairly sure people will call it whatever >> the >> > function to create it is called. So if the function is >> > typings.distinguish_type(...), then distinguished will stick. >> > >> > Top-posted from my Windows Phone >> > ________________________________ >> > From: Guido van Rossum >> > Sent: ?5/?28/?2016 7:38 >> > To: Steven D'Aprano >> > Cc: Python-Dev >> > Subject: Re: [Python-Dev] Adding NewType() to PEP 484 >> > >> > Just to add to the list of options, Twitter also came up with >> > >> > - invention >> > >> > - DomainType >> > >> > - TypedAlias >> > >> > But seriously I think we should just decide between Derived Type and >> > Distinguished Type [Alias]. >> > >> > The latter comes from the idea that when you write e.g. >> > >> > UserId = int >> > >> > then UserId is a type alias (that's existing PEP 484 terminology) and >> > the type checker doesn't distinguish it from int -- you can use it in >> > places where you logically expect a UserId but to the type checker >> > those variables have the type int. >> > >> > There is even a neat potential "origin story" that would explain why >> > we'd call it Distinguished Type Alias. >> > >> > The story is about gradually converting a large code base to being >> > consistent: initially you make UserId a regular type alias and you >> > start putting it in your code incrementally, making sure it has no >> > mypy errors as you go (but this just treats it as an int). After days, >> > when you think you are done, you change UserId to a distinguished type >> > alias and then mypy will point out the places where you've missed >> > something or you're doing something questionable with user IDs. >> > >> > And yes, in the wider context of subclassing, Derived Type is probably >> > confusing because a subclass is also called a derived class. >> > >> > >> > On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano >> > wrote: >> >> On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote: >> >> >> >>> We discussed this over dinner at PyCon, some ideas we came up with: >> >>> >> >>> - Dependent types, harking back to a similar concept in Ada >> >>> ( >> https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types) >> >>> which in that language is also spelled with "new". >> >> >> >> I started to explain this to my non-programmer wife, I got as far as >> >> explaining types, and that we need a name for this thing, and she >> >> stopped me and said >> >> >> >> "Please don't tell me this is leading to TypyMcTypeCheck." >> >> >> >> [...] >> >>> - BoatyMcBoatType >> >> >> >>> The nice thing about "distinguished" is that it's a relatively rare >> >>> word so it is easy to remember or look up. >> >> >> >> I would have thought that being rare, it would be *harder* to remember. >> >> >> >> >> >>> Personally I'm still in favor of Derived type (but I'm more into >> >>> ancient programming languages than most folks here). I could also live >> >>> with Distinguished Type. >> >> >> >> I think Derived Type is the nicest of the options. It accurately >> >> describes what it is: a type derived from another. And its shorter and >> >> easy to both say and write than "Distinguished type" (which sounds like >> >> "distinguished gentlemen" -- is it wearing a monocle and a top hat?). >> >> >> >> "Distinguished" is too vague for my tastes, it might as well be >> >> "flibblegubble type". *All* types are distinguished, the type checker >> >> has to distinguish int from float from list from str, so to call >> >> NewType("userid", int) a "distinguished type" is only to call it a >> type. >> >> >> >> >> >> >> >> -- >> >> Steve >> >> _______________________________________________ >> >> Python-Dev mailing list >> >> Python-Dev at python.org >> >> https://mail.python.org/mailman/listinfo/python-dev >> >> Unsubscribe: >> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > >> > >> > >> > -- >> > --Guido van Rossum (python.org/~guido) >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > >> https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/pludemann%40google.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat May 28 14:46:10 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 29 May 2016 03:46:10 +0900 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> <20160528122445.GK12028@ando.pearwood.info> Message-ID: <22345.59250.962744.337024@turnbull.sk.tsukuba.ac.jp> Guido van Rossum writes: > But seriously I think we should just decide between Derived Type and > Distinguished Type [Alias]. I like "typedef", but I'm pretty sure it wouldn't carry the same connotations to people who aren't C programming oldtimers. I dislike "derived" because that fits stuff like Union[int, str] or List[int] *better* than NewType("A", int). Bottom line: +1 for distinguished. I may have a little trouble remembering that NewType creates Distinguished Types (do they get a medal from the Queen, by the way?), but if you call something a "distinguished type" I will have zero trouble remembering that it's a new type created out of thin air by distinguishing it from an existing type. From mafagafogigante at gmail.com Sat May 28 14:49:09 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Sat, 28 May 2016 15:49:09 -0300 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> <20160528122445.GK12028@ando.pearwood.info> Message-ID: <281d0886-9202-851e-ac8b-8da5de1ad3f4@gmail.com> On 05/28/2016 12:19 PM, Steve Dower wrote: > Did anyone suggest "distinct type alias"? > I would just like to mention that "distinguished" seems to be more often associated with notability and excellence than "distinct", which is usually more neutral towards the quality of what it describes. Unless we want to say that these types are nobler than their counterparts, "distinguished" seems worse than "distinct". From christian at python.org Sat May 28 16:44:07 2016 From: christian at python.org (Christian Heimes) Date: Sat, 28 May 2016 13:44:07 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> Message-ID: <5913a7dc-5c67-6b30-1e34-71a729aa09bd@python.org> On 2016-05-27 03:44, Victor Stinner wrote: > Le 27 mai 2016 12:05 PM, "Donald Stufft" > a ?crit : >> BLAKE2 is an interesting one, because while SHA3 is a NIST standard > (so it?s going to gain adoption because of that), BLAKE2 is at least as > strong as SHA3 but is better in many ways, particularly in speed? it?s > actually faster than MD5 while being as secure as SHA3. > > BLAKE2 was part of the SHA3 competition and it was in finalists. The > SHA3 competition is interesting because each algorithm is deeply tested > and analyzed by many teams all around the world. Obvious vulnerabilities > are quickly found. Thanks Victor, minor correction, BLAKE was a finalist in the SHA3 competition, not BLAKE2. BLAKE2 is an improved version of BLAKE2 with additional features. Christian From christian at python.org Sat May 28 16:56:48 2016 From: christian at python.org (Christian Heimes) Date: Sat, 28 May 2016 13:56:48 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <57482779.6030100@egenix.com> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> Message-ID: <6c634ad5-5004-86a9-6338-8233c0c02b65@python.org> On 2016-05-27 03:54, M.-A. Lemburg wrote: > On 27.05.2016 06:54, Raymond Hettinger wrote: >> >>> On May 25, 2016, at 3:29 AM, Christian Heimes wrote: >>> >>> I have three hashing-related patches for Python 3.6 that are waiting for >>> review. Altogether the three patches add ten new hash algorithms to the >>> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256), >>> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256). >> >> Do we really need ten? I don't think the standard library is the place to offer all variants of hashing. And we should avoid getting in a cycle of "this was just released by NIST" and "nobody uses that one anymore". Is any one of them an emergent best practice (i.e. starting to be commonly used in network protocols because it is better, faster, stronger, etc)? >> >> Your last message on https://bugs.python.org/issue16113 suggests that these aren't essential and that there is room for debate about whether some of them are standard-library worthy (i.e. we will have them around forever). > > I can understand your eagerness to get this landed, since it's > been 4 years since work started, but I think we should wait with > the addition until OpenSSL has them: > > https://github.com/openssl/openssl/issues/439 > > The current patch is 1.2MB for SHA-3 - that's pretty heavy for just > a few hash functions, which aren't in any wide spread use yet and > probably won't be for quite a few years ahead. About 1 MB of the 1.2 MB are test vectors for SHA3. Strictly speaking the test vectors are not required. > IMO, relying on OpenSSL is a better strategy than providing > (and maintaining) our own compatibility versions. Until OpenSSL > has them, people can use Bj?rn's package: > > https://github.com/bjornedstrom/python-sha3 > > Perhaps you could join forces with Bj?rn to create a standard > SHA-3 standalone package on PyPI based on your two variants > which we could recommend to people in the docs ?! I have been maintaining my own SHA3 module for couple of years. A month ago I moved my code to github and ported it to the new Keccak Code Package. The standalone package uses the same code as my patch but also provides the old Keccak hashes and works on Python 2.7. https://github.com/tiran/pysha3 https://pypi.python.org/pypi/pysha3 From christian at python.org Sat May 28 17:01:35 2016 From: christian at python.org (Christian Heimes) Date: Sat, 28 May 2016 14:01:35 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid> <57487745.2080900@egenix.com> Message-ID: On 2016-05-27 09:41, Chris Barker wrote: > I'm probably showing my ignorance here, but couldn't we swap in the > OpenSSL implementation when that becomes available? No, not any time soon. As soon as we guarantee SHA3 support we have to keep our own implementation for a couple of additional releases. We can drop our own SHA3 code as soon as all supported OpenSSL versions have SHA3. For example when OpenSSL 1.2.0 is going to have SHA3 support, we must wait until OpenSSL 1.1 and 1.0.2 are no longer supported by OpenSSL. Christian From donald at stufft.io Sat May 28 17:04:55 2016 From: donald at stufft.io (Donald Stufft) Date: Sat, 28 May 2016 17:04:55 -0400 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid> <57487745.2080900@egenix.com> Message-ID: <7F576C05-72CC-48D6-82A7-6EF0A94F9B92@stufft.io> > On May 28, 2016, at 5:01 PM, Christian Heimes wrote: > > No, not any time soon. As soon as we guarantee SHA3 support we have to > keep our own implementation for a couple of additional releases. We can > drop our own SHA3 code as soon as all supported OpenSSL versions have SHA3. It still will be needed for as long as it?s possible to build Python without OpenSSL. ? Donald Stufft From brett at python.org Sat May 28 17:06:49 2016 From: brett at python.org (Brett Cannon) Date: Sat, 28 May 2016 21:06:49 +0000 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <6c634ad5-5004-86a9-6338-8233c0c02b65@python.org> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <6c634ad5-5004-86a9-6338-8233c0c02b65@python.org> Message-ID: On Sat, May 28, 2016, 13:58 Christian Heimes wrote: > On 2016-05-27 03:54, M.-A. Lemburg wrote: > > On 27.05.2016 06:54, Raymond Hettinger wrote: > >> > >>> On May 25, 2016, at 3:29 AM, Christian Heimes > wrote: > >>> > >>> I have three hashing-related patches for Python 3.6 that are waiting > for > >>> review. Altogether the three patches add ten new hash algorithms to the > >>> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256), > >>> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256). > >> > >> Do we really need ten? I don't think the standard library is the place > to offer all variants of hashing. And we should avoid getting in a cycle > of "this was just released by NIST" and "nobody uses that one anymore". Is > any one of them an emergent best practice (i.e. starting to be commonly > used in network protocols because it is better, faster, stronger, etc)? > >> > >> Your last message on https://bugs.python.org/issue16113 suggests that > these aren't essential and that there is room for debate about whether some > of them are standard-library worthy (i.e. we will have them around forever). > > > > I can understand your eagerness to get this landed, since it's > > been 4 years since work started, but I think we should wait with > > the addition until OpenSSL has them: > > > > https://github.com/openssl/openssl/issues/439 > > > > The current patch is 1.2MB for SHA-3 - that's pretty heavy for just > > a few hash functions, which aren't in any wide spread use yet and > > probably won't be for quite a few years ahead. > > About 1 MB of the 1.2 MB are test vectors for SHA3. Strictly speaking > the test vectors are not required. > We can always make the test vector file an external download like we do for some of the codec tests. -brett > > IMO, relying on OpenSSL is a better strategy than providing > > (and maintaining) our own compatibility versions. Until OpenSSL > > has them, people can use Bj?rn's package: > > > > https://github.com/bjornedstrom/python-sha3 > > > > Perhaps you could join forces with Bj?rn to create a standard > > SHA-3 standalone package on PyPI based on your two variants > > which we could recommend to people in the docs ?! > > I have been maintaining my own SHA3 module for couple of years. A month > ago I moved my code to github and ported it to the new Keccak Code > Package. The standalone package uses the same code as my patch but also > provides the old Keccak hashes and works on Python 2.7. > > https://github.com/tiran/pysha3 > https://pypi.python.org/pypi/pysha3 > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat May 28 17:06:43 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 28 May 2016 14:06:43 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid> <57487745.2080900@egenix.com> Message-ID: But you could choose which implementation to use at compile time based on the autoconf output, right? On Sat, May 28, 2016 at 2:01 PM, Christian Heimes wrote: > On 2016-05-27 09:41, Chris Barker wrote: >> I'm probably showing my ignorance here, but couldn't we swap in the >> OpenSSL implementation when that becomes available? > > No, not any time soon. As soon as we guarantee SHA3 support we have to > keep our own implementation for a couple of additional releases. We can > drop our own SHA3 code as soon as all supported OpenSSL versions have SHA3. > > For example when OpenSSL 1.2.0 is going to have SHA3 support, we must > wait until OpenSSL 1.1 and 1.0.2 are no longer supported by OpenSSL. > > Christian > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From christian at python.org Sat May 28 17:13:28 2016 From: christian at python.org (Christian Heimes) Date: Sat, 28 May 2016 14:13:28 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: <5748BF06.50606@egenix.com> References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <5748BF06.50606@egenix.com> Message-ID: On 2016-05-27 14:41, M.-A. Lemburg wrote: > On 27.05.2016 22:58, Ryan Gonzalez wrote: >> On May 27, 2016 3:04 PM, "Victor Stinner" wrote: >>> >>> Le vendredi 27 mai 2016, M.-A. Lemburg a ?crit : >>>> >>>> The current patch is 1.2MB for SHA-3 - that's pretty heavy for just >>>> a few hash functions, which aren't in any wide spread use yet and >>>> probably won't be for quite a few years ahead. >>> >>> >>> Oh wow, it's so fat? Why is it so big? Can't we use a lighter version? >>> >> >> The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which >> seems to be (as the file path implies) just test data. A whopping >1k LOC >> of really long hashes. > > Right. There's about 1MB test data in the patch, but even > without that data, the patch adds more than 6400 lines of code. The KeccakCodePackage is rather large. I already removed all unnecessary files and modified some files so more code is shared between 32 and 64bit optimized variants. Please keep in mind that the KCP contains multiple implementations with different optimizations for CPU architectures. I already removed the ARM NEON optimization. I also don't get your obsession with lines of code. The gzip and expat are far bigger than the KeccakCodePackage. > If we add this now, there should at least be an exit strategy > to remove the code again, when OpenSSL ships with the same > code, IMO. > > Aside: BLAKE2 has already landed in OpenSSL 1.1.0: > > https://github.com/openssl/openssl/tree/master/crypto/blake2 Except BLAKE2 in OpenSSL is severely castrated and tailored towards a very limited use case. The implementation does not support any of the useful advanced features like keyed hashing (MAC), salt, personalization, tree hashing and variable hash length. From donald at stufft.io Sat May 28 17:18:52 2016 From: donald at stufft.io (Donald Stufft) Date: Sat, 28 May 2016 17:18:52 -0400 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid> <57487745.2080900@egenix.com> Message-ID: <27853AD8-D31F-464F-A3F1-4B94907492AB@stufft.io> > On May 28, 2016, at 5:06 PM, Guido van Rossum wrote: > > But you could choose which implementation to use at compile time based > on the autoconf output, right? I think we should follow what hashlib already does. If we want to change the way it works that's fine but these hashes shouldn't be special. They should work the way that all the other standard hashes in hashlib work. From christian at python.org Sat May 28 17:30:37 2016 From: christian at python.org (Christian Heimes) Date: Sat, 28 May 2016 14:30:37 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid> <57487745.2080900@egenix.com> Message-ID: <71750130-8286-cb64-63e0-6d869145c512@python.org> On 2016-05-28 14:06, Guido van Rossum wrote: > But you could choose which implementation to use at compile time based > on the autoconf output, right? We compile all modules and then let hashlib decide which implementation is used. hashlib prefers OpenSSL but falls back to our builtin modules. For MD5, SHA1 and SHA2 OpenSSL's implementation has better performance (up to twice the speed). From christian at python.org Sat May 28 17:32:40 2016 From: christian at python.org (Christian Heimes) Date: Sat, 28 May 2016 14:32:40 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <6c634ad5-5004-86a9-6338-8233c0c02b65@python.org> Message-ID: On 2016-05-28 14:06, Brett Cannon wrote: > We can always make the test vector file an external download like we do > for some of the codec tests. That is actually a great idea! :) Thanks Brett From christian at python.org Sat May 28 17:56:00 2016 From: christian at python.org (Christian Heimes) Date: Sat, 28 May 2016 14:56:00 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io> <5748C566.1070904@egenix.com> Message-ID: On 2016-05-27 15:52, Nathaniel Smith wrote: > On Fri, May 27, 2016 at 3:08 PM, M.-A. Lemburg wrote: >> On 27.05.2016 23:46, Donald Stufft wrote: >>> >>>> On May 27, 2016, at 5:41 PM, M.-A. Lemburg wrote: >>>> >>>> If we add this now, there should at least be an exit strategy >>>> to remove the code again, when OpenSSL ships with the same >>>> code, IMO. >>> >>> I think it is a clear win to have the fallback implementations in cases where people either don?t have OpenSSL or don?t have a new enough OpenSSL for those implementations. Not having the fallback just makes it more difficult for people to rely on those hash functions. >> >> This will only be needed once the stdlib itself starts requiring >> support for some of these hashes and for that we could add >> a pure Python implementation, eg. >> >> https://github.com/coruus/py-keccak >> >> In all other cases, you can simply add the support via a >> package such as Bj?rn's or Christian's. > > SHA-3 and BLAKE are extremely widely accepted standards, our users > will expect them, and they're significant improvements over all the > current hashes in the algorithms_guaranteed list. If we demote them to > second-class support (by making them only available in some builds, or > using a slow pure Python implementation), then we'll be encouraging > users to use inferior hashes. We shouldn't do this without a very good > reason, and I don't see anything very convincing here... by all means > drop the megabyte of test data, but why does it matter how many lines > of code the algorithm is? No python developer will ever have to look > at it -- hash code by its nature is *very* low maintenance (it either > computes the right function or it doesn't, and the right answer never > changes). And in unlikely case where some terrible unexpected bug is > discovered then the only maintenance needed will be to delete the > current impl and drop-in whatever the new fixed one is. > > So +1 to adding SHA-3 and BLAKE to algorithms_guaranteed. Thanks Nathaniel, my patches don't add SHA3 and BLAKE2 to algorithms_guaranteed because Python still supports C89 platforms without a 64 bit integer type. Theoretically 64bit ints are not required except for BLAKE2b. Since Trent's snakebite.org is dead I don't have access to these old platforms any more. Christian From chris.jerdonek at gmail.com Sat May 28 18:59:35 2016 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 28 May 2016 15:59:35 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> Message-ID: On Fri, May 27, 2016 at 9:26 PM, Guido van Rossum wrote: > We discussed this over dinner at PyCon, some ideas we came up with: > > - Dependent types, harking back to a similar concept in Ada > (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types) > which in that language is also spelled with "new". > > - New type > > - Distinguished Type > > - Distinguished Subtype > > - Distinguished Type Alias > > - Distinguished Alias > > - BoatyMcBoatType Some more suggestions: - Cloned Type (or Type Clone) - Copied Type (or Type Copy) - Named Type - Renamed Type - Twin Type - Wrapped Type - Doppelganger Type (not serious) --Chris From cs at zip.com.au Sat May 28 19:24:23 2016 From: cs at zip.com.au (cs at zip.com.au) Date: Sun, 29 May 2016 09:24:23 +1000 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: <20160528232423.GA78519@cskk.homeip.net> On 28May2016 08:19, Steve Dower wrote: >Did anyone suggest "distinct type alias"? > >Regardless of what name, I'm fairly sure people will call it whatever the >function to create it is called. So if the function is >typings.distinguish_type(...), then distinguished will stick. Just casting an opinion in support of Greg Ewing's remark: I don't think we should use the word "alias", regardless of what else is used. In <5748FB66.7090107 at canterbury.ac.nz>, Greg said: Steven D'Aprano wrote: > TypeAlias? Because A is an alias for int? That suggests it's just another name for the same type, but it's not. It's a distinct type as far as the static type checker is concerned and I agree entirely. Cheers, Cameron Simpson From levkivskyi at gmail.com Sun May 29 01:07:17 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sun, 29 May 2016 07:07:17 +0200 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <20160528005826.GJ12028@ando.pearwood.info> <5748FB66.7090107@canterbury.ac.nz> Message-ID: Type clone sounds good. But I am still inclined more towards "distinct". -- Ivan 29 ??? 2016 01:04 "Chris Jerdonek" ????: > On Fri, May 27, 2016 at 9:26 PM, Guido van Rossum > wrote: > > We discussed this over dinner at PyCon, some ideas we came up with: > > > > - Dependent types, harking back to a similar concept in Ada > > (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types > ) > > which in that language is also spelled with "new". > > > > - New type > > > > - Distinguished Type > > > > - Distinguished Subtype > > > > - Distinguished Type Alias > > > > - Distinguished Alias > > > > - BoatyMcBoatType > > Some more suggestions: > > - Cloned Type (or Type Clone) > > - Copied Type (or Type Copy) > > - Named Type > > - Renamed Type > > - Twin Type > > - Wrapped Type > > - Doppelganger Type (not serious) > > --Chris > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sun May 29 02:51:23 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 29 May 2016 08:51:23 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io> <5748C566.1070904@egenix.com> Message-ID: Python 3.5 requires a 64 bit signed integer to build. Search for _PyTime type in pytime.h ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Sun May 29 10:08:24 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 29 May 2016 16:08:24 +0200 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <5748BF06.50606@egenix.com> Message-ID: <574AF7D8.30406@egenix.com> On 28.05.2016 23:13, Christian Heimes wrote: > On 2016-05-27 14:41, M.-A. Lemburg wrote: >> On 27.05.2016 22:58, Ryan Gonzalez wrote: >>> On May 27, 2016 3:04 PM, "Victor Stinner" wrote: >>>> >>>> Le vendredi 27 mai 2016, M.-A. Lemburg a ?crit : >>>>> >>>>> The current patch is 1.2MB for SHA-3 - that's pretty heavy for just >>>>> a few hash functions, which aren't in any wide spread use yet and >>>>> probably won't be for quite a few years ahead. >>>> >>>> >>>> Oh wow, it's so fat? Why is it so big? Can't we use a lighter version? >>>> >>> >>> The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which >>> seems to be (as the file path implies) just test data. A whopping >1k LOC >>> of really long hashes. >> >> Right. There's about 1MB test data in the patch, but even >> without that data, the patch adds more than 6400 lines of code. > > The KeccakCodePackage is rather large. I already removed all unnecessary > files and modified some files so more code is shared between 32 and > 64bit optimized variants. Please keep in mind that the KCP contains > multiple implementations with different optimizations for CPU > architectures. I already removed the ARM NEON optimization. > I also don't get your obsession with lines of code. The gzip and expat > are far bigger than the KeccakCodePackage. For a small piece of code, it's fine to have a copy in the stdlib, but for larger chunks such as this one, I think we ought to consider alternative options, since I don't think it's good to have to carry around this baggage forever. OpenSSL will eventually have good enough support for what most Python users will need from these new hash functions. That's why I think it's better to have a discussion of whether we need to full package in the stdlib or better only provide limited support built into the stdlib and refer people to PyPI packages for things that you don't need every day. Regarding the stories for zlib and expat, I only remember that expat was essentially unmaintained when we added it and the existing version at the time had known bugs (but could be wrong). For zlib, I have no clue as to why we have a copy in the stdlib. That lib is available on all systems nowadays. Perhaps it wasn't when we added it; don't remember. If so, it's a good example of why adding copies to the stdlib is not such a good idea :-) >> If we add this now, there should at least be an exit strategy >> to remove the code again, when OpenSSL ships with the same >> code, IMO. >> >> Aside: BLAKE2 has already landed in OpenSSL 1.1.0: >> >> https://github.com/openssl/openssl/tree/master/crypto/blake2 > > Except BLAKE2 in OpenSSL is severely castrated and tailored towards a > very limited use case. The implementation does not support any of the > useful advanced features like keyed hashing (MAC), salt, > personalization, tree hashing and variable hash length. I bet that the use cases they put into OpenSSL is what most people will eventually use, so essentially the same reasoning we use for putting stuff into the stdlib. Besides, the code just landed in OpenSSL. It's likely they'll continue to optimize it and possibly also add the variants they left out initially. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 29 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From jcronin at egh.com Sun May 29 12:57:01 2016 From: jcronin at egh.com (jon) Date: Sun, 29 May 2016 12:57:01 -0400 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: <20160528232423.GA78519@cskk.homeip.net> References: <20160528232423.GA78519@cskk.homeip.net> Message-ID: <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com> The aspects we want to capture in a name or adjective for these types are: a) The types have identical implementations or definitions. b) They are distinct types. I think ?Distinguished Type? or?Cloned Type? best captures these qualities. I think the following also capture the quality, but feel a bit pejorative. "Segregated Type" "Arbitrary Type" Jonathan From tjreedy at udel.edu Sun May 29 13:54:39 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 29 May 2016 13:54:39 -0400 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com> References: <20160528232423.GA78519@cskk.homeip.net> <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com> Message-ID: On 5/29/2016 12:57 PM, jon wrote: > > The aspects we want to capture in a name or adjective for these types are: > > a) The types have identical implementations or definitions. > b) They are distinct types. > > I think ?Distinguished Type? or?Cloned Type? best captures these qualities. 'Cloned Type' conveys the meaning above, "Distinguished Type' does not, unless one knows what it us supposed to mean. One meaning is 'distinct because superior (or inferior)'. > I think the following also capture the quality, but feel a bit pejorative. > > "Segregated Type" > "Arbitrary Type" Things are segregated because they are thought to be different. 'Arbitrary' means that it could be anything ;-). -- Terry Jan Reedy From jelle.zijlstra at gmail.com Sun May 29 15:07:57 2016 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Sun, 29 May 2016 12:07:57 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: 2016-05-27 16:01 GMT-07:00 Guido van Rossum : > Also -- the most important thing. :-) What to call these things? We're > pretty much settled on the semantics and how to create them (A = > NewType('A', int)) but what should we call types like A when we're > talking about them? "New types" sounds awkward. > For what it's worth, Haskell uses the term "newtype" for a very similar concept (https://wiki.haskell.org/Newtype), so maybe Python should follow suit in the interest of not creating new confusing terminology. "Dependent type" (proposed by a few people) already means something else ( https://en.wikipedia.org/wiki/Dependent_type), so it doesn't seem like a good choice here. > > On Fri, May 27, 2016 at 12:54 PM, Guido van Rossum > wrote: > > Sorry, this is more meant to be the start of a discussion about the > proposed > > feature. And typing.py has its own upstream repo (like asyncio). > > > > --Guido (mobile) > > > > On May 27, 2016 12:52 PM, "Brett Cannon" wrote: > >> > >> Patches to Python's stdlib should go through bugs.python.org so it > isn't > >> lost in email. > >> > >> > >> On Fri, May 27, 2016, 12:00 Ivan Levkivskyi > wrote: > >>> > >>> Hi, > >>> > >>> It has been proposed to enhance the typing module with a NewType > function > >>> that allows to define simple unique types with almost zero runtime > >>> overhead. > >>> > >>> The PR containing actual implementation and PEP 484 update is here: > >>> https://github.com/python/typing/pull/226 > >>> > >>> Review comments are very welcome. > >>> > >>> Best regards, > >>> Ivan > >>> _______________________________________________ > >>> Python-Dev mailing list > >>> Python-Dev at python.org > >>> https://mail.python.org/mailman/listinfo/python-dev > >>> Unsubscribe: > >>> https://mail.python.org/mailman/options/python-dev/brett%40python.org > >> > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > >> > > > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jelle.zijlstra%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Sun May 29 16:21:02 2016 From: christian at python.org (Christian Heimes) Date: Sun, 29 May 2016 13:21:02 -0700 Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512 In-Reply-To: References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org> <57482779.6030100@egenix.com> <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io> <5748C566.1070904@egenix.com> Message-ID: <44005240-8c05-f012-bb87-79d1d0fc18cf@python.org> On 2016-05-28 23:51, Victor Stinner wrote: > Python 3.5 requires a 64 bit signed integer to build. Search for _PyTime > type in pytime.h ;-) Awesome! Thanks :) From oscar.j.benjamin at gmail.com Sun May 29 19:08:42 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 30 May 2016 00:08:42 +0100 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: On 28 May 2016 00:03, "Guido van Rossum" wrote: > > Also -- the most important thing. :-) What to call these things? We're > pretty much settled on the semantics and how to create them (A = > NewType('A', int)) but what should we call types like A when we're > talking about them? "New types" sounds awkward. How about "nominal type" since the type is distinct only in name? -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Sun May 29 19:53:02 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun, 29 May 2016 16:53:02 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: On Sun, May 29, 2016 at 12:07 PM, Jelle Zijlstra wrote: > > > 2016-05-27 16:01 GMT-07:00 Guido van Rossum : >> >> Also -- the most important thing. :-) What to call these things? We're >> pretty much settled on the semantics and how to create them (A = >> NewType('A', int)) but what should we call types like A when we're >> talking about them? "New types" sounds awkward. > > > For what it's worth, Haskell uses the term "newtype" for a very similar > concept (https://wiki.haskell.org/Newtype), so maybe Python should follow > suit in the interest of not creating new confusing terminology. Yeah, I started out rejecting this because I don't like such neologisms. > "Dependent type" (proposed by a few people) already means something else > (https://en.wikipedia.org/wiki/Dependent_type), so it doesn't seem like a > good choice here. I apologize, when I brought that up I meant to say "derived type" (as should have been clear from the link I provided (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types). I agree dependent types would be a terrible name here. :-) I am currently in favor of Distinct Type [Alias]. -- --Guido van Rossum (python.org/~guido) From gvanrossum at gmail.com Sun May 29 19:55:50 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun, 29 May 2016 16:55:50 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: On Sun, May 29, 2016 at 4:08 PM, Oscar Benjamin wrote: > > On 28 May 2016 00:03, "Guido van Rossum" wrote: >> >> Also -- the most important thing. :-) What to call these things? We're >> pretty much settled on the semantics and how to create them (A = >> NewType('A', int)) but what should we call types like A when we're >> talking about them? "New types" sounds awkward. > > How about "nominal type" since the type is distinct only in name? I believe that would be confusing because nominal type systems are a different idea (https://en.wikipedia.org/wiki/Nominal_type_system). -- --Guido van Rossum (python.org/~guido) From fred at fdrake.net Sun May 29 20:12:04 2016 From: fred at fdrake.net (Fred Drake) Date: Sun, 29 May 2016 17:12:04 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: On Sun, May 29, 2016 at 4:53 PM, Guido van Rossum wrote: > I am currently in favor of Distinct Type [Alias]. I actually like distinguished type better: A = typing.distinguish("A", int) -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From steve at pearwood.info Sun May 29 21:53:59 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 30 May 2016 11:53:59 +1000 Subject: [Python-Dev] Python parser performance optimizations In-Reply-To: References: Message-ID: <20160530015359.GL12028@ando.pearwood.info> On Thu, May 26, 2016 at 10:19:05AM +0000, Artyom Skrobov wrote: [...] > The motivation for this patch was to enable a memory footprint > optimization, discussed at http://bugs.python.org/issue26415 My > proposed optimization reduces the memory footprint by up to 30% on the > standard benchmarks, and by 200% on a degenerate case which sparked > the discussion. The run time stays unaffected by this optimization. That can't be right. How can you reduce memory usage by more than one hundred percent? That would mean you have saved more memory than was originally used and are now using a negative amount of memory. -- Steve From mafagafogigante at gmail.com Mon May 30 05:50:46 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Mon, 30 May 2016 06:50:46 -0300 Subject: [Python-Dev] Python parser performance optimizations In-Reply-To: <20160530015359.GL12028@ando.pearwood.info> References: <20160530015359.GL12028@ando.pearwood.info> Message-ID: <29194dc2-496c-6a8b-1fc7-47637ef15258@gmail.com> On 05/29/2016 10:53 PM, Steven D'Aprano wrote: > On Thu, May 26, 2016 at 10:19:05AM +0000, Artyom Skrobov wrote: > [...] >> The motivation for this patch was to enable a memory footprint >> optimization, discussed at http://bugs.python.org/issue26415 My >> proposed optimization reduces the memory footprint by up to 30% on the >> standard benchmarks, and by 200% on a degenerate case which sparked >> the discussion. The run time stays unaffected by this optimization. > > That can't be right. How can you reduce memory usage by more than one > hundred percent? That would mean you have saved more memory than was > originally used and are now using a negative amount of memory. > It is not. It would be nice to have the values that were used to calculate these percentages. From greg.ewing at canterbury.ac.nz Mon May 30 08:45:46 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 31 May 2016 00:45:46 +1200 Subject: [Python-Dev] Python parser performance optimizations In-Reply-To: <20160530015359.GL12028@ando.pearwood.info> References: <20160530015359.GL12028@ando.pearwood.info> Message-ID: <574C35FA.9060103@canterbury.ac.nz> Steven D'Aprano wrote: > That can't be right. How can you reduce memory usage by more than one > hundred percent? That would mean you have saved more memory than was > originally used and are now using a negative amount of memory. It emails an order for more RAM to Amazon, who send out a robot drone to install it in your computer. -- Greg From v+python at g.nevcal.com Mon May 30 10:15:02 2016 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 30 May 2016 07:15:02 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com> References: <20160528232423.GA78519@cskk.homeip.net> <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com> Message-ID: On 5/29/2016 9:57 AM, jon wrote: > The aspects we want to capture in a name or adjective for these types are: > > a) The types have identical implementations or definitions. > b) They are distinct types. > > I think ?Distinguished Type? or?Cloned Type? best captures these qualities. > > I think the following also capture the quality, but feel a bit pejorative. > > "Segregated Type" > "Arbitrary Type" "Exclusive Type" "Separate Type" Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Mon May 30 12:31:29 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon, 30 May 2016 09:31:29 -0700 Subject: [Python-Dev] Python parser performance optimizations In-Reply-To: <574C35FA.9060103@canterbury.ac.nz> References: <20160530015359.GL12028@ando.pearwood.info> <574C35FA.9060103@canterbury.ac.nz> Message-ID: I know we're all just having fun, but that's probably a rather stressful welcome to the list. Maybe we can tone down the humor a bit and instead review the OP's patches? --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 30 17:59:29 2016 From: brett at python.org (Brett Cannon) Date: Mon, 30 May 2016 21:59:29 +0000 Subject: [Python-Dev] embedding - PyImport_AppendInittab() after Py_Initialize() In-Reply-To: References: Message-ID: On Fri, 27 May 2016 at 07:50 Vitaly Murashev wrote: > Could anyone please clarify whether it is correct in python3.5 to call > PyImport_AppendInittab() after Py_Initialize() > It should be called before as PyImport_AppendInittab() is shorthand for PyImport_ExtendInittab() and it says to call prior to Py_Initialize(): https://github.com/python/cpython/blob/f680b517e2701e9a3859afb62628a46eccdce17c/Python/import.c#L2153 . -Brett > > I found this case among the tests for boost.python > https://github.com/boostorg/python/blob/develop/test/exec.cpp > > And this test doesn't work for me with python3.5.An error looks like: > > exec.cpp(137): Python Error detected in function 'void __cdecl > check_pyerr(bool)' > Traceback (most recent call last): > File "", line 1, in > ImportError: 'embedded_hello' is not a built-in module > 1 error detected. > > After debugging I found out that root cause is in > importlib/_bootstrap.py which do the following: > > if fullname not in sys.builtin_module_names: > raise ImportError('{!r} is not a built-in > module'.format(fullname), name=fullname) > > but sys.builtin_module_names is the constant which initialized once in > Py_Initialize() > > --- > To summarize: > Is it a bug in python3.5 or original test inside boost.python should be > fixed ? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rndblnch at gmail.com Tue May 31 15:55:53 2016 From: rndblnch at gmail.com (rndblnch) Date: Tue, 31 May 2016 19:55:53 +0000 (UTC) Subject: [Python-Dev] Adding NewType() to PEP 484 References: Message-ID: Guido van Rossum gmail.com> writes: > > Also -- the most important thing. What to call these things? We're > pretty much settled on the semantics and how to create them (A = > NewType('A', int)) but what should we call types like A when we're > talking about them? "New types" sounds awkward. back in high school, i was introduced to c programming with the "disciplined C" preprocessor [0]. it made the distinction between information type and representation type (e.g. between the semantic and the implementation). those new types where created using typedefs and were named 'parallel types' below is the relevant part of the dcc presentation: """ a major innovation of Disciplined C is the notion of "parallel type", that allows a distinction between information type and representation type. The following: typedef int Tindex, Tval; typedef Tindex Trow, Tcol; creates four distinct types, but which all accept the same operations and the same constants as the "representation" type ('int' here). Tindex, Tval, Trow and Tcol are examples of "information" types, because they convey an idea of the semantics of the corresponding objects. For example, they may be put to use in a checkers playing program: Tval will name 'int's that represent values of checkers, Trow and Tcol, 'int's that represent row and column indexes, Tindex, generic type for indexes. Tindex, Tval, Trow and Tcol are called parallel types; in fact, a type T1 is said to be parallel to a type T2 iff both are defined through a chain of typedefs starting from the same 'baseType', with no intervening qualifier nor modifier (pointer/array/function decla- rator, see grammar in Appendix A). In other words, T and T2 must be strict synonyms of baseType. """ renaud 0. Disciplined C ACM SIGPLAN Notices Homepage archive Volume 30 Issue 12, Dec. 1995 Pages 43 - 50 http://dl.acm.org/citation.cfm?id=219726.219747 http://www.digiater.nl/openvms/freeware/v50/dcc/dcc-v2_7d/dccarticle.ps From v+python at g.nevcal.com Tue May 31 18:02:48 2016 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 31 May 2016 15:02:48 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: Message-ID: <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com> On 5/31/2016 12:55 PM, rndblnch wrote: > Guido van Rossum gmail.com> writes: > >> Also -- the most important thing. What to call these things? We're >> pretty much settled on the semantics and how to create them (A = >> NewType('A', int)) but what should we call types like A when we're >> talking about them? "New types" sounds awkward. > back in high school, i was introduced to c programming with the "disciplined > C" preprocessor [0]. > it made the distinction between information type and representation type > (e.g. between the semantic and the implementation). > those new types where created using typedefs and were named 'parallel types' > below is the relevant part of the dcc presentation: Interesting! Prior art. And parallel type isn't a bad name... > > """ > a major innovation of Disciplined C is the notion of "parallel type", that > allows a distinction between information type and representation type. The > following: > > typedef int Tindex, Tval; > typedef Tindex Trow, Tcol; > > creates four distinct types, but which all accept the same operations and > the same constants as the "representation" type ('int' here). Tindex, Tval, > Trow and Tcol are examples of "information" types, because they convey an > idea of the semantics of the corresponding objects. For example, they may be > put to use in a checkers playing program: Tval will name 'int's that > represent values of checkers, Trow and Tcol, 'int's that represent row and > column indexes, Tindex, generic type for indexes. > Tindex, Tval, Trow and Tcol are called parallel types; in fact, a type T1 is > said to be parallel to a type T2 iff both are defined through a chain of > typedefs starting from the same 'baseType', with no intervening qualifier > nor modifier (pointer/array/function decla- rator, see grammar in Appendix > A). In other words, T and T2 must be strict synonyms of baseType. > """ > > renaud > > 0. Disciplined C > > ACM SIGPLAN Notices Homepage archive > Volume 30 Issue 12, Dec. 1995 > Pages 43 - 50 > http://dl.acm.org/citation.cfm?id=219726.219747 > http://www.digiater.nl/openvms/freeware/v50/dcc/dcc-v2_7d/dccarticle.ps > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue May 31 19:58:50 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Jun 2016 09:58:50 +1000 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com> References: <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com> Message-ID: On 31 May 2016 3:12 pm, "Glenn Linderman" wrote: > > On 5/31/2016 12:55 PM, rndblnch wrote: >> >> Guido van Rossum gmail.com> writes: >> >>> >>> Also -- the most important thing. What to call these things? We're >>> pretty much settled on the semantics and how to create them (A = >>> NewType('A', int)) but what should we call types like A when we're >>> talking about them? "New types" sounds awkward. >> >> back in high school, i was introduced to c programming with the "disciplined >> C" preprocessor [0]. >> it made the distinction between information type and representation type >> (e.g. between the semantic and the implementation). >> those new types where created using typedefs and were named 'parallel types' >> below is the relevant part of the dcc presentation: > > > Interesting! Prior art. And parallel type isn't a bad name... If I heard "parallel type", I'd assume it had something to do with parallel processing. Of the options suggested so far, DistinctType seems the most promising to me. Cheers, Nick. > >> >> >> """ >> a major innovation of Disciplined C is the notion of "parallel type", that >> allows a distinction between information type and representation type. The >> following: >> >> typedef int Tindex, Tval; >> typedef Tindex Trow, Tcol; >> >> creates four distinct types, but which all accept the same operations and >> the same constants as the "representation" type ('int' here). Tindex, Tval, >> Trow and Tcol are examples of "information" types, because they convey an >> idea of the semantics of the corresponding objects. For example, they may be >> put to use in a checkers playing program: Tval will name 'int's that >> represent values of checkers, Trow and Tcol, 'int's that represent row and >> column indexes, Tindex, generic type for indexes. >> Tindex, Tval, Trow and Tcol are called parallel types; in fact, a type T1 is >> said to be parallel to a type T2 iff both are defined through a chain of >> typedefs starting from the same 'baseType', with no intervening qualifier >> nor modifier (pointer/array/function decla- rator, see grammar in Appendix >> A). In other words, T and T2 must be strict synonyms of baseType. >> """ >> >> renaud >> >> 0. Disciplined C >> >> ACM SIGPLAN Notices Homepage archive >> Volume 30 Issue 12, Dec. 1995 >> Pages 43 - 50 >> http://dl.acm.org/citation.cfm?id=219726.219747 >> http://www.digiater.nl/openvms/freeware/v50/dcc/dcc-v2_7d/dccarticle.ps >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Tue May 31 20:46:38 2016 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 31 May 2016 17:46:38 -0700 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com> Message-ID: On 5/31/2016 4:58 PM, Nick Coghlan wrote: > > > On 31 May 2016 3:12 pm, "Glenn Linderman" > wrote: > > > > On 5/31/2016 12:55 PM, rndblnch wrote: > >> > >> Guido van Rossum gmail.com > writes: > >> > >>> > >>> Also -- the most important thing. What to call these things? We're > >>> pretty much settled on the semantics and how to create them (A = > >>> NewType('A', int)) but what should we call types like A when we're > >>> talking about them? "New types" sounds awkward. > >> > >> back in high school, i was introduced to c programming with the > "disciplined > >> C" preprocessor [0]. > >> it made the distinction between information type and representation > type > >> (e.g. between the semantic and the implementation). > >> those new types where created using typedefs and were named > 'parallel types' > >> below is the relevant part of the dcc presentation: > > > > > > Interesting! Prior art. And parallel type isn't a bad name... > > If I heard "parallel type", I'd assume it had something to do with > parallel processing. > Google seems to think so also... "Disciplined C" probably predates parallel processing, but adding disciplined C preprocessor to the search does find a document about the preprocessor in the same sense used by @rndblnch. And the types do have "parallel" behavior... every behavior of one is found identically in the other. So it makes sense, but it would be hard to find the prior art without extra information, as the term is now overwhelmed by "Parallel Type-checking" for parallel processing. Still, some mention should probably be made of "parallel type in the 'Disciplined C preprocessor' " in the "prior art" category of PEP 484. From reading about Disciplined C, would seem that this feature could be used as a foundation to add "units" support to variables, validated only by a type checker, and thus having no runtime overhead, although it would take some more features from Disciplined C, regarding the definition of legal combinations of types and operations. > Of the options suggested so far, DistinctType seems the most promising > to me. > > Cheers, > Nick. > > > > >> > >> > >> """ > >> a major innovation of Disciplined C is the notion of "parallel > type", that > >> allows a distinction between information type and representation > type. The > >> following: > >> > >> typedef int Tindex, Tval; > >> typedef Tindex Trow, Tcol; > >> > >> creates four distinct types, but which all accept the same > operations and > >> the same constants as the "representation" type ('int' here). > Tindex, Tval, > >> Trow and Tcol are examples of "information" types, because they > convey an > >> idea of the semantics of the corresponding objects. For example, > they may be > >> put to use in a checkers playing program: Tval will name 'int's that > >> represent values of checkers, Trow and Tcol, 'int's that represent > row and > >> column indexes, Tindex, generic type for indexes. > >> Tindex, Tval, Trow and Tcol are called parallel types; in fact, a > type T1 is > >> said to be parallel to a type T2 iff both are defined through a > chain of > >> typedefs starting from the same 'baseType', with no intervening > qualifier > >> nor modifier (pointer/array/function decla- rator, see grammar in > Appendix > >> A). In other words, T and T2 must be strict synonyms of baseType. > >> """ > >> > >> renaud > >> > >> 0. Disciplined C > >> > >> ACM SIGPLAN Notices Homepage archive > >> Volume 30 Issue 12, Dec. 1995 > >> Pages 43 - 50 > >> http://dl.acm.org/citation.cfm?id=219726.219747 > >> http://www.digiater.nl/openvms/freeware/v50/dcc/dcc-v2_7d/dccarticle.ps > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com > >> > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mafagafogigante at gmail.com Tue May 31 21:32:48 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Tue, 31 May 2016 22:32:48 -0300 Subject: [Python-Dev] Adding NewType() to PEP 484 In-Reply-To: References: <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com> Message-ID: <614d2422-ce06-9b0c-672d-3102bbd90ddc@gmail.com> On 05/31/2016 08:58 PM, Nick Coghlan wrote: > On 31 May 2016 3:12 pm, "Glenn Linderman" wrote: >> >> On 5/31/2016 12:55 PM, rndblnch wrote: >>> >>> Guido van Rossum gmail.com> writes: >>> >>>> >>>> Also -- the most important thing. What to call these things? We're >>>> pretty much settled on the semantics and how to create them (A = >>>> NewType('A', int)) but what should we call types like A when we're >>>> talking about them? "New types" sounds awkward. >>> >>> back in high school, i was introduced to c programming with the > "disciplined >>> C" preprocessor [0]. >>> it made the distinction between information type and representation type >>> (e.g. between the semantic and the implementation). >>> those new types where created using typedefs and were named 'parallel > types' >>> below is the relevant part of the dcc presentation: >> >> >> Interesting! Prior art. And parallel type isn't a bad name... > > If I heard "parallel type", I'd assume it had something to do with parallel > processing. > This also happens to me every time I read it.