From ncoghlan at gmail.com Sun Feb 1 08:46:09 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Feb 2015 17:46:09 +1000 Subject: [Python-Dev] Workflow improvement PEPs 474 & 462 updated Message-ID: As per Brett's target timeline, I've just posted updated versions of my workflow improvement proposals in PEP 474 and 462 PEP 474 is my proposal to host a Kallithea instance at forge.python.org: https://www.python.org/dev/peps/pep-0474/ This update was fairly minor, mainly just added some timeline details based on Brett's proposed schedule for discussions and implementation: https://www.python.org/dev/peps/pep-0474/#pilot-objectives-and-timeline The updates to PEP 462, which covers proposed changes to the main CPython workflow, were more significant, as I've now rewritten that to depend on PEP 474, and propose replacing the current Rietveld patch review workflow with an updated approach based on Kallithea and Zuul: https://www.python.org/dev/peps/pep-0462/ Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From deisum at gmail.com Sun Feb 1 11:13:42 2015 From: deisum at gmail.com (Benjamin) Date: Sun, 1 Feb 2015 04:13:42 -0600 Subject: [Python-Dev] PEP 484 syntax: NONONONONONONO! Message-ID: The proposed syntax is abominable. It's the opposite of readable. The function annotation syntax is ugly, but potentially useful for things like documentation. While it may very well have been created with the idea of type-checking, actually using it for such quickly turns into an unreadable morass of information that is far more difficult for human brains to parse, which makes this usage the antithesis of pythonic. I much prefer the idea of a 'where' keyword to denote typing, as discussed http://aroberge.blogspot.com/2015/01/type-hinting-in-python-focus-on.html, but I think a refinement of their idea would prove even better: def retry(callback, timeout, retries=None) where ........callback is Callable[AnyArgs, Any[int, None]], ........timeout is Any[int, None], ........retries is in [int, None], # 'is in' construct is more readable, dunno about complexity ........return is None: ....pass def greeting(name) where name is str, return is str: ....return 'Hello ' + name x, y = [], [] where x, y is List[Employee], List[Manager] To me, this orders of magnitude more readable than the proposed nonsense. PS. Obviously the 8-space indent above would only a convention, not requirement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Sun Feb 1 16:59:22 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sun, 01 Feb 2015 15:59:22 +0000 Subject: [Python-Dev] PEP 484 syntax: NONONONONONONO! In-Reply-To: References: Message-ID: On 01/02/2015 10:13, Benjamin wrote: > The proposed syntax is abominable. It's the opposite of readable. > I have no views on the subject as I won't be using it, but there is no need to shout to get your point across. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From francismb at email.de Sun Feb 1 19:53:46 2015 From: francismb at email.de (francis) Date: Sun, 01 Feb 2015 19:53:46 +0100 Subject: [Python-Dev] Workflow improvement PEPs 474 & 462 updated In-Reply-To: References: Message-ID: <54CE763A.4020301@email.de> Hi Nick, On 02/01/2015 08:46 AM, Nick Coghlan wrote: [...] > The updates to PEP 462, which covers proposed changes to the main > CPython workflow, were more significant, as I've now rewritten that to > depend on PEP 474, and propose replacing the current Rietveld patch > review workflow with an updated approach based on Kallithea and Zuul: > https://www.python.org/dev/peps/pep-0462/ > A small question on: """ Push races would also be a thing of the past - if lots of core developers are approving patches at a sprint, then that just means the queue gets deeper in Zuul, rather than developers getting frustrated trying to merge changes and failing. """ How does the Tool Zuul resolve the case where the patches are not fully compatible. E.g. they touch the same file and some manually merging is needed? (Isn't that a push race? or I'm missing something?) And one minor detail. Early on "Current Tools": """ This proposal suggests replacing the use of Rietveld for code review with the more full-featured Kallithea-based forge.python.org service proposed in PEP 474 . """ and then later on "Perceived Benefits": """ the merge queue would allow that developer to focus more of their time on reviewing patches and helping the other contributors at the sprint, since accepting a patch for inclusion would now be a single click in the Rietveld UI, rather [...] """ Isn't "Kallithea UI" meant here? And +1 for self-hosting on: """ This proposal respects that history by recommending only tools that are available for self-hosting as sponsored or PSF funded infrastructure, and are also open source Python projects that can be customised to meet the needs of the CPython core development team """ I like the PEP. PS: Should this be forwarded to python-workflow or is that other list to be considered obsolete? Regards, francis From rosuav at gmail.com Sun Feb 1 21:54:51 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 Feb 2015 07:54:51 +1100 Subject: [Python-Dev] PEP 484 syntax: NONONONONONONO! In-Reply-To: References: Message-ID: On Sun, Feb 1, 2015 at 9:13 PM, Benjamin wrote: > I much prefer the idea of a 'where' keyword to denote typing, as discussed > http://aroberge.blogspot.com/2015/01/type-hinting-in-python-focus-on.html, > but I think a refinement of their idea would prove even better: > > def retry(callback, timeout, retries=None) where > ........callback is Callable[AnyArgs, Any[int, None]], > ........timeout is Any[int, None], > ........retries is in [int, None], # 'is in' construct is more readable, > dunno about complexity > ........return is None: > ....pass Massively verbose, and requires duplication of your argument names. If you're going to have the args all listed down below, the first line is redundant; all you need to do is incorporate the "=None" default into the lower part, and you can drop the first line altogether. def retry( callback: Callable[AnyArgs, Optional[int]], timeout: Optional[int], retries:Optional[int]=None ) -> None: pass Indent that any way you like. Apart from taking advantage of Optional[] rather than explicitly Anying with None, that's a direct translation from your code to actual annotations. Advantage: It works in current code, as long as you backport those names (which typing.py will do). I'm stating the parameter names exactly once each, I'm laying it out pretty much the same way you had it, and there's no need to add any more syntax beyond PEP 3107. ChrisA From ncoghlan at gmail.com Mon Feb 2 01:11:06 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Feb 2015 10:11:06 +1000 Subject: [Python-Dev] Workflow improvement PEPs 474 & 462 updated In-Reply-To: <54CE763A.4020301@email.de> References: <54CE763A.4020301@email.de> Message-ID: On 2 Feb 2015 04:56, "francis" wrote: > > Hi Nick, > > On 02/01/2015 08:46 AM, Nick Coghlan wrote: > [...] > > The updates to PEP 462, which covers proposed changes to the main > > CPython workflow, were more significant, as I've now rewritten that to > > depend on PEP 474, and propose replacing the current Rietveld patch > > review workflow with an updated approach based on Kallithea and Zuul: > > https://www.python.org/dev/peps/pep-0462/ > > > > > A small question on: > """ > Push races would also be a thing of the past - if lots of core > developers are approving patches at a sprint, then that just means the > queue gets deeper in Zuul, rather than developers getting frustrated > trying to merge changes and failing. > """ > How does the Tool Zuul resolve the case where the patches are not fully > compatible. E.g. they touch the same file and some manually merging is > needed? (Isn't that a push race? or I'm missing something?) That's an actual merge conflict, the push races I'm referring to are the ones where there's no conflict, but Mercurial objects to the non-linear history. Zuul takes care of linearising everything and making sure the tests pass before committing the change to the relevant branch. > And one minor detail. Early on "Current Tools": > """ > This proposal suggests replacing the use of Rietveld for code review > with the more full-featured Kallithea-based forge.python.org service > proposed in PEP 474 . > """ > > and then later on "Perceived Benefits": > """ > the merge queue would allow that developer to focus more of their time > on reviewing patches and helping the other contributors at the sprint, > since accepting a patch for inclusion would now be a single click in the > Rietveld UI, rather [...] > """ > Isn't "Kallithea UI" meant here? Good catch - I must have missed that one in the update. > And +1 for self-hosting on: > """ > This proposal respects that history by recommending only tools that are > available for self-hosting as sponsored or PSF funded infrastructure, > and are also open source Python projects that can be customised to meet > the needs of the CPython core development team > """ > > I like the PEP. > > > PS: Should this be forwarded to python-workflow or is that other list to > be considered obsolete? I withdrew from participating in that list when an individual banned from the core mailing lists and the issue tracker for persistently failing to respect other participants in those communities chose to participate in it despite an explicit request from me that he refrain from doing so (after wasting years trying to coach him into more productive modes of engagement, I now just cut my losses and flat out refuse to work in any environment where he has a significant presence). Since our moderation practices don't currently include a way to request transferring bans between lists, and I'm reluctant to push for that to change when I have such a clear personal stake in the outcome (it reads like a personal vendetta against the individual concerned), that's the way things are likely to stay unless/until he also gets himself banned from the core workflow list. Regards, Nick. > > Regards, > francis > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Mon Feb 2 15:35:47 2015 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 02 Feb 2015 14:35:47 +0000 Subject: [Python-Dev] Workflow PEP proposals are now closed Message-ID: The PEPs under consideration are PEPs 474 and 462 from Nick Coghlan to use Kallithea and do self-hosting, and PEP 481 from Donald Stufft that proposes using GitHub. At this point I expect final PEPs by PyCon US so I can try and make a decision by May 1. Longer still is to hopefully have whatever solution we choose in place right after Python 3.5 is released. And just a reminder to people, the lofty goal is to improve the overall workflow for CPython itself such that our patch submission queue can actually be cleared regularly. This not only benefits core devs by letting us be more effective, but also contributors by making sure their hard work gets addressed quickly and thus doesn't languish on the issue tracker for very long. If we can't find a solution for fixing our CPython workflow I will then be willing to entertain these PEPs narrowing their scopes and only focus on ancillary repos like the devguide, etc. where the workflows are simple. I know the absolute worst case is nothing changes, but honestly I think the worst case is Nick's work gets us off of Rietveld, the ancillary repos move to GitHub, and we make the GitHub and Bitbucket mirrors of CPython official ones for people to work from (bonus points if we get the issue tracker to have push button patch pulling from GitHub; Bitbucket is already covered thanks to our remote hg repo support). IOW I see nothing but a win for contributors and core devs as well as everyone proposing solutions which is a nice place to start from. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre-yves.david at ens-lyon.org Mon Feb 2 09:07:30 2015 From: pierre-yves.david at ens-lyon.org (Pierre-Yves David) Date: Mon, 02 Feb 2015 09:07:30 +0100 Subject: [Python-Dev] Workflow improvement PEPs 474 & 462 updated In-Reply-To: References: <54CE763A.4020301@email.de> Message-ID: <54CF3042.70907@ens-lyon.org> On 02/02/2015 01:11 AM, Nick Coghlan wrote: > > On 2 Feb 2015 04:56, "francis" > wrote: > > > > Hi Nick, > > > > On 02/01/2015 08:46 AM, Nick Coghlan wrote: > > [...] > > > The updates to PEP 462, which covers proposed changes to the main > > > CPython workflow, were more significant, as I've now rewritten that to > > > depend on PEP 474, and propose replacing the current Rietveld patch > > > review workflow with an updated approach based on Kallithea and Zuul: > > > https://www.python.org/dev/peps/pep-0462/ > > > > > > > > > A small question on: > > """ > > Push races would also be a thing of the past - if lots of core > > developers are approving patches at a sprint, then that just means the > > queue gets deeper in Zuul, rather than developers getting frustrated > > trying to merge changes and failing. > > """ > > How does the Tool Zuul resolve the case where the patches are not fully > > compatible. E.g. they touch the same file and some manually merging is > > needed? (Isn't that a push race? or I'm missing something?) > > That's an actual merge conflict, the push races I'm referring to are the > ones where there's no conflict, but Mercurial objects to the non-linear > history. Zuul takes care of linearising everything and making sure the > tests pass before committing the change to the relevant branch. On this topic, Facebook recently open-sourced a mercurial extension doing server side rebasing to linearise history in simple cases. special flag during push will get your new changes seamlessly rebased on the new head. This however does not run the tests so the Zuul approach is more complete. > > PS: Should this be forwarded to python-workflow or is that other list to > > be considered obsolete? > > I withdrew from participating in that list when an individual banned > from the core mailing lists and the issue tracker for persistently > failing to respect other participants in those communities chose to > participate in it despite an explicit request from me that he refrain > from doing so (after wasting years trying to coach him into more > productive modes of engagement, I now just cut my losses and flat out > refuse to work in any environment where he has a significant presence). > > Since our moderation practices don't currently include a way to request > transferring bans between lists, and I'm reluctant to push for that to > change when I have such a clear personal stake in the outcome (it reads > like a personal vendetta against the individual concerned), that's the > way things are likely to stay unless/until he also gets himself banned > from the core workflow list. Without emitting any judgment on your decision, I'm deeply sad that this list have been "abandoned". -- Pierre-Yves David From brett at python.org Mon Feb 2 15:55:32 2015 From: brett at python.org (Brett Cannon) Date: Mon, 02 Feb 2015 14:55:32 +0000 Subject: [Python-Dev] Workflow improvement PEPs 474 & 462 updated References: <54CE763A.4020301@email.de> <54CF3042.70907@ens-lyon.org> Message-ID: On Mon Feb 02 2015 at 9:52:29 AM Pierre-Yves David < pierre-yves.david at ens-lyon.org> wrote: > > > On 02/02/2015 01:11 AM, Nick Coghlan wrote: > > > > On 2 Feb 2015 04:56, "francis" > > wrote: > [SNIP] > > > PS: Should this be forwarded to python-workflow or is that other list > to > > > be considered obsolete? > > > > I withdrew from participating in that list when an individual banned > > from the core mailing lists and the issue tracker for persistently > > failing to respect other participants in those communities chose to > > participate in it despite an explicit request from me that he refrain > > from doing so (after wasting years trying to coach him into more > > productive modes of engagement, I now just cut my losses and flat out > > refuse to work in any environment where he has a significant presence). > > > > Since our moderation practices don't currently include a way to request > > transferring bans between lists, and I'm reluctant to push for that to > > change when I have such a clear personal stake in the outcome (it reads > > like a personal vendetta against the individual concerned), that's the > > way things are likely to stay unless/until he also gets himself banned > > from the core workflow list. > > Without emitting any judgment on your decision, I'm deeply sad that this > list have been "abandoned". > Others do participate there so the list is not dead or abandoned, simply that Nick is not participating on that list. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Feb 2 16:00:26 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 2 Feb 2015 10:00:26 -0500 Subject: [Python-Dev] Workflow PEP proposals are now closed In-Reply-To: References: Message-ID: <1514D26E-392E-4417-86F1-38D9E747DE46@stufft.io> > On Feb 2, 2015, at 9:35 AM, Brett Cannon wrote: > > The PEPs under consideration are PEPs 474 and 462 from Nick Coghlan to use Kallithea and do self-hosting, and PEP 481 from Donald Stufft that proposes using GitHub. > > At this point I expect final PEPs by PyCon US so I can try and make a decision by May 1. Longer still is to hopefully have whatever solution we choose in place right after Python 3.5 is released. > > And just a reminder to people, the lofty goal is to improve the overall workflow for CPython itself such that our patch submission queue can actually be cleared regularly. This not only benefits core devs by letting us be more effective, but also contributors by making sure their hard work gets addressed quickly and thus doesn't languish on the issue tracker for very long. > > If we can't find a solution for fixing our CPython workflow I will then be willing to entertain these PEPs narrowing their scopes and only focus on ancillary repos like the devguide, etc. where the workflows are simple. > > I know the absolute worst case is nothing changes, but honestly I think the worst case is Nick's work gets us off of Rietveld, the ancillary repos move to GitHub, and we make the GitHub and Bitbucket mirrors of CPython official ones for people to work from (bonus points if we get the issue tracker to have push button patch pulling from GitHub; Bitbucket is already covered thanks to our remote hg repo support). IOW I see nothing but a win for contributors and core devs as well as everyone proposing solutions which is a nice place to start from. =) > Is there going to be discussion between the two approaches or should the PEPs themselves address each other? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Mon Feb 2 16:24:30 2015 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 02 Feb 2015 15:24:30 +0000 Subject: [Python-Dev] Workflow PEP proposals are now closed References: <1514D26E-392E-4417-86F1-38D9E747DE46@stufft.io> Message-ID: On Mon Feb 02 2015 at 10:00:30 AM Donald Stufft wrote: > > On Feb 2, 2015, at 9:35 AM, Brett Cannon wrote: > > The PEPs under consideration are PEPs 474 > and 462 > from Nick Coghlan to use > Kallithea and do self-hosting, and PEP 481 > from Donald Stufft that > proposes using GitHub. > > At this point I expect final PEPs by PyCon US so I can try and make a > decision by May 1. Longer still is to hopefully have whatever solution we > choose in place right after Python 3.5 is released. > > And just a reminder to people, the lofty goal is to improve the overall > workflow for CPython itself such that our patch submission queue can > actually be cleared regularly. This not only benefits core devs by letting > us be more effective, but also contributors by making sure their hard work > gets addressed quickly and thus doesn't languish on the issue tracker for > very long. > > If we can't find a solution for fixing our CPython workflow I will then be > willing to entertain these PEPs narrowing their scopes and only focus on > ancillary repos like the devguide, etc. where the workflows are simple. > > I know the absolute worst case is nothing changes, but honestly I think > the worst case is Nick's work gets us off of Rietveld, the ancillary repos > move to GitHub, and we make the GitHub and Bitbucket mirrors of CPython > official ones for people to work from (bonus points if we get the issue > tracker to have push button patch pulling from GitHub; Bitbucket is already > covered thanks to our remote hg repo support). IOW I see nothing but a win > for contributors and core devs as well as everyone proposing solutions > which is a nice place to start from. =) > > > > Is there going to be discussion between the two approaches or should the > PEPs themselves address each other? > Since PEPs are meant to act as a record of what was discussed on a topic then it probably wouldn't hurt to incorporate why your approach is better than the other one in the PEP itself. We can obviously talk openly here when you feel the PEP is ready for it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 2 20:39:41 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Feb 2015 20:39:41 +0100 Subject: [Python-Dev] Workflow PEP proposals are now closed References: Message-ID: <20150202203941.5bc0fdc5@fsol> Hi, What does "closed" mean in this context? Regards Antoine. On Mon, 02 Feb 2015 14:35:47 +0000 Brett Cannon wrote: > The PEPs under consideration are PEPs 474 > and 462 > from Nick Coghlan to use > Kallithea and do self-hosting, and PEP 481 > from Donald Stufft that > proposes using GitHub. > > At this point I expect final PEPs by PyCon US so I can try and make a > decision by May 1. Longer still is to hopefully have whatever solution we > choose in place right after Python 3.5 is released. > > And just a reminder to people, the lofty goal is to improve the overall > workflow for CPython itself such that our patch submission queue can > actually be cleared regularly. This not only benefits core devs by letting > us be more effective, but also contributors by making sure their hard work > gets addressed quickly and thus doesn't languish on the issue tracker for > very long. > > If we can't find a solution for fixing our CPython workflow I will then be > willing to entertain these PEPs narrowing their scopes and only focus on > ancillary repos like the devguide, etc. where the workflows are simple. > > I know the absolute worst case is nothing changes, but honestly I think the > worst case is Nick's work gets us off of Rietveld, the ancillary repos move > to GitHub, and we make the GitHub and Bitbucket mirrors of CPython official > ones for people to work from (bonus points if we get the issue tracker to > have push button patch pulling from GitHub; Bitbucket is already covered > thanks to our remote hg repo support). IOW I see nothing but a win for > contributors and core devs as well as everyone proposing solutions which is > a nice place to start from. =) > From solipsis at pitrou.net Mon Feb 2 20:41:11 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Feb 2015 20:41:11 +0100 Subject: [Python-Dev] cpython (merge 3.4 -> default): Merge 3.4 (asyncio) References: <20150202173756.39270.61855@psf.io> Message-ID: <20150202204111.6eceeffd@fsol> On Mon, 02 Feb 2015 17:38:10 +0000 victor.stinner wrote: > https://hg.python.org/cpython/rev/42b376c8cf60 > changeset: 94465:42b376c8cf60 > parent: 94463:0b3bc51341aa > parent: 94464:2cd6621a9fbc > user: Victor Stinner > date: Mon Feb 02 18:36:59 2015 +0100 > summary: > Merge 3.4 (asyncio) IMO it would be nice to keep the original code in the default branch, since it helps exercise generators. Regards Antoine. From victor.stinner at gmail.com Mon Feb 2 20:53:49 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 2 Feb 2015 20:53:49 +0100 Subject: [Python-Dev] cpython (merge 3.4 -> default): Merge 3.4 (asyncio) In-Reply-To: <20150202204111.6eceeffd@fsol> References: <20150202173756.39270.61855@psf.io> <20150202204111.6eceeffd@fsol> Message-ID: Hi, The current workflow of asyncio is to first commit changes in the external tulip project, then *copy* files to Python 3.4; to finish with a merge from Python 3.4 into Python 3.5. To be complete, I also merge tulip into trollius, but that's unrelated to your question :-) To simplify the workflow, the code is currently exactly the same in tulip, python 3.4 and python 3.5. It implies some tests on the Python version in the code like "if _PY34: ...". I would prefer to stick to this workflow. The code is still heavily modified to fix issues. The behaviour of generators in already tested in test_generators and test_exceptions. Victor 2015-02-02 20:41 GMT+01:00 Antoine Pitrou : > On Mon, 02 Feb 2015 17:38:10 +0000 > victor.stinner wrote: > >> https://hg.python.org/cpython/rev/42b376c8cf60 >> changeset: 94465:42b376c8cf60 >> parent: 94463:0b3bc51341aa >> parent: 94464:2cd6621a9fbc >> user: Victor Stinner >> date: Mon Feb 02 18:36:59 2015 +0100 >> summary: >> Merge 3.4 (asyncio) > > IMO it would be nice to keep the original code in the default branch, > since it helps exercise generators. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From brett at python.org Mon Feb 2 21:04:33 2015 From: brett at python.org (Brett Cannon) Date: Mon, 02 Feb 2015 20:04:33 +0000 Subject: [Python-Dev] Workflow PEP proposals are now closed References: <20150202203941.5bc0fdc5@fsol> Message-ID: On Mon Feb 02 2015 at 2:40:21 PM Antoine Pitrou wrote: > > Hi, > > What does "closed" mean in this context? > No new PEPs on this topic will be taken under consideration, so submissions are now "closed" to new participants. -Brett > > Regards > > Antoine. > > > > On Mon, 02 Feb 2015 14:35:47 +0000 > Brett Cannon wrote: > > The PEPs under consideration are PEPs 474 > > and 462 > > from Nick Coghlan to use > > Kallithea and do self-hosting, and PEP 481 > > from Donald Stufft that > > proposes using GitHub. > > > > At this point I expect final PEPs by PyCon US so I can try and make a > > decision by May 1. Longer still is to hopefully have whatever solution we > > choose in place right after Python 3.5 is released. > > > > And just a reminder to people, the lofty goal is to improve the overall > > workflow for CPython itself such that our patch submission queue can > > actually be cleared regularly. This not only benefits core devs by > letting > > us be more effective, but also contributors by making sure their hard > work > > gets addressed quickly and thus doesn't languish on the issue tracker for > > very long. > > > > If we can't find a solution for fixing our CPython workflow I will then > be > > willing to entertain these PEPs narrowing their scopes and only focus on > > ancillary repos like the devguide, etc. where the workflows are simple. > > > > I know the absolute worst case is nothing changes, but honestly I think > the > > worst case is Nick's work gets us off of Rietveld, the ancillary repos > move > > to GitHub, and we make the GitHub and Bitbucket mirrors of CPython > official > > ones for people to work from (bonus points if we get the issue tracker to > > have push button patch pulling from GitHub; Bitbucket is already covered > > thanks to our remote hg repo support). IOW I see nothing but a win for > > contributors and core devs as well as everyone proposing solutions which > is > > a nice place to start from. =) > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 2 21:45:04 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Feb 2015 21:45:04 +0100 Subject: [Python-Dev] PEP 475 accepted Message-ID: <20150202214504.06613207@fsol> Hello, I'm now accepting PEP 475 - "Retry system calls failing with EINTR". You can read it at https://www.python.org/dev/peps/pep-0475/ The implementation is more or less ready at http://bugs.python.org/issue23285, so you can expect it to land in the main repo relatively soon. Regards Antoine. From guido at python.org Mon Feb 2 21:49:32 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 2 Feb 2015 12:49:32 -0800 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: <20150202214504.06613207@fsol> References: <20150202214504.06613207@fsol> Message-ID: W00t! Congratulations les Fran?ais! On Mon, Feb 2, 2015 at 12:45 PM, Antoine Pitrou wrote: > > Hello, > > I'm now accepting PEP 475 - "Retry system calls failing with EINTR". > You can read it at https://www.python.org/dev/peps/pep-0475/ > > The implementation is more or less ready at > http://bugs.python.org/issue23285, so you can expect it to land in the > main repo relatively soon. > > Regards > > Antoine. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaselton at gmail.com Mon Feb 2 20:36:29 2015 From: chaselton at gmail.com (Cyd Haselton) Date: Mon, 02 Feb 2015 13:36:29 -0600 Subject: [Python-Dev] Import Fails in setup.py On Android Message-ID: After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions I've managed to narrow the problem down to the following line: importlib._bootstrap._SpecMethods(spec).load() Googling this brings up a few hits from bugs.python.org and not much else. I'm new to Python; any ideas what this does...or where I can read up on it for troubleshooting purposes? -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaselton at gmail.com Mon Feb 2 21:04:42 2015 From: chaselton at gmail.com (Cyd Haselton) Date: Mon, 02 Feb 2015 14:04:42 -0600 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: References: Message-ID: <85AF8EB7-ECA0-49E6-8771-B27BF935769D@gmail.com> Update: While waiting for replies I made the change referenced here: https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80 specifically changing importlib. _bootstrap._SpecMethods(spec).load() to importlib._bootstrap.load(spec) I no longer get a terminating 'undefined reference to dlopen' error, but I do get 'importing extensions failed' errors for each extension...like this: *** WARNING: importing extension "_pickle" failed with : 'module' object has no attribute 'load' On February 2, 2015 1:36:29 PM CST, Cyd Haselton wrote: >After fixing a segfault issue (many thanks Ryan) I'm back to the same >issue I was having with Python 2.7.8; the newly built python throws an >undefined reference to dlopen when running setup.py...specifically when >importing just-built extensions > >I've managed to narrow the problem down to the following line: > >importlib._bootstrap._SpecMethods(spec).load() > >Googling this brings up a few hits from bugs.python.org and not much >else. I'm new to Python; any ideas what this does...or where I can >read up on it for troubleshooting purposes? >-- >Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Feb 2 21:58:55 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 2 Feb 2015 21:58:55 +0100 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: References: <20150202214504.06613207@fsol> Message-ID: 2015-02-02 21:49 GMT+01:00 Guido van Rossum : > W00t! Congratulations les Fran?ais! We will celebrate this acceptance with a glass of red wine and cheese. Thanks Antoine! I hate EINTR. It's a pain to handle them for each syscall in subprocess and asyncio (where signals are likely). It's good to not have to handle them explicitly anymore! FYI in Python 3.5 it's now also possible to use signal.set_wakeup_fd() with a socket on Windows. Victor From solipsis at pitrou.net Mon Feb 2 22:00:29 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Feb 2015 22:00:29 +0100 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: References: <20150202214504.06613207@fsol> Message-ID: <20150202220029.6ab85440@fsol> On Mon, 2 Feb 2015 12:49:32 -0800 Guido van Rossum wrote: > W00t! Congratulations les Fran?ais! I hoped nobody would notice :-) Regards Antoine. > > On Mon, Feb 2, 2015 at 12:45 PM, Antoine Pitrou wrote: > > > > > Hello, > > > > I'm now accepting PEP 475 - "Retry system calls failing with EINTR". > > You can read it at https://www.python.org/dev/peps/pep-0475/ > > > > The implementation is more or less ready at > > http://bugs.python.org/issue23285, so you can expect it to land in the > > main repo relatively soon. > > > > Regards > > > > Antoine. > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > From g.rodola at gmail.com Mon Feb 2 22:11:59 2015 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Mon, 2 Feb 2015 22:11:59 +0100 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: <20150202214504.06613207@fsol> References: <20150202214504.06613207@fsol> Message-ID: On Mon, Feb 2, 2015 at 9:45 PM, Antoine Pitrou wrote: > > Hello, > > I'm now accepting PEP 475 - "Retry system calls failing with EINTR". > You can read it at https://www.python.org/dev/peps/pep-0475/ > > The implementation is more or less ready at > http://bugs.python.org/issue23285, so you can expect it to land in the > main repo relatively soon. > > Regards > > Antoine. > I may be chiming in a little late, however, I'd have a question: does this affect non-blocking applications somehow? How often should we expect to receive EINTR? Is it correct to state that in case many EINTR signals are sent repeatedly a non-blocking framework such as asyncio may hang for "too long"? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Feb 2 22:17:54 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 2 Feb 2015 15:17:54 -0600 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: <85AF8EB7-ECA0-49E6-8771-B27BF935769D@gmail.com> References: <85AF8EB7-ECA0-49E6-8771-B27BF935769D@gmail.com> Message-ID: In reality, things just got broken even more. I don't know when that patch was created, but it's now very out of date: importlib._bootstrap has no load function. That's what the error you're getting is telling you. Since it isn't getting to load anything, the issue seems "solved". Not really. What's the full traceback for the undefined reference exception? On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton wrote: > Update: While waiting for replies I made the change referenced here: > https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80 > > specifically changing > importlib. _bootstrap._SpecMethods(spec).load() > to > importlib._bootstrap.load(spec) > > I no longer get a terminating 'undefined reference to dlopen' error, but I > do get 'importing extensions failed' errors for each extension...like this: > > *** WARNING: importing extension "_pickle" failed with 'AttributeError'>: 'module' object has no attribute 'load' > > > On February 2, 2015 1:36:29 PM CST, Cyd Haselton > wrote: >> >> After fixing a segfault issue (many thanks Ryan) I'm back to the same >> issue I was having with Python 2.7.8; the newly built python throws an >> undefined reference to dlopen when running setup.py...specifically when >> importing just-built extensions >> >> I've managed to narrow the problem down to the following line: >> >> importlib._bootstrap._SpecMethods(spec).load() >> >> Googling this brings up a few hits from bugs.python.org and not much >> else. I'm new to Python; any ideas what this does...or where I can read up >> on it for troubleshooting purposes? >> > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaselton at gmail.com Mon Feb 2 22:24:41 2015 From: chaselton at gmail.com (Cyd Haselton) Date: Mon, 02 Feb 2015 15:24:41 -0600 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: References: <85AF8EB7-ECA0-49E6-8771-B27BF935769D@gmail.com> Message-ID: No traceback unfortunately...the fakechroot in the environment throws the error and setup.py fails. I'll roll back that change...any idea where I could find info about the original method? On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez wrote: >In reality, things just got broken even more. I don't know when that >patch >was created, but it's now very out of date: importlib._bootstrap has no >load function. That's what the error you're getting is telling you. >Since >it isn't getting to load anything, the issue seems "solved". Not >really. > >What's the full traceback for the undefined reference exception? > >On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton >wrote: > >> Update: While waiting for replies I made the change referenced here: >> >https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80 >> >> specifically changing >> importlib. _bootstrap._SpecMethods(spec).load() >> to >> importlib._bootstrap.load(spec) >> >> I no longer get a terminating 'undefined reference to dlopen' error, >but I >> do get 'importing extensions failed' errors for each extension...like >this: >> >> *** WARNING: importing extension "_pickle" failed with > 'AttributeError'>: 'module' object has no attribute 'load' >> >> >> On February 2, 2015 1:36:29 PM CST, Cyd Haselton > >> wrote: >>> >>> After fixing a segfault issue (many thanks Ryan) I'm back to the >same >>> issue I was having with Python 2.7.8; the newly built python throws >an >>> undefined reference to dlopen when running setup.py...specifically >when >>> importing just-built extensions >>> >>> I've managed to narrow the problem down to the following line: >>> >>> importlib._bootstrap._SpecMethods(spec).load() >>> >>> Googling this brings up a few hits from bugs.python.org and not much >>> else. I'm new to Python; any ideas what this does...or where I can >read up >>> on it for troubleshooting purposes? >>> >> >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com >> >> > > >-- >Ryan >If anybody ever asks me why I prefer C++ to C, my answer will be >simple: >"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that >was >nul-terminated." >Personal reality distortion fields are immune to contradictory >evidence. - >srean >Check out my website: http://kirbyfan64.github.io/ -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Feb 2 22:32:38 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 2 Feb 2015 15:32:38 -0600 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: References: <85AF8EB7-ECA0-49E6-8771-B27BF935769D@gmail.com> Message-ID: Unfortunately, I have no idea. You might want to ask someone who's more familiar with the Python source than I am. What exactly appears? Does it say anything about the source of the error? On Mon, Feb 2, 2015 at 3:24 PM, Cyd Haselton wrote: > No traceback unfortunately...the fakechroot in the environment throws the > error and setup.py fails. > > I'll roll back that change...any idea where I could find info about the > original method? > > > On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez > wrote: >> >> In reality, things just got broken even more. I don't know when that >> patch was created, but it's now very out of date: importlib._bootstrap has >> no load function. That's what the error you're getting is telling you. >> Since it isn't getting to load anything, the issue seems "solved". Not >> really. >> >> What's the full traceback for the undefined reference exception? >> >> On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton wrote: >> >>> Update: While waiting for replies I made the change referenced here: >>> https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80 >>> >>> specifically changing >>> importlib. _bootstrap._SpecMethods(spec).load() >>> to >>> importlib._bootstrap.load(spec) >>> >>> I no longer get a terminating 'undefined reference to dlopen' error, but >>> I do get 'importing extensions failed' errors for each extension...like >>> this: >>> >>> *** WARNING: importing extension "_pickle" failed with >> 'AttributeError'>: 'module' object has no attribute 'load' >>> >>> >>> On February 2, 2015 1:36:29 PM CST, Cyd Haselton >>> wrote: >>>> >>>> After fixing a segfault issue (many thanks Ryan) I'm back to the same >>>> issue I was having with Python 2.7.8; the newly built python throws an >>>> undefined reference to dlopen when running setup.py...specifically >>>> when importing just-built extensions >>>> >>>> I've managed to narrow the problem down to the following line: >>>> >>>> importlib._bootstrap._SpecMethods(spec).load() >>>> >>>> Googling this brings up a few hits from bugs.python.org and not much >>>> else. I'm new to Python; any ideas what this does...or where I can read up >>>> on it for troubleshooting purposes? >>>> >>> >>> -- >>> Sent from my Android device with K-9 Mail. Please excuse my brevity. >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com >>> >>> >> >> >> -- >> Ryan >> If anybody ever asks me why I prefer C++ to C, my answer will be simple: >> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was >> nul-terminated." >> Personal reality distortion fields are immune to contradictory evidence. >> - srean >> Check out my website: http://kirbyfan64.github.io/ >> > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Feb 2 22:46:53 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 2 Feb 2015 22:46:53 +0100 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: References: <20150202214504.06613207@fsol> Message-ID: 2015-02-02 22:11 GMT+01:00 Giampaolo Rodola' : > I may be chiming in a little late, however, I'd have a question: does this > affect non-blocking applications somehow? > How often should we expect to receive EINTR? Each time a program is interrupted by a signal while it was waiting for a sycall. Most syscalls are fast, especially in an application written with non blocking operations. For example, timeit says that os.getuid() takes 370 nanoseconds (on Linux). getpid() only takes 285 nanoseconds, but it looks to be cached in the libc: strace doesn't show me syscalls. Network syscalls are probably slower. haypo at selma$ ./python -Wignore -m timeit -s 'import socket; s,c=socket.socketpair()' 's.send(b"a"); c.recv(1)' 100000 loops, best of 3: 2.26 usec per loop > Is it correct to state that in case many EINTR signals are sent > repeatedly a non-blocking framework such as asyncio may hang for "too long"? A syscall fails with EINTR each time it gets a signal. You are unlikely if the same syscall requires to be retried twice (executed 3 times) because it got EINTR twice. I don't see why the kernel would make a syscall fails EINTR multiple times. asyncio doesn't process the signal immediatly. asyncio uses signal.set_wakeup_fd(). At the C level, the C signal handler writes the signal number into a pipe. At the Python level, the selector is awaken by the write. Later, asyncio executes the Python handler of the signal (if a Python signal handler was registered). A nice side effect of the PEP is to avoid to awake the application if it's not necessary. If no Python signal handler is registered, no byte is written into the pipe, the selector continues to wait for events. Victor From matthew.i.frank at intel.com Mon Feb 2 23:09:20 2015 From: matthew.i.frank at intel.com (Frank, Matthew I) Date: Mon, 2 Feb 2015 22:09:20 +0000 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: References: <85AF8EB7-ECA0-49E6-8771-B27BF935769D@gmail.com> Message-ID: <8FEF00DDD6C58843BF4ED3C1DCF80E9F6E13A2A5@FMSMSX106.amr.corp.intel.com> There?s now (as of a couple days ago) a python mobile-sig (https://mail.python.org/mailman/listinfo/mobile-sig). While that group is much smaller than python-dev and probably not as knowledgeable about the Cpython source base, that?s where you?re going to find folks who have a vested interest in getting Python working on mobile platforms like Android. (And who would be interested in hearing about your experiences, no matter whether they can help you or not.) What you are doing (trying to run the entire C compiler toolchain on an Android machine, instead of using the cross-compilers that come in Google?s Android NDK (native development kit)) is way outside the mainstream. (I.e., you?re not going to find very many people trying to do this here or anywhere.) Since you compiled all the libraries that you are linking against, there is some possibility (likelihood) that the problem is in one of those libraries, not anywhere in the CPython source. (Some other library was compiled without the necessary ?dl flag.) The right way to track down this problem (no matter where it is) is to run under gdb and type ?where? when the program crashes. The reason I suspect that the problem is in one of the libraries you compiled before building Python, rather than a problem in the CPython sources or build scripts, is that I don?t have this problem when I cross-compile Python 3.4.2 on a Linux machine, then take the result and copy it over to the Android machine. I?ve posted instructions and patches for successfully performing this cross compilation (for Android KitKat running on an x86) here: https://github.com/wandering-logic/android_x86_python-3.4. Best, -Matt From: Cyd Haselton [mailto:chaselton at gmail.com] Sent: Monday, February 02, 2015 3:25 PM To: Ryan Gonzalez Cc: Python-Dev Subject: Re: [Python-Dev] Import Fails in setup.py On Android No traceback unfortunately...the fakechroot in the environment throws the error and setup.py fails. I'll roll back that change...any idea where I could find info about the original method? On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez > wrote: In reality, things just got broken even more. I don't know when that patch was created, but it's now very out of date: importlib._bootstrap has no load function. That's what the error you're getting is telling you. Since it isn't getting to load anything, the issue seems "solved". Not really. What's the full traceback for the undefined reference exception? On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton > wrote: Update: While waiting for replies I made the change referenced here: https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80 specifically changing importlib. _bootstrap._SpecMethods(spec).load() to importlib._bootstrap.load(spec) I no longer get a terminating 'undefined reference to dlopen' error, but I do get 'importing extensions failed' errors for each extension...like this: *** WARNING: importing extension "_pickle" failed with : 'module' object has no attribute 'load' On February 2, 2015 1:36:29 PM CST, Cyd Haselton > wrote: After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions I've managed to narrow the problem down to the following line: importlib._bootstrap._SpecMethods(spec).load() Googling this brings up a few hits from bugs.python.org and not much else. I'm new to Python; any ideas what this does...or where I can read up on it for troubleshooting purposes? -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Tue Feb 3 07:10:54 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 2 Feb 2015 23:10:54 -0700 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: References: Message-ID: On Mon, Feb 2, 2015 at 12:36 PM, Cyd Haselton wrote: > After fixing a segfault issue (many thanks Ryan) I'm back to the same issue > I was having with Python 2.7.8; the newly built python throws an undefined > reference to dlopen when running setup.py...specifically when importing > just-built extensions > > I've managed to narrow the problem down to the following line: > > importlib._bootstrap._SpecMethods(spec).load() That call is where modules are created (and executed) via the loader designated for the module by the import machinery. So a problem here may simply reflect a problem with the loader. Which module is being imported when you run into trouble? Is it a C-extension module? Also keep in mind that basically everything in importlib._bootstrap is frozen into your Python binary when you build Python (or should be). So if you are seeing any errors related to something missing or broken in importlib._bootstrap, it could be related to the build step where importlib gets frozen. Either way, more info (e.g. a traceback) would be great if you need more help. -eric From ncoghlan at gmail.com Tue Feb 3 13:02:11 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Feb 2015 22:02:11 +1000 Subject: [Python-Dev] Workflow PEP proposals are now closed In-Reply-To: References: <1514D26E-392E-4417-86F1-38D9E747DE46@stufft.io> Message-ID: On 3 Feb 2015 01:26, "Brett Cannon" wrote: >>> >> >> >> Is there going to be discussion between the two approaches or should the PEPs themselves address each other? > > > Since PEPs are meant to act as a record of what was discussed on a topic then it probably wouldn't hurt to incorporate why your approach is better than the other one in the PEP itself. We can obviously talk openly here when you feel the PEP is ready for it. One key point worth noting is that the addition of Phabricator to Donald's proposal actually addresses my most critical concerns regarding workflow lock-in to a proprietary platform. While I'm genuinely unreasonable on that particular topic in an upstream context, part of my vehemence in the original thread was due to bleedover from arguments in other contexts that happened to be running concurrently, so my belated apologies for that. In my view, this is now a contest between two different proposals where I think both represent significant improvements over the status quo. I obviously still prefer mine, though :) Cheers, Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 3 13:04:56 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Feb 2015 22:04:56 +1000 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: <20150202214504.06613207@fsol> References: <20150202214504.06613207@fsol> Message-ID: On 3 Feb 2015 06:46, "Antoine Pitrou" wrote: > > > Hello, > > I'm now accepting PEP 475 - "Retry system calls failing with EINTR". > You can read it at https://www.python.org/dev/peps/pep-0475/ > > The implementation is more or less ready at > http://bugs.python.org/issue23285, so you can expect it to land in the > main repo relatively soon. Great news! Thanks for working through that :) Cheers, Nick. > > Regards > > Antoine. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Tue Feb 3 15:25:48 2015 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 3 Feb 2015 15:25:48 +0100 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: References: <20150202214504.06613207@fsol> Message-ID: On Mon, Feb 2, 2015 at 10:46 PM, Victor Stinner wrote: > 2015-02-02 22:11 GMT+01:00 Giampaolo Rodola' : > > I may be chiming in a little late, however, I'd have a question: does > this > > affect non-blocking applications somehow? > > How often should we expect to receive EINTR? > > Each time a program is interrupted by a signal while it was waiting > for a sycall. Most syscalls are fast, especially in an application > written with non blocking operations. > > For example, timeit says that os.getuid() takes 370 nanoseconds (on > Linux). getpid() only takes 285 nanoseconds, but it looks to be cached > in the libc: strace doesn't show me syscalls. > > Network syscalls are probably slower. > > haypo at selma$ ./python -Wignore -m timeit -s 'import socket; > s,c=socket.socketpair()' 's.send(b"a"); c.recv(1)' > 100000 loops, best of 3: 2.26 usec per loop > > > Is it correct to state that in case many EINTR signals are sent > > repeatedly a non-blocking framework such as asyncio may hang for "too > long"? > > A syscall fails with EINTR each time it gets a signal. You are > unlikely if the same syscall requires to be retried twice (executed 3 > times) because it got EINTR twice. > > I don't see why the kernel would make a syscall fails EINTR multiple times. > > asyncio doesn't process the signal immediatly. asyncio uses > signal.set_wakeup_fd(). At the C level, the C signal handler writes > the signal number into a pipe. At the Python level, the selector is > awaken by the write. Later, asyncio executes the Python handler of the > signal (if a Python signal handler was registered). > > A nice side effect of the PEP is to avoid to awake the application if > it's not necessary. If no Python signal handler is registered, no byte > is written into the pipe, the selector continues to wait for events. > > Victor > OK, thanks for clarifying, this is a very nice addition. One last thing: should InterruptedError exception be deprecated? As far as I understand it should never occur again, right? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Feb 3 16:22:22 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 3 Feb 2015 16:22:22 +0100 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: References: <20150202214504.06613207@fsol> Message-ID: 2015-02-03 15:25 GMT+01:00 Giampaolo Rodola' : > OK, thanks for clarifying, this is a very nice addition. One last thing: > should InterruptedError exception be deprecated? As far as I understand it > should never occur again, right? signal.setinterrupt() is not deprecated so you can still "disable" the PEP for a specific signal. Charles-Fran?ois didn't want to deprecate this function. https://docs.python.org/dev/library/signal.html#signal.siginterrupt Since the code to handle InterruptedError will be removed from the stdlib, the purpose of this function becomes less obvious to me... Victor From larry at hastings.org Tue Feb 3 16:25:46 2015 From: larry at hastings.org (Larry Hastings) Date: Tue, 03 Feb 2015 07:25:46 -0800 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: References: <20150202214504.06613207@fsol> Message-ID: <54D0E87A.9020404@hastings.org> On 02/02/2015 12:58 PM, Victor Stinner wrote: > 2015-02-02 21:49 GMT+01:00 Guido van Rossum : >> W00t! Congratulations les Fran?ais! > We will celebrate this acceptance with a glass of red wine and cheese. If it were me, I'd use separate glasses. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaselton at gmail.com Tue Feb 3 12:37:18 2015 From: chaselton at gmail.com (Cyd Haselton) Date: Tue, 03 Feb 2015 05:37:18 -0600 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: <8FEF00DDD6C58843BF4ED3C1DCF80E9F6E13A2A5@FMSMSX106.amr.corp.intel.com> References: <85AF8EB7-ECA0-49E6-8771-B27BF935769D@gmail.com> <8FEF00DDD6C58843BF4ED3C1DCF80E9F6E13A2A5@FMSMSX106.amr.corp.intel.com> Message-ID: <7EA0A5AA-31B5-4BEA-82B2-EC675361BCD8@gmail.com> Reply in body of email below for clarity On February 2, 2015 4:09:20 PM CST, "Frank, Matthew I" wrote: >There?s now (as of a couple days ago) a python mobile-sig >(https://mail.python.org/mailman/listinfo/mobile-sig). While that >group is much smaller than python-dev and probably not as knowledgeable >about the Cpython source base, that?s where you?re going to find folks >who have a vested interest in getting Python working on mobile >platforms like Android. (And who would be interested in hearing about >your experiences, no matter whether they can help you or not.) > Thanks for the notification. I'll definitely shoot them an email. >What you are doing (trying to run the entire C compiler toolchain on an >Android machine, instead of using the cross-compilers that come in >Google?s Android NDK (native development kit)) is way outside the >mainstream. (I.e., you?re not going to find very many people trying to >do this here or anywhere.) There are several IDEs made for on-device compilation that include the GCC toolchain ported to the Android platform. http://www.spartacusrex.com/terminalide.htm http://kevinboone.net/kbox2_how_it_works.html I use the latter of the two (but contribute binaries to the former) because of its focus on emulating a Linux filesystem in which to run software. > >Since you compiled all the libraries that you are linking against, >there is some possibility (likelihood) that the problem is in one of >those libraries, not anywhere in the CPython source. (Some other >library was compiled without the necessary ?dl flag.) The right way to >track down this problem (no matter where it is) is to run under gdb and >type ?where? when the program crashes. > I did not compile all of the libraries that I'm linking against. The GCC port I am using makes use of a sysroot, which contains the necessary Android system libs and includes. The KBOX environment is built on a fakechroot that supports dynamic calls to Android's libc (and, I suspect, those functions that USUALLY reside in a normal libc...like dlopen) only. Provided a build includes the correct -ldl -lc links, it will work in KBOX. >The reason I suspect that the problem is in one of the libraries you >compiled before building Python, rather than a problem in the CPython >sources or build scripts, is that I don?t have this problem when I >cross-compile Python 3.4.2 on a Linux machine, then take the result and >copy it over to the Android machine. I?ve posted instructions and >patches for successfully performing this cross compilation (for Android >KitKat running on an x86) here: >https://github.com/wandering-logic/android_x86_python-3.4. > See answer above. I'm working in an environment on the Android device. Hopefully this clarifies things. I'll re-post this over at python-sig. >Best, >-Matt > >From: Cyd Haselton [mailto:chaselton at gmail.com] >Sent: Monday, February 02, 2015 3:25 PM >To: Ryan Gonzalez >Cc: Python-Dev >Subject: Re: [Python-Dev] Import Fails in setup.py On Android > >No traceback unfortunately...the fakechroot in the environment throws >the error and setup.py fails. > >I'll roll back that change...any idea where I could find info about the >original method? >On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez >> wrote: >In reality, things just got broken even more. I don't know when that >patch was created, but it's now very out of date: importlib._bootstrap >has no load function. That's what the error you're getting is telling >you. Since it isn't getting to load anything, the issue seems "solved". >Not really. > >What's the full traceback for the undefined reference exception? > >On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton >> wrote: >Update: While waiting for replies I made the change referenced here: >https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80 > >specifically changing >importlib. _bootstrap._SpecMethods(spec).load() >to >importlib._bootstrap.load(spec) > >I no longer get a terminating 'undefined reference to dlopen' error, >but I do get 'importing extensions failed' errors for each >extension...like this: > >*** WARNING: importing extension "_pickle" failed with 'AttributeError'>: 'module' object has no attribute 'load' > >On February 2, 2015 1:36:29 PM CST, Cyd Haselton >> wrote: >After fixing a segfault issue (many thanks Ryan) I'm back to the same >issue I was having with Python 2.7.8; the newly built python throws an >undefined reference to dlopen when running >setup.py...specifically when importing just-built >extensions > >I've managed to narrow the problem down to the following line: > >importlib._bootstrap._SpecMethods(spec).load() > >Googling this brings up a few hits from >bugs.python.org and not much else. I'm new to >Python; any ideas what this does...or where I can read up on it for >troubleshooting purposes? > >-- >Sent from my Android device with K-9 Mail. Please excuse my brevity. > >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > > >-- >Ryan >If anybody ever asks me why I prefer C++ to C, my answer will be >simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think >that was nul-terminated." >Personal reality distortion fields are immune to contradictory >evidence. - srean >Check out my website: http://kirbyfan64.github.io/ > >-- >Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From chaselton at gmail.com Tue Feb 3 14:20:06 2015 From: chaselton at gmail.com (Cyd Haselton) Date: Tue, 03 Feb 2015 07:20:06 -0600 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: References: Message-ID: <0BC1670D-6571-455A-A5A9-0449B9760E43@gmail.com> The array module is the current culprit. Are there details about the freeze process at Python.org? That may help me figure out what and why my build breaks? I'll try my best to get a traceback...I don't have a working gdb in this env yet...and if I do get one I'll add it to the email I send to python-sig On February 3, 2015 12:10:54 AM CST, Eric Snow wrote: >On Mon, Feb 2, 2015 at 12:36 PM, Cyd Haselton >wrote: >> After fixing a segfault issue (many thanks Ryan) I'm back to the same >issue >> I was having with Python 2.7.8; the newly built python throws an >undefined >> reference to dlopen when running setup.py...specifically when >importing >> just-built extensions >> >> I've managed to narrow the problem down to the following line: >> >> importlib._bootstrap._SpecMethods(spec).load() > >That call is where modules are created (and executed) via the loader >designated for the module by the import machinery. So a problem here >may simply reflect a problem with the loader. Which module is being >imported when you run into trouble? Is it a C-extension module? > >Also keep in mind that basically everything in importlib._bootstrap is >frozen into your Python binary when you build Python (or should be). >So if you are seeing any errors related to something missing or broken >in importlib._bootstrap, it could be related to the build step where >importlib gets frozen. > >Either way, more info (e.g. a traceback) would be great if you need >more help. > >-eric -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaselton at gmail.com Tue Feb 3 14:48:34 2015 From: chaselton at gmail.com (Cyd Haselton) Date: Tue, 03 Feb 2015 07:48:34 -0600 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: References: <85AF8EB7-ECA0-49E6-8771-B27BF935769D@gmail.com> Message-ID: It's not very helpful, but here it us if you want to look at it. I added the INFO/ALERT statements in setup.py for troubleshooting purposes. INFO: removing statically built modules INFO: Parsing Modules/Setup and Setup.local INFO: Getting env var to pass to setup.py INFO: Building remaining exts ALERT: building extension "array" None building 'array' extension gcc --sysroot=/usr/gcc-4.9.2/sysroot -fPIC -Wno-unused-result -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -mandroid -mbionic -fno-builtin-sin -fno-builtin-cos -mandroid -mbionic -fno-builtin-sin -fno-builtin-cos -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/gcc-4.9.2/sysroot/usr/include -I/bld/python/Python-3.4.2/Include -I/bld/python/Python-3.4.2 -c /bld/python/Python-3.4.2/Modules/arraymodule.c -o build/temp.linux-armv7l-3.4/bld/python/Python-3.4.2/Modules/arraymodule.o gcc --sysroot=/usr/gcc-4.9.2/sysroot -shared -Wl,--dynamic-linker=/system/bin/linker -Wl,--sysroot=/usr/gcc-4.9.2/sysroot -Wl,--dynamic-linker=/system/bin/linker -Wl,--sysroot=/usr/gcc-4.9.2/sysroot -L/usr/local/lib -Wl,--dynamic-linker=/system/bin/linker -Wl,--sysroot=/usr/gcc-4.9.2/sysroot -L/usr/local/lib -mandroid -mbionic -fno-builtin-sin -fno-builtin-cos build/temp.linux-armv7l-3.4/bld/python/Python-3.4.2/Modules/arraymodule.o -L. -L/usr/local/lib -L/usr/gcc-4.9.2/sysroot/usr/lib -lc -ldl -lm -lpython3.4m -o build/lib.linux-armv7l-3.4/array.cpython-34m.so INFO: building "array" complete None ALERT: importing built module "array" None fakechroot: dlopen: undefined symbol: dlopen On February 2, 2015 3:32:38 PM CST, Ryan Gonzalez wrote: >Unfortunately, I have no idea. You might want to ask someone who's more >familiar with the Python source than I am. > >What exactly appears? Does it say anything about the source of the >error? > >On Mon, Feb 2, 2015 at 3:24 PM, Cyd Haselton >wrote: > >> No traceback unfortunately...the fakechroot in the environment throws >the >> error and setup.py fails. >> >> I'll roll back that change...any idea where I could find info about >the >> original method? >> >> >> On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez >> wrote: >>> >>> In reality, things just got broken even more. I don't know when that >>> patch was created, but it's now very out of date: >importlib._bootstrap has >>> no load function. That's what the error you're getting is telling >you. >>> Since it isn't getting to load anything, the issue seems "solved". >Not >>> really. >>> >>> What's the full traceback for the undefined reference exception? >>> >>> On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton >wrote: >>> >>>> Update: While waiting for replies I made the change referenced >here: >>>> >https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80 >>>> >>>> specifically changing >>>> importlib. _bootstrap._SpecMethods(spec).load() >>>> to >>>> importlib._bootstrap.load(spec) >>>> >>>> I no longer get a terminating 'undefined reference to dlopen' >error, but >>>> I do get 'importing extensions failed' errors for each >extension...like >>>> this: >>>> >>>> *** WARNING: importing extension "_pickle" failed with >>> 'AttributeError'>: 'module' object has no attribute 'load' >>>> >>>> >>>> On February 2, 2015 1:36:29 PM CST, Cyd Haselton > >>>> wrote: >>>>> >>>>> After fixing a segfault issue (many thanks Ryan) I'm back to the >same >>>>> issue I was having with Python 2.7.8; the newly built python >throws an >>>>> undefined reference to dlopen when running setup.py...specifically >>>>> when importing just-built extensions >>>>> >>>>> I've managed to narrow the problem down to the following line: >>>>> >>>>> importlib._bootstrap._SpecMethods(spec).load() >>>>> >>>>> Googling this brings up a few hits from bugs.python.org and not >much >>>>> else. I'm new to Python; any ideas what this does...or where I can >read up >>>>> on it for troubleshooting purposes? >>>>> >>>> >>>> -- >>>> Sent from my Android device with K-9 Mail. Please excuse my >brevity. >>>> >>>> _______________________________________________ >>>> Python-Dev mailing list >>>> Python-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/python-dev >>>> Unsubscribe: >>>> >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com >>>> >>>> >>> >>> >>> -- >>> Ryan >>> If anybody ever asks me why I prefer C++ to C, my answer will be >simple: >>> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that >was >>> nul-terminated." >>> Personal reality distortion fields are immune to contradictory >evidence. >>> - srean >>> Check out my website: http://kirbyfan64.github.io/ >>> >> >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. >> > > > >-- >Ryan >If anybody ever asks me why I prefer C++ to C, my answer will be >simple: >"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that >was >nul-terminated." >Personal reality distortion fields are immune to contradictory >evidence. - >srean >Check out my website: http://kirbyfan64.github.io/ -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Tue Feb 3 21:01:49 2015 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 03 Feb 2015 21:01:49 +0100 Subject: [Python-Dev] PEP 475 accepted In-Reply-To: <54D0E87A.9020404@hastings.org> References: <20150202214504.06613207@fsol> <54D0E87A.9020404@hastings.org> Message-ID: On 02/03/2015 04:25 PM, Larry Hastings wrote: > > On 02/02/2015 12:58 PM, Victor Stinner wrote: >> 2015-02-02 21:49 GMT+01:00 Guido van Rossum : >>> W00t! Congratulations les Fran?ais! >> We will celebrate this acceptance with a glass of red wine and cheese. > > If it were me, I'd use separate glasses. Don't worry. It might have been runny enough to put in a glass, but the cat's eaten it. Georg From jdhardy at gmail.com Wed Feb 4 22:10:38 2015 From: jdhardy at gmail.com (Jeff Hardy) Date: Wed, 4 Feb 2015 21:10:38 +0000 Subject: [Python-Dev] Python Mobile-sig Message-ID: There is now a Python SIG dedicated to running Python on mobile devices. Sign up for the mobile-sig at python.org mailing list and find out more at [1]. The Mobile-SIG exists to improve the usability of Python on mobile devices, as mobile platforms have their own unique requirements and constraints. There are two goals: * To collaborate on porting CPython to mobile platforms and ensure that other Python implementations (i.e. Jython, IronPython) present much the same environment as CPython when running on mobile devices. * To collaborate on the design of modules for mobile-centric features such as cameras, GPS, accelerometers, and other sensors to ensure cross-platform and cross-implementation compatibility. The intent is to work on mobile-centric features in this group before proposal to python-dev, similar to how the distutils-sig operates. - Jeff [1] https://www.python.org/community/sigs/current/mobile-sig/ From info at martin-thoma.de Wed Feb 4 20:04:03 2015 From: info at martin-thoma.de (Martin Thoma) Date: Wed, 4 Feb 2015 20:04:03 +0100 Subject: [Python-Dev] Why does pip upgrade to a random version? Message-ID: Could somebody please have a look at the following SO question? It seems as if I might have found a bug in pip: http://stackoverflow.com/q/28282671/562769 TL;DR of the SO question: I executed `$ sudo pip install hwrt --upgrade` mutiple times and got a - seemingly random - version of hwrt installed. Best regards, Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhou at rocketsoftware.com Thu Feb 5 00:18:08 2015 From: jzhou at rocketsoftware.com (Jianhua Zhou) Date: Wed, 4 Feb 2015 23:18:08 +0000 Subject: [Python-Dev] Hi, I am new to this board and have a question Message-ID: <0639c2e30a8e4432bc36cc5917ee2d28@wal-vm-mbx1.rocketsoftware.com> Hi Everyone, I am a core software engineer at Rocket Software Inc. I am working on database system called UniData and Universe. Now we plan to introduce Python as the new programming language to our customer. When I try to build the python 3.4.1 on Red Hat Linux platform. I found some problem and need help. After I run configure and make the python, at the very end, it said some optional modules we missing, Python build finished successfully! The necessary bits to build these optional modules were not found: _bz2 _lzma _ssl _tkinter zlib So, I went to python development guide site and found out I have to download some other source to build the optional modules. Since I am on RH Linux and yum is installed on my Linux box, so I run following command # yum install yum-utils Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. WANdisco | 951 B 00:00 WANdisco/primary | 94 kB 00:00 WANdisco 450/450 my_repo | 3.9 kB 00:00 ... Setting up Install Process Package yum-utils-1.1.30-14.el6.noarch already installed and latest version Nothing to do Looks like yum-utils is already installed. Then # yum-builddep python3 Loaded plugins: product-id, refresh-packagekit No such package(s): python3 So what package name should I gave to download the additional source to build the optional module? Thanks! Jianhua Zhou ================================ Rocket Software, Inc. and subsidiaries ? 77 Fourth Avenue, Waltham MA 02451 ? +1 800.966.3270 ? +1 781.577.4321 Unsubscribe From Commercial Email - unsubscribe at rocketsoftware.com Manage Your Subscription Preferences - http://info.rocketsoftware.com/GlobalSubscriptionManagementEmailFooter_SubscriptionCenter.html Privacy Policy - http://www.rocketsoftware.com/company/legal/privacy-policy ================================ This communication and any attachments may contain confidential information of Rocket Software, Inc. All unauthorized use, disclosure or distribution is prohibited. If you are not the intended recipient, please notify Rocket Software immediately and destroy all copies of this communication. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdunck at gmail.com Thu Feb 5 01:41:30 2015 From: jdunck at gmail.com (Jeremy Dunck) Date: Wed, 4 Feb 2015 16:41:30 -0800 Subject: [Python-Dev] Why does pip upgrade to a random version? In-Reply-To: References: Message-ID: Does this happen with other packages? I wonder if perhaps the issue is with the package itself. On Wed, Feb 4, 2015 at 11:04 AM, Martin Thoma wrote: > Could somebody please have a look at the following SO question? It seems > as if I might have found a bug in pip: > http://stackoverflow.com/q/28282671/562769 > > TL;DR of the SO question: > I executed `$ sudo pip install hwrt --upgrade` mutiple times and got a - > seemingly random - version of hwrt installed. > > Best regards, > Martin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jdunck%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Thu Feb 5 03:23:31 2015 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 5 Feb 2015 13:23:31 +1100 Subject: [Python-Dev] Hi, I am new to this board and have a question In-Reply-To: <0639c2e30a8e4432bc36cc5917ee2d28@wal-vm-mbx1.rocketsoftware.com> References: <0639c2e30a8e4432bc36cc5917ee2d28@wal-vm-mbx1.rocketsoftware.com> Message-ID: <20150205022331.GA76368@cskk.homeip.net> On 04Feb2015 23:18, Jianhua Zhou wrote: >Hi Everyone, > >I am a core software engineer at Rocket Software Inc. I am working on database system called UniData and Universe. > >Now we plan to introduce Python as the new programming language to our customer. When I try to build the python 3.4.1 on Red Hat Linux platform. I found some problem and need help. > >After I run configure and make the python, at the very end, it said some optional modules we missing, > >Python build finished successfully! >The necessary bits to build these optional modules were not found: >_bz2 _lzma _ssl >_tkinter zlib > >So, I went to python development guide site and found out I have to download some other source to build the optional modules. >Since I am on RH Linux and yum is installed on my Linux box, so I run following command ># yum install yum-utils [...] >Package yum-utils-1.1.30-14.el6.noarch already installed and latest version > >Looks like yum-utils is already installed. Then ># yum-builddep python3 >Loaded plugins: product-id, refresh-packagekit >No such package(s): python3 Redhat don't supply python 3 on RedHat 5 or 6. I don't have access to a RHEL7 box, but if they did and you're using RHEL7 you could just "yum install" it and not bother with building anything. If you're building from source you do not need to involve RPM or yum at all. Unless you intend to ship python3 RPMs to your customers. Do you? Otherwise you can just: - install the relevant library and -devel modules from RedHat, at a guess from a RHEL6 host: _bz2: bzip2-devel bzip2-libs _tkinter: tkinter tk-devel _lzma: xz-devel xz-libs xz-lzma-compat zlib: zlib zlib-devel _ssl: openssl openssl-devel - rebuild from source as before and see what it says (configure, make, etc) Cheers, Cameron Simpson From alexander.belopolsky at gmail.com Thu Feb 5 03:26:36 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 4 Feb 2015 21:26:36 -0500 Subject: [Python-Dev] Hi, I am new to this board and have a question In-Reply-To: <0639c2e30a8e4432bc36cc5917ee2d28@wal-vm-mbx1.rocketsoftware.com> References: <0639c2e30a8e4432bc36cc5917ee2d28@wal-vm-mbx1.rocketsoftware.com> Message-ID: On Wed, Feb 4, 2015 at 6:18 PM, Jianhua Zhou wrote: > The necessary bits to build these optional modules were not found: > > _bz2 _lzma _ssl > > _tkinter zlib > > > > .. > > > > So what package name should I gave to download the additional source to > build the optional module? > > > Questions like this are really better asked on Red Hat forums, but off the top of my head, you need *-devel packages such as zlib-devel, openssl-devel, etc. It is usually easy to figure out from yum search. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Thu Feb 5 04:40:21 2015 From: brian at python.org (Brian Curtin) Date: Wed, 4 Feb 2015 21:40:21 -0600 Subject: [Python-Dev] Why does pip upgrade to a random version? In-Reply-To: References: Message-ID: On Wed, Feb 4, 2015 at 1:04 PM, Martin Thoma wrote: > Could somebody please have a look at the following SO question? It seems as > if I might have found a bug in pip: > http://stackoverflow.com/q/28282671/562769 > > TL;DR of the SO question: > I executed `$ sudo pip install hwrt --upgrade` mutiple times and got a - > seemingly random - version of hwrt installed. Try the pip mailing list: https://groups.google.com/forum/#!forum/pypa-dev From ncoghlan at gmail.com Thu Feb 5 12:09:47 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Feb 2015 21:09:47 +1000 Subject: [Python-Dev] Hi, I am new to this board and have a question In-Reply-To: References: <0639c2e30a8e4432bc36cc5917ee2d28@wal-vm-mbx1.rocketsoftware.com> Message-ID: On 5 February 2015 at 12:26, Alexander Belopolsky wrote: > > On Wed, Feb 4, 2015 at 6:18 PM, Jianhua Zhou > wrote: >> >> The necessary bits to build these optional modules were not found: >> _bz2 _lzma _ssl >> _tkinter zlib >> So what package name should I gave to download the additional source to >> build the optional module? > > Questions like this are really better asked on Red Hat forums, but off the > top of my head, you need *-devel packages such as zlib-devel, openssl-devel, > etc. It is usually easy to figure out from yum search. I actually have a few suggestions here: 1. The system build deps for Python 2 should be the same as those for Python 3, so "yum-builddep python" should solve the problem of getting dependencies to do your own build 2. You may also be interested in Slavek's proposal to add a Python 3.4 stack to EPEL: https://fedoraproject.org/wiki/User:Bkabrda/EPEL7_Python3 (Relevant discussions at [1] and [2]) 3. Alternatively, Software Collections may be a better fit for your use case: http://wiki.centos.org/AdditionalResources/Repositories/SCL As Alexander says though, we're getting a bit off topic for upstream python-dev - this is more a topic for EPEL, CentOS or the Red Hat customer portal. Regards, Nick. [1] https://lists.fedoraproject.org/pipermail/epel-devel/2014-December/010548.html [2] https://lists.fedoraproject.org/pipermail/epel-devel/2014-January/009043.html > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bkabrda at redhat.com Thu Feb 5 16:37:15 2015 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Thu, 5 Feb 2015 10:37:15 -0500 (EST) Subject: [Python-Dev] Hi, I am new to this board and have a question In-Reply-To: <0639c2e30a8e4432bc36cc5917ee2d28@wal-vm-mbx1.rocketsoftware.com> References: <0639c2e30a8e4432bc36cc5917ee2d28@wal-vm-mbx1.rocketsoftware.com> Message-ID: <1813690782.8690510.1423150635637.JavaMail.zimbra@redhat.com> ----- Original Message ----- > Hi Everyone, > I am a core software engineer at Rocket Software Inc. I am working on > database system called UniData and Universe. > Now we plan to introduce Python as the new programming language to our > customer. When I try to build the python 3.4.1 on Red Hat Linux platform. I > found some problem and need help. > After I run configure and make the python, at the very end, it said some > optional modules we missing, > Python build finished successfully! > The necessary bits to build these optional modules were not found: > _bz2 _lzma _ssl > _tkinter zlib > So, I went to python development guide site and found out I have to download > some other source to build the optional modules. > Since I am on RH Linux and yum is installed on my Linux box, so I run > following command > # yum install yum-utils > Loaded plugins: product-id, refresh-packagekit, security, > subscription-manager > This system is not registered to Red Hat Subscription Management. You can use > subscription-manager to register. > WANdisco | 951 B 00:00 > WANdisco/primary | 94 kB 00:00 > WANdisco 450/450 > my_repo | 3.9 kB 00:00 ... > Setting up Install Process > Package yum-utils-1.1.30-14.el6.noarch already installed and latest version > Nothing to do > Looks like yum-utils is already installed. Then > # yum-builddep python3 > Loaded plugins: product-id, refresh-packagekit > No such package(s): python3 > So what package name should I gave to download the additional source to build > the optional module? > Thanks! > Jianhua Zhou > ================================ > Rocket Software, Inc. and subsidiaries ? 77 Fourth Avenue, Waltham MA 02451 ? > +1 800.966.3270 ? +1 781.577.4321 > Unsubscribe From Commercial Email ? unsubscribe at rocketsoftware.com > Manage Your Subscription Preferences - > http://info.rocketsoftware.com/GlobalSubscriptionManagementEmailFooter_SubscriptionCenter.html > Privacy Policy - http://www.rocketsoftware.com/company/legal/privacy-policy > ================================ > This communication and any attachments may contain confidential information > of Rocket Software, Inc. All unauthorized use, disclosure or distribution is > prohibited. If you are not the intended recipient, please notify Rocket > Software immediately and destroy all copies of this communication. Thank > you. Hi Jianhua, this is probably not a question to be asked on this list. If you need some help, please drop me a private mail and I can help you sort this out (I'm the RHEL Python maintainer, so I consider myself pretty knowledgeable about these things ;)). Slavek -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaselton at gmail.com Thu Feb 5 20:56:58 2015 From: chaselton at gmail.com (Cyd Haselton) Date: Thu, 5 Feb 2015 13:56:58 -0600 Subject: [Python-Dev] Import Fails in setup.py On Android In-Reply-To: References: Message-ID: While I'm waiting for a response from mobile-sig, I'm going through the documentation (and something called "fossies") of importlib. I hope that it's okay to ask some follow up questions...even though I've redirected this post to mobile-sig. If it is, my first question(s) is/are about the import machinery and C-extension modules. Assuming that C-extension modules are ones that include files that are named as *.c, how does the import machinery handle specified shared libraries, such as -ldl? Additionally, how does the import machinery handle the symbols from those libraries...if it does at all? On Tue, Feb 3, 2015 at 12:10 AM, Eric Snow wrote: > On Mon, Feb 2, 2015 at 12:36 PM, Cyd Haselton wrote: >> After fixing a segfault issue (many thanks Ryan) I'm back to the same issue >> I was having with Python 2.7.8; the newly built python throws an undefined >> reference to dlopen when running setup.py...specifically when importing >> just-built extensions >> >> I've managed to narrow the problem down to the following line: >> >> importlib._bootstrap._SpecMethods(spec).load() > > That call is where modules are created (and executed) via the loader > designated for the module by the import machinery. So a problem here > may simply reflect a problem with the loader. Which module is being > imported when you run into trouble? Is it a C-extension module? > > Also keep in mind that basically everything in importlib._bootstrap is > frozen into your Python binary when you build Python (or should be). > So if you are seeing any errors related to something missing or broken > in importlib._bootstrap, it could be related to the build step where > importlib gets frozen. > > Either way, more info (e.g. a traceback) would be great if you need more help. > > -eric From francis.giraldeau at gmail.com Fri Feb 6 00:27:24 2015 From: francis.giraldeau at gmail.com (Francis Giraldeau) Date: Thu, 5 Feb 2015 18:27:24 -0500 Subject: [Python-Dev] Encoding of PyFrameObject members Message-ID: I need to access frame members from within a signal handler for tracing purpose. My first attempt to access co_filename was like this (omitting error checking): PyFrameObject *frame = PyEval_GetFrame(); PyObject *ob = PyUnicode_AsUTF8String(frame->f_code->co_filename) char *str = PyBytes_AsString(ob) However, the function PyUnicode_AsUTF8String() calls PyObject_Malloc(), which is not reentrant. If the signal handler nest over PyObject_Malloc(), it causes a segfault, and it could also deadlock. Instead, I access members directly: char *str = PyUnicode_DATA(frame->f_code->co_filename); size_t len = PyUnicode_GET_DATA_SIZE(frame->f_code->co_filename); Is it safe to assume that unicode objects co_filename and co_name are always UTF-8 data for loaded code? I looked at the PyTokenizer_FromString() and it seems to convert everything to UTF-8 upfront, and I would like to make sure this assumption is valid. Thanks! Francis -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 6 01:56:36 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 Feb 2015 11:56:36 +1100 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: Message-ID: On Fri, Feb 6, 2015 at 10:27 AM, Francis Giraldeau wrote: > Instead, I access members directly: > char *str = PyUnicode_DATA(frame->f_code->co_filename); > size_t len = PyUnicode_GET_DATA_SIZE(frame->f_code->co_filename); > > Is it safe to assume that unicode objects co_filename and co_name are always > UTF-8 data for loaded code? I looked at the PyTokenizer_FromString() and it > seems to convert everything to UTF-8 upfront, and I would like to make sure > this assumption is valid. I don't think you should be using _GET_DATA_SIZE with _DATA - they're mix-and-matched from old and new APIs. If you want a raw, no-allocation look at the data, you'd need to check PyUnicode_KIND and then read Latin-1, UCS-2, or UCS-4 data: https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_1BYTE_DATA (By the way, I don't think the name "UCS-1" is part of the Unicode spec. But it's an obvious parallel to UCS-2 and UCS-4.) Getting UTF-8 data out of the structure, if it had indeed been cached, ought to be possible. But I can't see a documented function or macro for doing it. Is there a way? Reaching into the structure and grabbing the utf8 and utf8_length members seems like a bad idea. ChrisA From syedk at pacificloud.com Fri Feb 6 06:26:20 2015 From: syedk at pacificloud.com (syed khalid) Date: Thu, 5 Feb 2015 21:26:20 -0800 Subject: [Python-Dev] Azure event hub network access Message-ID: I am getting http error 404. I am able to access the site via telnet which eliminates network issues. Here is the code and subsequent errors user/bin/python import sys import azure import socket from azure.servicebus import ( _service_bus_error_handler ) from azure.servicebus.servicebusservice import ( ServiceBusService, ServiceBusSASAuthentication ) from azure.http import ( HTTPRequest, HTTPError ) from azure.http.httpclient import _HTTPClient class EventHubClient(object): def sendMessage(self,body,partition): eventHubHost = "pac-ns.servicebus.windows.net" httpclient = _HTTPClient(service_instance=self) sasKeyName = "pac-pl" sasKeyValue = "IhkEepQPLfSy9jo6H2Yxxxxxxxxxxxxxxxxxxxx=" authentication = ServiceBusSASAuthentication(sasKeyName,sasKeyValue) request = HTTPRequest() request.method = "POST" request.host = eventHubHost request.protocol_override = "https" # request.path = "/myhub/publishers/" + partition + "/messages?api-version=20 14-05" request.body = body request.headers.append(('Content-Type', 'application/atom+xml;type=entry;cha rset=utf-8')) authentication.sign_request(request, httpclient) request.headers.append(('Content-Length', str(len(request.body)))) status = 0 try: resp = httpclient.perform_request(request) status = resp.status except HTTPError as ex: status = ex.status return status class EventDataParser(object): def getMessage(self,payload,sensorId): host = socket.gethostname() body = "{ \"DeviceId\" : \"" + host + "\",\"SensorData\": [ " msgs = payload.split(",") first = True for msg in msgs: # print msg sensorType = msg.split(":")[0] sensorValue = msg.split(":")[1] if first == True: first = False else: body += "," body += "{ \"SensorId\" : \"" + sensorId + "\", \"SensorType\" : \"" + sen sorType + "\", \"SensorValue\" : " + sensorValue + " }" body += "]}" return body hubClient = EventHubClient() parser = EventDataParser() hostname = socket.gethostname() sensor = sys.argv[2] body = parser.getMessage(sys.argv[1],sensor) hubStatus = hubClient.sendMessage(body,hostname) # return the HTTP status to the caller print hubStatus print hostname print sensor ~/IOT/AZURE$ python send.py temperature:22,humidity:20 deviceid 404 ubuntu deviceid { "DeviceId" : "ubuntu","SensorData": [ { "SensorId" : "deviceid", "SensorType" : "temperature", "SensorValue" : 22 },{ "SensorId" : "deviceid", "SensorType" : "humidity", "SensorValue" : 20 }]} -- *Syed Khalid* -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Fri Feb 6 06:45:37 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 6 Feb 2015 05:45:37 +0000 Subject: [Python-Dev] Azure event hub network access In-Reply-To: References: Message-ID: This would be much better posted on the github page for the project. I don't have the URL handy, but if you search github for "Python Azure SDK" you'll find it. Cheers, Steve Sent from my Windows Phone ________________________________ From: syed khalid Sent: ?2/?5/?2015 21:27 To: python-dev at python.org Cc: python-list at python.org Subject: [Python-Dev] Azure event hub network access I am getting http error 404. I am able to access the site via telnet which eliminates network issues. Here is the code and subsequent errors user/bin/python import sys import azure import socket from azure.servicebus import ( _service_bus_error_handler ) from azure.servicebus.servicebusservice import ( ServiceBusService, ServiceBusSASAuthentication ) from azure.http import ( HTTPRequest, HTTPError ) from azure.http.httpclient import _HTTPClient class EventHubClient(object): def sendMessage(self,body,partition): eventHubHost = "pac-ns.servicebus.windows.net" httpclient = _HTTPClient(service_instance=self) sasKeyName = "pac-pl" sasKeyValue = "IhkEepQPLfSy9jo6H2Yxxxxxxxxxxxxxxxxxxxx=" authentication = ServiceBusSASAuthentication(sasKeyName,sasKeyValue) request = HTTPRequest() request.method = "POST" request.host = eventHubHost request.protocol_override = "https" # request.path = "/myhub/publishers/" + partition + "/messages?api-version=20 14-05" request.body = body request.headers.append(('Content-Type', 'application/atom+xml;type=entry;cha rset=utf-8')) authentication.sign_request(request, httpclient) request.headers.append(('Content-Length', str(len(request.body)))) status = 0 try: resp = httpclient.perform_request(request) status = resp.status except HTTPError as ex: status = ex.status return status class EventDataParser(object): def getMessage(self,payload,sensorId): host = socket.gethostname() body = "{ \"DeviceId\" : \"" + host + "\",\"SensorData\": [ " msgs = payload.split(",") first = True for msg in msgs: # print msg sensorType = msg.split(":")[0] sensorValue = msg.split(":")[1] if first == True: first = False else: body += "," body += "{ \"SensorId\" : \"" + sensorId + "\", \"SensorType\" : \"" + sen sorType + "\", \"SensorValue\" : " + sensorValue + " }" body += "]}" return body hubClient = EventHubClient() parser = EventDataParser() hostname = socket.gethostname() sensor = sys.argv[2] body = parser.getMessage(sys.argv[1],sensor) hubStatus = hubClient.sendMessage(body,hostname) # return the HTTP status to the caller print hubStatus print hostname print sensor ~/IOT/AZURE$ python send.py temperature:22,humidity:20 deviceid 404 ubuntu deviceid { "DeviceId" : "ubuntu","SensorData": [ { "SensorId" : "deviceid", "SensorType" : "temperature", "SensorValue" : 22 },{ "SensorId" : "deviceid", "SensorType" : "humidity", "SensorValue" : 20 }]} -- Syed Khalid -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri Feb 6 07:47:25 2015 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 06 Feb 2015 06:47:25 +0000 Subject: [Python-Dev] Encoding of PyFrameObject members References: Message-ID: On Thu Feb 05 2015 at 4:36:30 PM Francis Giraldeau < francis.giraldeau at gmail.com> wrote: > I need to access frame members from within a signal handler for tracing > purpose. My first attempt to access co_filename was like this (omitting > error checking): > > PyFrameObject *frame = PyEval_GetFrame(); > PyObject *ob = PyUnicode_AsUTF8String(frame->f_code->co_filename) > char *str = PyBytes_AsString(ob) > > However, the function PyUnicode_AsUTF8String() calls PyObject_Malloc(), > which is not reentrant. If the signal handler nest over PyObject_Malloc(), > it causes a segfault, and it could also deadlock. > > Instead, I access members directly: > char *str = PyUnicode_DATA(frame->f_code->co_filename); > size_t len = PyUnicode_GET_DATA_SIZE(frame->f_code->co_filename); > > Is it safe to assume that unicode objects co_filename and co_name are > always UTF-8 data for loaded code? I looked at the PyTokenizer_FromString() > and it seems to convert everything to UTF-8 upfront, and I would like to > make sure this assumption is valid. > The faulthandler module calls into Python/traceback.c in signal context which has some low level code for extracting something useful from PyUnicode objects without triggering a conversion: https://hg.python.org/cpython/file/f1a82e949fb8/Python/traceback.c#l531 That code is written to write output to a fd. It's much more useful for signal handler tracing data collection from a signal handler if you refactor it to dump its output into a preallocated char* buffer. -gps > > Thanks! > > Francis > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Fri Feb 6 08:24:31 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 6 Feb 2015 09:24:31 +0200 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: Message-ID: Hi Francis I don't think it's safe to assume f_code is properly filled by the time you might read it, depending a bit where you find the frame object. Are you sure it's not full of garbage? Besides, are you writing a profiler, or what exactly are you doing? On Fri, Feb 6, 2015 at 1:27 AM, Francis Giraldeau wrote: > I need to access frame members from within a signal handler for tracing > purpose. My first attempt to access co_filename was like this (omitting > error checking): > > PyFrameObject *frame = PyEval_GetFrame(); > PyObject *ob = PyUnicode_AsUTF8String(frame->f_code->co_filename) > char *str = PyBytes_AsString(ob) > > However, the function PyUnicode_AsUTF8String() calls PyObject_Malloc(), > which is not reentrant. If the signal handler nest over PyObject_Malloc(), > it causes a segfault, and it could also deadlock. > > Instead, I access members directly: > char *str = PyUnicode_DATA(frame->f_code->co_filename); > size_t len = PyUnicode_GET_DATA_SIZE(frame->f_code->co_filename); > > Is it safe to assume that unicode objects co_filename and co_name are always > UTF-8 data for loaded code? I looked at the PyTokenizer_FromString() and it > seems to convert everything to UTF-8 upfront, and I would like to make sure > this assumption is valid. > > Thanks! > > Francis > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > From victor.stinner at gmail.com Fri Feb 6 11:07:42 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 6 Feb 2015 11:07:42 +0100 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: Message-ID: Hi, 2015-02-06 0:27 GMT+01:00 Francis Giraldeau : > I need to access frame members from within a signal handler for tracing > purpose. IMO you have a big technical or design issue here. Accessing Python internals in a signal handler is not reliable. A signal can occur anytime, between two instructions. > However, the function PyUnicode_AsUTF8String() calls PyObject_Malloc(), > which is not reentrant. If the signal handler nest over PyObject_Malloc(), > it causes a segfault, and it could also deadlock. Yes, the list of async signal-safe function is very very short :-) It's something like: read(), write(), and use the stack (but not too much stack or you will get a stack overflow). I spent many weeks to implement the faulthandler module (try to write a safe and portable implementation). To write the traceback, I only use write(). But to read the traceback, I inspect Python internals which is completly unsafe. faulthandler is written to only be called when something really bad happen (a "crash"), so it's not so important if it does crash too :-) See also the tracemalloc module which also inspects the traceback, but it does *not* use signals (which would be unsafe). It uses hooks on the memory allocator. Python has sys.settrace() and sys.setprofile(). Why not using these functions? Victor From mal at egenix.com Fri Feb 6 11:44:46 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 06 Feb 2015 11:44:46 +0100 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: Message-ID: <54D49B1E.6090201@egenix.com> On 06.02.2015 00:27, Francis Giraldeau wrote: > I need to access frame members from within a signal handler for tracing > purpose. My first attempt to access co_filename was like this (omitting > error checking): > > PyFrameObject *frame = PyEval_GetFrame(); > PyObject *ob = PyUnicode_AsUTF8String(frame->f_code->co_filename) > char *str = PyBytes_AsString(ob) > > However, the function PyUnicode_AsUTF8String() calls PyObject_Malloc(), > which is not reentrant. If the signal handler nest over PyObject_Malloc(), > it causes a segfault, and it could also deadlock. > > Instead, I access members directly: > char *str = PyUnicode_DATA(frame->f_code->co_filename); > size_t len = PyUnicode_GET_DATA_SIZE(frame->f_code->co_filename); > > Is it safe to assume that unicode objects co_filename and co_name are > always UTF-8 data for loaded code? I looked at the PyTokenizer_FromString() > and it seems to convert everything to UTF-8 upfront, and I would like to > make sure this assumption is valid. The macros won't work in all cases, as they don't pay attention to the different kinds used in the Unicode implementation. I don't think there's any API you can use to extract the underlying data without going through PyObject_Malloc() at some point (you may be lucky if there already is a UTF-8 version available, but it's not guaranteed). I guess your best bet is to write your own UTF-8 codec which then copies the data to a buffer that you can control. Have a look at Objects/stringlib/codecs.h: utf8_encode. Alternatively, you can copy the data to a Py_UCS4 buffer which you allocate using code such as this (untested, adapted from the UTF-8 encoder): Py_UCS4 *p; enum PyUnicode_Kind repkind; void *repdata; Py_ssize_t repsize, k; if (PyUnicode_READY(rep) < 0) goto error; repkind = PyUnicode_KIND(rep); repdata = PyUnicode_DATA(rep); repsize = PyUnicode_GET_LENGTH(rep); p = malloc((repsize + 1) * sizeof(Py_UCS4)); for(k=0; k>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From arigo at tunes.org Fri Feb 6 12:04:07 2015 From: arigo at tunes.org (Armin Rigo) Date: Fri, 6 Feb 2015 12:04:07 +0100 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: Message-ID: Hi, On 6 February 2015 at 08:24, Maciej Fijalkowski wrote: > I don't think it's safe to assume f_code is properly filled by the > time you might read it, depending a bit where you find the frame > object. Are you sure it's not full of garbage? Yes, before discussing how to do the utf8 decoding, we should realize that it is really unsafe code starting from the line before. From a signal handler you're only supposed to read data that was written to "volatile" fields. So even PyEval_GetFrame(), which is done by reading the thread state's "frame" field, is not safe: this is not a volatile. This means that the compiler is free to do crazy things like *first* write into this field and *then* initialize the actual content of the frame. The uninitialized content may be garbage, not just NULLs. A bient?t, Armin. From status at bugs.python.org Fri Feb 6 18:08:15 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 6 Feb 2015 18:08:15 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20150206170815.D80D3561F1@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-01-30 - 2015-02-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4765 ( +1) closed 30398 (+47) total 35163 (+48) Open issues with patches: 2224 Issues opened (29) ================== #16632: Enable DEP and ASLR http://bugs.python.org/issue16632 reopened by haypo #23354: Loading 2 GiLOC file which raises exception causes wrong trace http://bugs.python.org/issue23354 opened by SoniEx2 #23356: In argparse docs simplify example about argline http://bugs.python.org/issue23356 opened by py.user #23357: pyvenv help shows incorrect usage http://bugs.python.org/issue23357 opened by raulcd #23359: Speed-up set_lookkey() http://bugs.python.org/issue23359 opened by rhettinger #23360: Content-Type when sending data with urlopen() http://bugs.python.org/issue23360 opened by vadmium #23361: integer overflow in winapi_createprocess http://bugs.python.org/issue23361 opened by pkt #23362: integer overflow in string translate http://bugs.python.org/issue23362 opened by pkt #23367: integer overflow in unicodedata.normalize http://bugs.python.org/issue23367 opened by pkt #23368: integer overflow in _PyUnicode_AsKind http://bugs.python.org/issue23368 opened by pkt #23371: mimetypes initialization fails on Windows because of TypeError http://bugs.python.org/issue23371 opened by Slion #23372: defaultdict.fromkeys should accept a callable factory http://bugs.python.org/issue23372 opened by justanr #23374: pydoc 3.x raises UnicodeEncodeError on sqlite3 package http://bugs.python.org/issue23374 opened by skip.montanaro #23375: test_py3kwarn fails on Windows http://bugs.python.org/issue23375 opened by serhiy.storchaka #23376: getargs.c: redundant C-contiguity check http://bugs.python.org/issue23376 opened by skrah #23377: HTTPResponse may drop buffer holding next response http://bugs.python.org/issue23377 opened by vadmium #23378: argparse.add_argument action parameter should allow value exte http://bugs.python.org/issue23378 opened by the.mulhern #23382: Maybe can not shutdown ThreadPoolExecutor when call the method http://bugs.python.org/issue23382 opened by miles #23383: Clean up bytes formatting http://bugs.python.org/issue23383 opened by serhiy.storchaka #23384: urllib.proxy_bypass_registry slow down under Windows if websit http://bugs.python.org/issue23384 opened by aristotel #23387: test_urllib2 fails with HTTP Error 502: Bad Gateway http://bugs.python.org/issue23387 opened by berker.peksag #23388: datetime.strftime('%s') does not take timezone into account http://bugs.python.org/issue23388 opened by cameris #23389: pkgutil.find_loader raises an ImportError on PEP 420 implicit http://bugs.python.org/issue23389 opened by alynn #23391: Documentation of EnvironmentError (OSError) arguments disappea http://bugs.python.org/issue23391 opened by vadmium #23394: No garbage collection at end of main thread http://bugs.python.org/issue23394 opened by Fran??ois.Trahan #23395: _thread.interrupt_main() errors if SIGINT handler in SIG_DFL, http://bugs.python.org/issue23395 opened by takluyver #23397: PEP 431 implementation http://bugs.python.org/issue23397 opened by berker.peksag #23400: Inconsistent behaviour of multiprocessing.Queue() if sem_open http://bugs.python.org/issue23400 opened by olebole #23401: Add pickle support of Mapping views http://bugs.python.org/issue23401 opened by serhiy.storchaka Most recent 15 issues with no replies (15) ========================================== #23400: Inconsistent behaviour of multiprocessing.Queue() if sem_open http://bugs.python.org/issue23400 #23395: _thread.interrupt_main() errors if SIGINT handler in SIG_DFL, http://bugs.python.org/issue23395 #23391: Documentation of EnvironmentError (OSError) arguments disappea http://bugs.python.org/issue23391 #23387: test_urllib2 fails with HTTP Error 502: Bad Gateway http://bugs.python.org/issue23387 #23384: urllib.proxy_bypass_registry slow down under Windows if websit http://bugs.python.org/issue23384 #23378: argparse.add_argument action parameter should allow value exte http://bugs.python.org/issue23378 #23377: HTTPResponse may drop buffer holding next response http://bugs.python.org/issue23377 #23368: integer overflow in _PyUnicode_AsKind http://bugs.python.org/issue23368 #23367: integer overflow in unicodedata.normalize http://bugs.python.org/issue23367 #23354: Loading 2 GiLOC file which raises exception causes wrong trace http://bugs.python.org/issue23354 #23331: Add non-interactive version of Bdb.runcall http://bugs.python.org/issue23331 #23330: h2py.py regular expression missing http://bugs.python.org/issue23330 #23325: Turn SIG_DFL and SIG_IGN into functions http://bugs.python.org/issue23325 #23319: Missing SWAP_INT??in I_set_sw http://bugs.python.org/issue23319 #23314: Disabling CRT asserts in debug build http://bugs.python.org/issue23314 Most recent 15 issues waiting for review (15) ============================================= #23401: Add pickle support of Mapping views http://bugs.python.org/issue23401 #23387: test_urllib2 fails with HTTP Error 502: Bad Gateway http://bugs.python.org/issue23387 #23383: Clean up bytes formatting http://bugs.python.org/issue23383 #23377: HTTPResponse may drop buffer holding next response http://bugs.python.org/issue23377 #23376: getargs.c: redundant C-contiguity check http://bugs.python.org/issue23376 #23374: pydoc 3.x raises UnicodeEncodeError on sqlite3 package http://bugs.python.org/issue23374 #23360: Content-Type when sending data with urlopen() http://bugs.python.org/issue23360 #23359: Speed-up set_lookkey() http://bugs.python.org/issue23359 #23357: pyvenv help shows incorrect usage http://bugs.python.org/issue23357 #23356: In argparse docs simplify example about argline http://bugs.python.org/issue23356 #23353: generator bug with exception: tstate->exc_value is not cleared http://bugs.python.org/issue23353 #23350: Content-length is incorrect when request body is a list or tup http://bugs.python.org/issue23350 #23344: Faster marshalling http://bugs.python.org/issue23344 #23342: run() - unified high-level interface for subprocess http://bugs.python.org/issue23342 #23338: PyErr_Format in ctypes uses invalid parameter http://bugs.python.org/issue23338 Top 10 most discussed issues (10) ================================= #23353: generator bug with exception: tstate->exc_value is not cleared http://bugs.python.org/issue23353 18 msgs #23376: getargs.c: redundant C-contiguity check http://bugs.python.org/issue23376 12 msgs #18932: Optimize selectors.EpollSelector.modify() http://bugs.python.org/issue18932 11 msgs #23357: pyvenv help shows incorrect usage http://bugs.python.org/issue23357 10 msgs #23359: Speed-up set_lookkey() http://bugs.python.org/issue23359 10 msgs #17911: traceback: add a new thin class storing a traceback without st http://bugs.python.org/issue17911 9 msgs #2292: Missing *-unpacking generalizations http://bugs.python.org/issue2292 7 msgs #6634: sys.exit() called from threads other than the main one: undocu http://bugs.python.org/issue6634 7 msgs #23374: pydoc 3.x raises UnicodeEncodeError on sqlite3 package http://bugs.python.org/issue23374 7 msgs #12916: Add inspect.splitdoc http://bugs.python.org/issue12916 6 msgs Issues closed (47) ================== #8534: multiprocessing not working from egg http://bugs.python.org/issue8534 closed by davin #11152: pb in zipfile module http://bugs.python.org/issue11152 closed by serhiy.storchaka #13128: httplib debuglevel on CONNECT doesn't print response headers http://bugs.python.org/issue13128 closed by berker.peksag #14203: PEP-3118: remove obsolete write-locks http://bugs.python.org/issue14203 closed by skrah #17189: Add zip64 support to shutil http://bugs.python.org/issue17189 closed by serhiy.storchaka #18982: Add tests for CLI of the calendar module http://bugs.python.org/issue18982 closed by serhiy.storchaka #20289: Make cgi.FieldStorage a context manager http://bugs.python.org/issue20289 closed by berker.peksag #21295: Python 3.4 gives wrong col_offset for Call nodes returned from http://bugs.python.org/issue21295 closed by python-dev #22079: Ensure in PyType_Ready() that base class of static type is sta http://bugs.python.org/issue22079 closed by serhiy.storchaka #22445: Memoryviews require more strict contiguous checks then necessa http://bugs.python.org/issue22445 closed by skrah #22765: Fixes for test_gdb (first frame address, entry values) http://bugs.python.org/issue22765 closed by serhiy.storchaka #22818: Deprecate splitting on possible zero-width re patterns http://bugs.python.org/issue22818 closed by serhiy.storchaka #22896: Don't use PyObject_As*Buffer() functions http://bugs.python.org/issue22896 closed by serhiy.storchaka #22904: make profile-opt includes -fprofile* flags in _sysconfigdata C http://bugs.python.org/issue22904 closed by gregory.p.smith #22986: Improved handling of __class__ assignment http://bugs.python.org/issue22986 closed by python-dev #23099: BytesIO and StringIO values unavailable when closed http://bugs.python.org/issue23099 closed by serhiy.storchaka #23212: Update Windows and OS X installer copies of OpenSSL to 1.0.1l http://bugs.python.org/issue23212 closed by ned.deily #23258: Cannot Install Python 3.4.2 on Windows 7 64 bit / screen print http://bugs.python.org/issue23258 closed by steve.dower #23260: Update Windows installer http://bugs.python.org/issue23260 closed by steve.dower #23308: a bug in Instructions section 4.1 http://bugs.python.org/issue23308 closed by zach.ware #23321: Crash in str.decode() with special error handler http://bugs.python.org/issue23321 closed by serhiy.storchaka #23326: Remove redundant __ne__ implementations http://bugs.python.org/issue23326 closed by serhiy.storchaka #23345: test_ssl fails on OS X 10.10.2 with latest patch level of Open http://bugs.python.org/issue23345 closed by ned.deily #23348: distutils.LooseVersion fails to compare two valid versions http://bugs.python.org/issue23348 closed by eric.araujo #23349: PyBuffer_ToContiguous() off-by-one error for non-contiguous bu http://bugs.python.org/issue23349 closed by skrah #23351: socket.settimeout(5.0) does not have any effect http://bugs.python.org/issue23351 closed by neologix #23352: Document "suboffsets if needed" in 2.7 http://bugs.python.org/issue23352 closed by skrah #23355: rsplit duplicates split behavior http://bugs.python.org/issue23355 closed by zach.ware #23358: BaseServer missing from socketserver.__all__ http://bugs.python.org/issue23358 closed by berker.peksag #23363: integer overflow in itertools.permutations http://bugs.python.org/issue23363 closed by python-dev #23364: integer overflow in itertools.product http://bugs.python.org/issue23364 closed by python-dev #23365: integer overflow in itertools.combinations_with_replacement http://bugs.python.org/issue23365 closed by python-dev #23366: integer overflow in itertools.combinations http://bugs.python.org/issue23366 closed by python-dev #23369: integer overflow in _json.encode_basestring_ascii http://bugs.python.org/issue23369 closed by python-dev #23370: PyBuffer_FromContiguous() off-by-one error for non-contiguous http://bugs.python.org/issue23370 closed by skrah #23373: curses.textpad crashes in insert mode on wide window http://bugs.python.org/issue23373 closed by berker.peksag #23379: Incorrect links within PEPs http://bugs.python.org/issue23379 closed by berker.peksag #23380: Python 2.7.9 test_gdb fails on Fedora 21 http://bugs.python.org/issue23380 closed by vlee #23381: Python 2.7.9+ test_gdb regression on Ubuntu 10.04 http://bugs.python.org/issue23381 closed by serhiy.storchaka #23385: snmp - mib error http://bugs.python.org/issue23385 closed by berker.peksag #23386: snmp - mib error http://bugs.python.org/issue23386 closed by eric.smith #23390: make profile-opt: test_distutils failure http://bugs.python.org/issue23390 closed by gregory.p.smith #23392: Add tests for marshal FILE* API http://bugs.python.org/issue23392 closed by serhiy.storchaka #23393: [Windows] Unable to link with Python in debug configuration http://bugs.python.org/issue23393 closed by sgallou #23396: Wrong print for 2.7.9 http://bugs.python.org/issue23396 closed by python-dev #23398: calendar.monthrange for February 2015 http://bugs.python.org/issue23398 closed by ethan.furman #23399: venv should create relative symlinks where possible http://bugs.python.org/issue23399 closed by barry From francis.giraldeau at gmail.com Fri Feb 6 23:48:35 2015 From: francis.giraldeau at gmail.com (Francis Giraldeau) Date: Fri, 6 Feb 2015 17:48:35 -0500 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: Message-ID: 2015-02-06 6:04 GMT-05:00 Armin Rigo : > Hi, > > On 6 February 2015 at 08:24, Maciej Fijalkowski wrote: > > I don't think it's safe to assume f_code is properly filled by the > > time you might read it, depending a bit where you find the frame > > object. Are you sure it's not full of garbage? > Yes, before discussing how to do the utf8 decoding, we should realize > that it is really unsafe code starting from the line before. From a > signal handler you're only supposed to read data that was written to > "volatile" fields. So even PyEval_GetFrame(), which is done by > reading the thread state's "frame" field, is not safe: this is not a > volatile. This means that the compiler is free to do crazy things > like *first* write into this field and *then* initialize the actual > content of the frame. The uninitialized content may be garbage, not > just NULLs. > Thanks for these comments. Of course accessing frames withing a signal handler is racy. I confirm that code encoded in non-ascii is not accessible from the uft8 buffer pointer. However, a call to PyUnicode_AsUTF8() encodes the data and caches it in the unicode object. Later access returns the byte buffer without memory allocation and re-encoding. I think it is possible to solve both safety problems by registering a handler with PyPyEval_SetProfile(). On function entry, the handler will call PyUnicode_AsUTF8() on the required frame members to make sure the utf8 encoded string is available. Then, we increment the refcount of the frame and assign it to a thread local pointer. On function return, the refcount is decremented. These operations occurs in the normal context and they are not racy. The signal handler will use the thread local frame pointer instead of calling PyEval_GetFrame(). Does that sounds good? Thanks again for your feedback! Francis -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Sat Feb 7 06:39:22 2015 From: lac at openend.se (Laura Creighton) Date: Sat, 7 Feb 2015 06:39:22 +0100 Subject: [Python-Dev] installing python 2.7.9 on a Mac Message-ID: <201502070539.t175dMcj003003@fido.openend.se> webmaster just got mail from a novice who is trying to learn Python in an introductory class. She got a "The version of Tcl/Tk (8.5.7) in use may be unstable" message. I think that the download page should have a link. If you get download and install . Any reason we cannot do that? Laura From fijall at gmail.com Sat Feb 7 09:45:34 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 7 Feb 2015 10:45:34 +0200 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: Message-ID: On Sat, Feb 7, 2015 at 12:48 AM, Francis Giraldeau wrote: > 2015-02-06 6:04 GMT-05:00 Armin Rigo : > >> Hi, >> >> On 6 February 2015 at 08:24, Maciej Fijalkowski wrote: >> > I don't think it's safe to assume f_code is properly filled by the >> > time you might read it, depending a bit where you find the frame >> > object. Are you sure it's not full of garbage? >> >> >> Yes, before discussing how to do the utf8 decoding, we should realize >> that it is really unsafe code starting from the line before. From a >> signal handler you're only supposed to read data that was written to >> "volatile" fields. So even PyEval_GetFrame(), which is done by >> reading the thread state's "frame" field, is not safe: this is not a >> volatile. This means that the compiler is free to do crazy things >> like *first* write into this field and *then* initialize the actual >> content of the frame. The uninitialized content may be garbage, not >> just NULLs. > > > Thanks for these comments. Of course accessing frames withing a signal > handler is racy. I confirm that code encoded in non-ascii is not accessible > from the uft8 buffer pointer. However, a call to PyUnicode_AsUTF8() encodes > the data and caches it in the unicode object. Later access returns the byte > buffer without memory allocation and re-encoding. > > I think it is possible to solve both safety problems by registering a > handler with PyPyEval_SetProfile(). On function entry, the handler will call > PyUnicode_AsUTF8() on the required frame members to make sure the utf8 > encoded string is available. Then, we increment the refcount of the frame > and assign it to a thread local pointer. On function return, the refcount is > decremented. These operations occurs in the normal context and they are not > racy. The signal handler will use the thread local frame pointer instead of > calling PyEval_GetFrame(). Does that sounds good? > > Thanks again for your feedback! > > Francis You still didn't explain what are you trying to achieve nor adressed armins questions about volatile. However, you can't access thread locals from signal handlers (since in some cases it mallocs, thread locals are built lazily if you're inside the .so, e.g. if python is built with --shared) From xdegaye at gmail.com Sat Feb 7 11:13:00 2015 From: xdegaye at gmail.com (Xavier de Gaye) Date: Sat, 07 Feb 2015 11:13:00 +0100 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: Message-ID: <54D5E52C.3010700@gmail.com> On 02/06/2015 11:48 PM, Francis Giraldeau wrote: > 2015-02-06 6:04 GMT-05:00 Armin Rigo: > > Hi, > > On 6 February 2015 at 08:24, Maciej Fijalkowski > wrote: > > I don't think it's safe to assume f_code is properly filled by the > > time you might read it, depending a bit where you find the frame > > object. Are you sure it's not full of garbage? > > > Yes, before discussing how to do the utf8 decoding, we should realize > that it is really unsafe code starting from the line before. From a > signal handler you're only supposed to read data that was written to > "volatile" fields. So even PyEval_GetFrame(), which is done by > reading the thread state's "frame" field, is not safe: this is not a > volatile. This means that the compiler is free to do crazy things > like *first* write into this field and *then* initialize the actual > content of the frame. The uninitialized content may be garbage, not > just NULLs. > > > Thanks for these comments. Of course accessing frames withing a signal handler is racy. I confirm that code encoded in non-ascii is not accessible from the uft8 buffer pointer. However, a call > to PyUnicode_AsUTF8() encodes the data and caches it in the unicode object. Later access returns the byte buffer without memory allocation and re-encoding. > > I think it is possible to solve both safety problems by registering a handler with PyPyEval_SetProfile(). On function entry, the handler will call PyUnicode_AsUTF8() on the required frame members to > make sure the utf8 encoded string is available. Then, we increment the refcount of the frame and assign it to a thread local pointer. On function return, the refcount is decremented. These operations > occurs in the normal context and they are not racy. The signal handler will use the thread local frame pointer instead of calling PyEval_GetFrame(). Does that sounds good? You could call Py_AddPendingCall() from your signal handler and access the frame members from the function scheduled by Py_AddPendingCall(). Xavier From greg.ewing at canterbury.ac.nz Sat Feb 7 22:34:03 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 08 Feb 2015 10:34:03 +1300 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: Message-ID: <54D684CB.2050101@canterbury.ac.nz> Maciej Fijalkowski wrote: > However, you can't access thread > locals from signal handlers (since in some cases it mallocs, thread > locals are built lazily if you're inside the .so, e.g. if python is > built with --shared) You might be able to use Py_AddPendingCall to schedule what you want done outside the context of the signal handler. The call will be made by the main thread, though, so if you need to access the frame of whatever thread was running when the signal occured, you will have to track down its PyThreadState somehow and get the frame from there. Not sure what would be involved in doing that. -- Greg From nad at acm.org Sun Feb 8 00:53:39 2015 From: nad at acm.org (Ned Deily) Date: Sat, 07 Feb 2015 15:53:39 -0800 Subject: [Python-Dev] installing python 2.7.9 on a Mac References: <201502070539.t175dMcj003003@fido.openend.se> Message-ID: In article <201502070539.t175dMcj003003 at fido.openend.se>, Laura Creighton wrote: > webmaster just got mail from a novice who is trying to learn Python in > an introductory class. She got a "The version of Tcl/Tk (8.5.7) in > use may be unstable" message. > > I think that the download page should have a link. > If you get > download and install . Any reason we cannot do that? The truncated "use may be unstable" message above is displayed when you run IDLE. The full message displayed already contains a link to the python.org web page that explains the most up-to-date recommendations for Tcl/Tk use with python.org installers on OS X and includes links to download options. That same link is included in the Welcome message and README text when running the installer for Python. It is also already linked to on the download pages for each release. Until we start shipping our own version of Tcl/Tk for OS X (which has been in the cards), I don't think we can make it much easier. -- Ned Deily, nad at acm.org From greg at krypto.org Sun Feb 8 05:39:00 2015 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 08 Feb 2015 04:39:00 +0000 Subject: [Python-Dev] Encoding of PyFrameObject members References: <54D684CB.2050101@canterbury.ac.nz> Message-ID: To get at the Python thread state from a signal handler (using 2.7 as a reference here; but i don't believe 3.4 has changed this part much) you need to modify the interpreter to expose pystate.c's "autoTLSkey" and thread.c's "struct key" as well as "keyhead" and "keymutex". >From there, in your signal handler you must try to acquire the newly exposed keymutex and do nothing if you were unable to get it. If you did acquire it (rare not to), you can walk the keyhead list looking for autoTLSkey to find the current valid thread state. I had an intern (hi Elena!) write a signal sampling based low overhead Python CPU profiler based on that last summer. I believe there are still bugs to shaken out (if they are even possible to fix... Armin's comments are true: signal handler code is super limited). I am stating this here because I want someone to pester me at PyCon if I haven't released our work as a proof of concept by then. The important take away: From what I could figure out, you need to modify the CPython interpreter to be more amenable to such introspection. A downside of a signal based profiler: *ALL* of the EINTR mishandling bugs within the Python interpreter, stdlib, and your own code will show up in your application. So until those are fixed (hooray for Antoine's PEP!), it may not be practical for use on production processes which is sort of the entire point of a low overhead sampling profiler... I'd like to get a buildbot setup that runs the testsuite while a continual barrage of signals are being generated. We really don't stress test that stuff (as evidence by the EINTR mishandling issues that are rampant) as non-fatal signals are so rare for most things... until they aren't. As a side note and encouragement: I wonder what PyPy could do for dynamically enabled and disabled low overhead CPU profiling. (take that as a hint that I want someone else to get extremely creative!) -gps On Sat Feb 07 2015 at 1:34:26 PM Greg Ewing wrote: > Maciej Fijalkowski wrote: > > However, you can't access thread > > locals from signal handlers (since in some cases it mallocs, thread > > locals are built lazily if you're inside the .so, e.g. if python is > > built with --shared) > > You might be able to use Py_AddPendingCall to schedule > what you want done outside the context of the signal > handler. > > The call will be made by the main thread, though, > so if you need to access the frame of whatever thread > was running when the signal occured, you will have > to track down its PyThreadState somehow and get the > frame from there. Not sure what would be involved > in doing that. > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sun Feb 8 09:15:43 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 8 Feb 2015 09:15:43 +0100 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: <54D684CB.2050101@canterbury.ac.nz> References: <54D684CB.2050101@canterbury.ac.nz> Message-ID: Le 7 f?vr. 2015 22:34, "Greg Ewing" a ?crit : with --shared) > You might be able to use Py_AddPendingCall to schedule > what you want done outside the context of the signal > handler. I don't how it could work. You have to increment the reference counting, but also maybe increment references to other frames. Again there is no guarantee that python structures are consistent in a signal handler. While a faulthandler is only called once, a profiler is called very frequently, up to once per python instruction. Unlikely bugs become very likely. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sun Feb 8 09:21:36 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 8 Feb 2015 09:21:36 +0100 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: <54D684CB.2050101@canterbury.ac.nz> Message-ID: Le 8 f?vr. 2015 05:39, "Gregory P. Smith" a ?crit : > From there, in your signal handler you must try to acquire the newly exposed keymutex and (...) Stop! Acquiring a lock in a signal handler is just a crazy idea. It's very far from an async signal-safe function. > So until those are fixed (hooray for Antoine's PEP!), ... I wrote the PEP with Charles Fran?ois Natali, but Charles wrote the whole implementation. Antoine also helped us and approved the PEP. It's the french connection! Just to say that Charles did most of the work. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Sun Feb 8 10:01:30 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 8 Feb 2015 11:01:30 +0200 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: <54D684CB.2050101@canterbury.ac.nz> Message-ID: I'm working on vmprof (github.com/vmprof/vmprof-python) which works for both cpython and pypy (pypy has special support, cpython is patched on-the fly) On Sun, Feb 8, 2015 at 6:39 AM, Gregory P. Smith wrote: > To get at the Python thread state from a signal handler (using 2.7 as a > reference here; but i don't believe 3.4 has changed this part much) you need > to modify the interpreter to expose pystate.c's "autoTLSkey" and thread.c's > "struct key" as well as "keyhead" and "keymutex". > > From there, in your signal handler you must try to acquire the newly exposed > keymutex and do nothing if you were unable to get it. If you did acquire it > (rare not to), you can walk the keyhead list looking for autoTLSkey to find > the current valid thread state. > > I had an intern (hi Elena!) write a signal sampling based low overhead > Python CPU profiler based on that last summer. I believe there are still > bugs to shaken out (if they are even possible to fix... Armin's comments are > true: signal handler code is super limited). I am stating this here because > I want someone to pester me at PyCon if I haven't released our work as a > proof of concept by then. The important take away: From what I could figure > out, you need to modify the CPython interpreter to be more amenable to such > introspection. > > A downside of a signal based profiler: *ALL* of the EINTR mishandling bugs > within the Python interpreter, stdlib, and your own code will show up in > your application. So until those are fixed (hooray for Antoine's PEP!), it > may not be practical for use on production processes which is sort of the > entire point of a low overhead sampling profiler... > > I'd like to get a buildbot setup that runs the testsuite while a continual > barrage of signals are being generated. We really don't stress test that > stuff (as evidence by the EINTR mishandling issues that are rampant) as > non-fatal signals are so rare for most things... until they aren't. > > As a side note and encouragement: I wonder what PyPy could do for > dynamically enabled and disabled low overhead CPU profiling. (take that as a > hint that I want someone else to get extremely creative!) > > -gps > > On Sat Feb 07 2015 at 1:34:26 PM Greg Ewing > wrote: >> >> Maciej Fijalkowski wrote: >> > However, you can't access thread >> > locals from signal handlers (since in some cases it mallocs, thread >> > locals are built lazily if you're inside the .so, e.g. if python is >> > built with --shared) >> >> You might be able to use Py_AddPendingCall to schedule >> what you want done outside the context of the signal >> handler. >> >> The call will be made by the main thread, though, >> so if you need to access the frame of whatever thread >> was running when the signal occured, you will have >> to track down its PyThreadState somehow and get the >> frame from there. Not sure what would be involved >> in doing that. >> >> -- >> Greg >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > From fijall at gmail.com Sun Feb 8 21:11:53 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 8 Feb 2015 22:11:53 +0200 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: <54D684CB.2050101@canterbury.ac.nz> Message-ID: Hi Francis Feel free to steal most of vmprof code, it should generally work without requiring to patch cpython (python 3 patches appreciated :-). As far as timer goes - it seems not to be going anywhere, I would rather use a background thread or something On Sun, Feb 8, 2015 at 10:03 PM, Francis Giraldeau wrote: > 2015-02-08 4:01 GMT-05:00 Maciej Fijalkowski : >> >> I'm working on vmprof (github.com/vmprof/vmprof-python) which works >> for both cpython and pypy (pypy has special support, cpython is >> patched on-the fly) > > > This looks interesting. I'm working on a profiler that is similar, but not > based on timer. Instead, the signal is generated when an hardware > performance counter overflows. It required a special linux kernel module, > and the tracepoint is recorded using LTTng-UST. > > https://github.com/giraldeau/perfuser > https://github.com/giraldeau/perfuser-modules > https://github.com/giraldeau/python-profile-ust > > This is of course very experimental, requires a special setup, an I don't > even know if it's going to produce good results. I'll report the results in > the coming weeks. > > Cheers, > > Francis Giraldeau From francis.giraldeau at gmail.com Sun Feb 8 21:03:33 2015 From: francis.giraldeau at gmail.com (Francis Giraldeau) Date: Sun, 8 Feb 2015 15:03:33 -0500 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: <54D684CB.2050101@canterbury.ac.nz> Message-ID: 2015-02-08 4:01 GMT-05:00 Maciej Fijalkowski : > I'm working on vmprof (github.com/vmprof/vmprof-python) which works > for both cpython and pypy (pypy has special support, cpython is > patched on-the fly) This looks interesting. I'm working on a profiler that is similar, but not based on timer. Instead, the signal is generated when an hardware performance counter overflows. It required a special linux kernel module, and the tracepoint is recorded using LTTng-UST. https://github.com/giraldeau/perfuser https://github.com/giraldeau/perfuser-modules https://github.com/giraldeau/python-profile-ust This is of course very experimental, requires a special setup, an I don't even know if it's going to produce good results. I'll report the results in the coming weeks. Cheers, Francis Giraldeau -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Feb 8 23:00:33 2015 From: larry at hastings.org (Larry Hastings) Date: Sun, 08 Feb 2015 14:00:33 -0800 Subject: [Python-Dev] [RELEASE] Python 3.4.3rc1 is now available Message-ID: <54D7DC81.4000105@hastings.org> On behalf of the Python development community and the Python 3.4 release team, I'm happy to announce the availability of Python 3.4.3rc1. Python 3.4.3rc1 has many bugfixes and other small improvements over 3.4.2. You can download it here: https://www.python.org/download/releases/3.4.3 Not done yet, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Feb 8 23:03:05 2015 From: larry at hastings.org (Larry Hastings) Date: Sun, 08 Feb 2015 14:03:05 -0800 Subject: [Python-Dev] [RELEASE] Python 3.5.0a1 is now available Message-ID: <54D7DD19.5000400@hastings.org> On behalf of the Python development community and the Python 3.5 release team, I'm also pleased to announce the availability of Python 3.5.0a1. Python 3.5.0a1 is the first alpha release of Python 3.5, which will be the next major release of Python. Python 3.5 is still under heavy development, and is far from complete. This is a preview release, and its use is not recommended for production settings. You can download it here: https://www.python.org/download/releases/3.5.0a1 Happy hacking, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Mon Feb 9 01:36:38 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 9 Feb 2015 13:36:38 +1300 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: <54D684CB.2050101@canterbury.ac.nz> Message-ID: On 9 February 2015 at 09:11, Maciej Fijalkowski wrote: > Hi Francis > > Feel free to steal most of vmprof code, it should generally work > without requiring to patch cpython (python 3 patches appreciated :-). > As far as timer goes - it seems not to be going anywhere, I would > rather use a background thread or something What about setting a flag when the signal arrives and checking it at the next bytecode evaluation or something? -Rob From victor.stinner at gmail.com Mon Feb 9 07:35:46 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 9 Feb 2015 07:35:46 +0100 Subject: [Python-Dev] Encoding of PyFrameObject members In-Reply-To: References: <54D684CB.2050101@canterbury.ac.nz> Message-ID: This is exactly how signals are implemented in Python :-) Victor Le lundi 9 f?vrier 2015, Robert Collins a ?crit : > On 9 February 2015 at 09:11, Maciej Fijalkowski > wrote: > > Hi Francis > > > > Feel free to steal most of vmprof code, it should generally work > > without requiring to patch cpython (python 3 patches appreciated :-). > > As far as timer goes - it seems not to be going anywhere, I would > > rather use a background thread or something > > What about setting a flag when the signal arrives and checking it at > the next bytecode evaluation or something? > > -Rob > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Mon Feb 9 08:16:12 2015 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 09 Feb 2015 07:16:12 +0000 Subject: [Python-Dev] Encoding of PyFrameObject members References: <54D684CB.2050101@canterbury.ac.nz> Message-ID: On Sun Feb 08 2015 at 12:21:36 AM Victor Stinner wrote: > > Le 8 f?vr. 2015 05:39, "Gregory P. Smith" a ?crit : > > From there, in your signal handler you must try to acquire the newly > exposed keymutex and (...) > > Stop! Acquiring a lock in a signal handler is just a crazy idea. It's very > far from an async signal-safe function. > I'd normally agree... In this case, sem_trywait() is non-blocking. It is unfortunately not on the official POSIX list of async-signal-safe functions (of the SEMs, only sem_post() is) but many implementations actually are safe, documented or not. https://github.com/lattera/glibc/blob/master/nptl/sysdeps/unix/sysv/linux/x86_64/sem_trywait.S - appears safe. So long as the underlying semaphore is implemented using an atomic instruction on the target platform I wouldn't expect any implementation of sem_trywait to be async signal unsafe. (nor would I depend on that without checking) -gps > > So until those are fixed (hooray for Antoine's PEP!), ... > > I wrote the PEP with Charles Fran?ois Natali, but Charles wrote the whole > implementation. Antoine also helped us and approved the PEP. It's the > french connection! Just to say that Charles did most of the work. > > Victor > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Feb 9 21:27:36 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 09 Feb 2015 22:27:36 +0200 Subject: [Python-Dev] cpython: Mirco-optimizations to reduce register spills and reloads observed on CLANG and In-Reply-To: <20150209124834.50335.39056@psf.io> References: <20150209124834.50335.39056@psf.io> Message-ID: On 09.02.15 14:48, raymond.hettinger wrote: > https://hg.python.org/cpython/rev/dc820b44ce21 > changeset: 94572:dc820b44ce21 > user: Raymond Hettinger > date: Mon Feb 09 06:48:29 2015 -0600 > summary: > Mirco-optimizations to reduce register spills and reloads observed on CLANG and GCC. > > files: > Objects/setobject.c | 6 ++++-- > 1 files changed, 4 insertions(+), 2 deletions(-) > > > diff --git a/Objects/setobject.c b/Objects/setobject.c > --- a/Objects/setobject.c > +++ b/Objects/setobject.c > @@ -84,8 +84,9 @@ > return set_lookkey(so, key, hash); > if (cmp > 0) /* likely */ > return entry; > + mask = so->mask; /* help avoid a register spill */ Could you please explain in more details what this line do? The mask variable is actually constant and so->mask isn't changed in this loop. From mistersheik at gmail.com Mon Feb 9 22:06:20 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Feb 2015 16:06:20 -0500 Subject: [Python-Dev] (no subject) Message-ID: Hello all, The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is implemented now based on some early work by Thomas Wouters (in 2008) and Florian Hahn (2013) and recently completed by Joshua Landau and me. The issue tracker http://bugs.python.org/issue2292 has a working patch. Would someone be able to review it? Thank you very much, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Feb 9 22:28:13 2015 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 09 Feb 2015 16:28:13 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: Message-ID: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> On Mon, Feb 9, 2015, at 16:06, Neil Girdhar wrote: > Hello all, > > The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is > implemented now based on some early work by Thomas Wouters (in 2008) and > Florian Hahn (2013) and recently completed by Joshua Landau and me. > > The issue tracker http://bugs.python.org/issue2292 has a working patch. > Would someone be able to review it? The PEP is not even accepted. From guido at python.org Mon Feb 9 22:32:25 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Feb 2015 13:32:25 -0800 Subject: [Python-Dev] (no subject) In-Reply-To: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> Message-ID: FWIW, I've encouraged Neil and others to complete this code as a prerequisite for a code review (but I can't review it myself). I am mildly in favor of the PEP -- if the code works and looks maintainable I would accept it. (A few things got changed in the PEP as a result of the work.) On Mon, Feb 9, 2015 at 1:28 PM, Benjamin Peterson wrote: > > > On Mon, Feb 9, 2015, at 16:06, Neil Girdhar wrote: > > Hello all, > > > > The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is > > implemented now based on some early work by Thomas Wouters (in 2008) and > > Florian Hahn (2013) and recently completed by Joshua Landau and me. > > > > The issue tracker http://bugs.python.org/issue2292 has a working > patch. > > Would someone be able to review it? > > The PEP is not even accepted. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Feb 9 22:34:41 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 09 Feb 2015 13:34:41 -0800 Subject: [Python-Dev] (no subject) In-Reply-To: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> Message-ID: <54D927F1.4090708@stoneleaf.us> On 02/09/2015 01:28 PM, Benjamin Peterson wrote: > On Mon, Feb 9, 2015, at 16:06, Neil Girdhar wrote: >> >> The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is >> implemented now based on some early work by Thomas Wouters (in 2008) and >> Florian Hahn (2013) and recently completed by Joshua Landau and me. >> >> The issue tracker http://bugs.python.org/issue2292 has a working patch. >> Would someone be able to review it? > > The PEP is not even accepted. I believe somebody (Guido?) commented "Why worry about accepting the PEP when there's no working patch?" -- or something to that effect. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From benjamin at python.org Mon Feb 9 23:10:10 2015 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 09 Feb 2015 17:10:10 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> Message-ID: <1423519810.3340207.225326489.5FC19250@webmail.messagingengine.com> On Mon, Feb 9, 2015, at 16:32, Guido van Rossum wrote: > FWIW, I've encouraged Neil and others to complete this code as a > prerequisite for a code review (but I can't review it myself). I am > mildly > in favor of the PEP -- if the code works and looks maintainable I would > accept it. (A few things got changed in the PEP as a result of the work.) In a way, it's a simplification, since functions are now simply called with a sequence of "generalized arguments"; there's no privileged kwarg or vararg. Of course, I wonder how much of f(**w, x, y, *k, *b, **d, c) we would see... From benjamin at python.org Mon Feb 9 23:10:58 2015 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 09 Feb 2015 17:10:58 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: <54D927F1.4090708@stoneleaf.us> References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <54D927F1.4090708@stoneleaf.us> Message-ID: <1423519858.3340452.225336093.38A61898@webmail.messagingengine.com> On Mon, Feb 9, 2015, at 16:34, Ethan Furman wrote: > On 02/09/2015 01:28 PM, Benjamin Peterson wrote: > > On Mon, Feb 9, 2015, at 16:06, Neil Girdhar wrote: > >> > >> The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is > >> implemented now based on some early work by Thomas Wouters (in 2008) and > >> Florian Hahn (2013) and recently completed by Joshua Landau and me. > >> > >> The issue tracker http://bugs.python.org/issue2292 has a working patch. > >> Would someone be able to review it? > > > > The PEP is not even accepted. > > I believe somebody (Guido?) commented "Why worry about accepting the PEP > when there's no working patch?" -- or > something to that effect. On the other hand, I'd rather not do detailed reviews of patches that won't be accepted. :) From ncoghlan at gmail.com Mon Feb 9 23:43:53 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Feb 2015 08:43:53 +1000 Subject: [Python-Dev] (no subject) In-Reply-To: <1423519858.3340452.225336093.38A61898@webmail.messagingengine.com> References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <54D927F1.4090708@stoneleaf.us> <1423519858.3340452.225336093.38A61898@webmail.messagingengine.com> Message-ID: On 10 Feb 2015 08:13, "Benjamin Peterson" wrote: > > > > On Mon, Feb 9, 2015, at 16:34, Ethan Furman wrote: > > On 02/09/2015 01:28 PM, Benjamin Peterson wrote: > > > On Mon, Feb 9, 2015, at 16:06, Neil Girdhar wrote: > > >> > > >> The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is > > >> implemented now based on some early work by Thomas Wouters (in 2008) and > > >> Florian Hahn (2013) and recently completed by Joshua Landau and me. > > >> > > >> The issue tracker http://bugs.python.org/issue2292 has a working patch. > > >> Would someone be able to review it? > > > > > > The PEP is not even accepted. > > > > I believe somebody (Guido?) commented "Why worry about accepting the PEP > > when there's no working patch?" -- or > > something to that effect. > > On the other hand, I'd rather not do detailed reviews of patches that > won't be accepted. :) It's more a matter of the PEP being acceptable in principle, but a reference implementation being needed to confirm feasibility and to iron out corner cases. For example, the potential for arcane call arguments suggests the need for a PEP 8 addition saying "first standalone args, then iterable expansions, then mapping expansions", even though syntactically any order would now be permitted at call time. Cheers, Nick. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Feb 9 23:12:02 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Feb 2015 17:12:02 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: <1423519810.3340207.225326489.5FC19250@webmail.messagingengine.com> References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <1423519810.3340207.225326489.5FC19250@webmail.messagingengine.com> Message-ID: Right, Just to be clear though: **-args must follow any *-args and position arguments. So at worst, your example is: f(x, y, *k, *b, c, **w, **d) Best, Neil On Mon, Feb 9, 2015 at 5:10 PM, Benjamin Peterson wrote: > > > On Mon, Feb 9, 2015, at 16:32, Guido van Rossum wrote: > > FWIW, I've encouraged Neil and others to complete this code as a > > prerequisite for a code review (but I can't review it myself). I am > > mildly > > in favor of the PEP -- if the code works and looks maintainable I would > > accept it. (A few things got changed in the PEP as a result of the work.) > > In a way, it's a simplification, since functions are now simply called > with a sequence of "generalized arguments"; there's no privileged kwarg > or vararg. Of course, I wonder how much of f(**w, x, y, *k, *b, **d, c) > we would see... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Feb 10 00:14:32 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Feb 2015 00:14:32 +0100 Subject: [Python-Dev] (no subject) References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <54D927F1.4090708@stoneleaf.us> <1423519858.3340452.225336093.38A61898@webmail.messagingengine.com> Message-ID: <20150210001432.0be7747f@fsol> On Tue, 10 Feb 2015 08:43:53 +1000 Nick Coghlan wrote: > > For example, the potential for arcane call arguments suggests the need for > a PEP 8 addition saying "first standalone args, then iterable expansions, > then mapping expansions", even though syntactically any order would now be > permitted at call time. There are other concerns: - inspect.signature() must be updated to cover the new call possibilities - function call performance must not be crippled by the new possibilities Regards Antoine. From benjamin at python.org Tue Feb 10 00:16:33 2015 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 09 Feb 2015 18:16:33 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <1423519810.3340207.225326489.5FC19250@webmail.messagingengine.com> Message-ID: <1423523793.3352441.225361169.3C140AE4@webmail.messagingengine.com> On Mon, Feb 9, 2015, at 17:12, Neil Girdhar wrote: > Right, > > Just to be clear though: **-args must follow any *-args and position > arguments. So at worst, your example is: > > f(x, y, *k, *b, c, **w, **d) > > Best, Ah, I guess I was confused by this sentence in the PEP: " Function calls currently have the restriction that keyword arguments must follow positional arguments and ** unpackings must additionally follow * unpackings." That suggests that that rule is going to change. From mistersheik at gmail.com Tue Feb 10 00:20:06 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Feb 2015 18:20:06 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: <1423523793.3352441.225361169.3C140AE4@webmail.messagingengine.com> References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <1423519810.3340207.225326489.5FC19250@webmail.messagingengine.com> <1423523793.3352441.225361169.3C140AE4@webmail.messagingengine.com> Message-ID: That wording is my fault. I'll update the PEP to remove the word "currently" after waiting a bit to see if there are any other problems. Best, Neil On Mon, Feb 9, 2015 at 6:16 PM, Benjamin Peterson wrote: > > > On Mon, Feb 9, 2015, at 17:12, Neil Girdhar wrote: > > Right, > > > > Just to be clear though: **-args must follow any *-args and position > > arguments. So at worst, your example is: > > > > f(x, y, *k, *b, c, **w, **d) > > > > Best, > > Ah, I guess I was confused by this sentence in the PEP: " Function calls > currently have the restriction that keyword arguments must follow > positional arguments and ** unpackings must additionally follow * > unpackings." > > That suggests that that rule is going to change. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Feb 10 00:33:55 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 10 Feb 2015 00:33:55 +0100 Subject: [Python-Dev] (no subject) In-Reply-To: References: Message-ID: Hi, 2015-02-09 22:06 GMT+01:00 Neil Girdhar : > The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is > implemented now based on some early work by Thomas Wouters (in 2008) and > Florian Hahn (2013) and recently completed by Joshua Landau and me. I don't like this PEP. IMO it makes the Python syntax more complex and more difficult to read. Extract of the PEP: > Current usage of the * iterable unpacking operator features unnecessary restrictions that can harm readability. Yes, the current syntax is more verbose, but it's simpler to understand and simpler to debug. -- Example: >>> ranges = [range(i) for i in range(5)] >>> [*item for item in ranges] [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] I don't understand this code. It looks like you forgot something before *item, I would expect 2*item for example. If it's really to unpack something, I still don't understand the syntax. Does "*" apply to item or to the whole "item for item in ranges"? It's not clear to me. If it applies to the whole generator, the syntax is really strange and I would expect parenthesis: [*(item for item in ranges)]. -- > function(**kw_arguments, **more_arguments) If the key "key1" is in both dictionaries, more_arguments wins, right? I never suffered of the lack of the PEP 448. But I remember that a friend learning Python asked me that * and ** are limited to functions. I had no answer. The answer is maybe to keep the language simple? :-) I should maybe read the PEP one more time and think about it. Victor From solipsis at pitrou.net Tue Feb 10 00:51:39 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Feb 2015 00:51:39 +0100 Subject: [Python-Dev] (no subject) PEP 448 References: Message-ID: <20150210005139.18eb7d0e@fsol> On Mon, 9 Feb 2015 16:06:20 -0500 Neil Girdhar wrote: > Hello all, > > The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is > implemented now based on some early work by Thomas Wouters (in 2008) and > Florian Hahn (2013) and recently completed by Joshua Landau and me. To be clear, the PEP will probably be useful for one single line of Python code every 10000. This is a very weak case for such an intrusive syntax addition. I would support the PEP if it only added the simple cases of tuple unpacking, left alone function call conventions, and didn't introduce **-unpacking. Barring that, I really don't want to review the patch and I'm a rather decided -1 on the current PEP. Regards Antoine. From victor.stinner at gmail.com Tue Feb 10 00:56:54 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 10 Feb 2015 00:56:54 +0100 Subject: [Python-Dev] (no subject) PEP 448 In-Reply-To: <20150210005139.18eb7d0e@fsol> References: <20150210005139.18eb7d0e@fsol> Message-ID: 2015-02-10 0:51 GMT+01:00 Antoine Pitrou : >> The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is >> implemented now based on some early work by Thomas Wouters (in 2008) and >> Florian Hahn (2013) and recently completed by Joshua Landau and me. > > To be clear, the PEP will probably be useful for one single line of > Python code every 10000. @Neil: Can you maybe show us some examples of usage of the PEP 448 in the Python stdlib? I mean find some functions where using the PEP would be useful and show the code before/after (maybe in a code review). It would help to get a better opinion on the PEP. I'm not sure that examples in the PEP are the most revelant. Victor From donald at stufft.io Tue Feb 10 01:12:33 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 9 Feb 2015 19:12:33 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: Message-ID: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> > On Feb 9, 2015, at 4:06 PM, Neil Girdhar wrote: > > Hello all, > > The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/ ) is implemented now based on some early work by Thomas Wouters (in 2008) and Florian Hahn (2013) and recently completed by Joshua Landau and me. > > The issue tracker http://bugs.python.org/issue2292 has a working patch. Would someone be able to review it? > I just skimmed over the PEP and it seems like it?s trying to solve a few different things: * Making it easy to combine multiple lists and additional positional args into a function call * Making it easy to combine multiple dicts and additional keyword args into a functional call * Making it easy to do a single level of nested iterable "flatten". Looking at the syntax in the PEP I had a hard time detangling what exactly it was doing even with reading the PEP itself. I wonder if there isn?t a way to combine simpler more generic things to get the same outcome. Looking at the "Making it easy to combine multiple lists and additional positional args into a function call" aspect of this, why is: print(*[1], *[2], 3) better than print(*[1] + [2] + [3])? That's already doable in Python right now and doesn't require anything new to handle it. Looking at the "making it easy to do a single level of nsted iterable 'flatten'"" aspect of this, the example of: >>> ranges = [range(i) for i in range(5)] >>> [*item for item in ranges] [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] Conceptually a list comprehension like [thing for item in iterable] can be mapped to a for loop like this: result = [] for item in iterable: result.append(thing) However [*item for item in ranges] is mapped more to something like this: result = [] for item in iterable: result.extend(*item) I feel like switching list comprehensions from append to extend just because of a * is really confusing and it acts differently than if you just did *item outside of a list comprehension. I feel like the itertools.chain() way of doing this is *much* clearer. Finally there's the "make it easy to combine multiple dicts into a function call" aspect of this. This I think is the biggest thing that this PEP actually adds, however I think it goes around it the wrong way. Sadly there is nothing like [1] + [2] for dictionaries. The closest thing is: kwargs = dict1.copy() kwargs.update(dict2) func(**kwargs) So what I wonder is if this PEP wouldn't be better off just using the existing methods for doing the kinds of things that I pointed out above, and instead defining + or | or some other symbol for something similar to [1] + [2] but for dictionaries. This would mean that you could simply do: func(**dict1 | dict(y=1) | dict2) instead of dict(**{'x': 1}, y=2, **{'z': 3}) I feel like not only does this genericize way better but it limits the impact and new syntax being added to Python and is a ton more readable. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Feb 10 01:29:44 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Feb 2015 19:29:44 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: For some reason I can't seem to reply using Google groups, which is is telling "this is a read-only mirror" (anyone know why?) Anyway, I'm going to answer as best I can the concerns. Antoine said: To be clear, the PEP will probably be useful for one single line of > Python code every 10000. This is a very weak case for such an intrusive > syntax addition. I would support the PEP if it only added the simple > cases of tuple unpacking, left alone function call conventions, and > didn't introduce **-unpacking. To me this is more of a syntax simplification than a syntax addition. For me the **-unpacking is the most useful part. Regarding utility, it seems that a many of the people on r/python were pretty excited about this PEP: http://www.reddit.com/r/Python/comments/2synry/so_8_peps_are_currently_being_proposed_for_python/ ? Victor noticed that there's a mistake with the code: >>> ranges = [range(i) for i in range(5)] > >>> [*item for item in ranges] > [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] It should be a range(4) in the code. The "*" applies to only item. It is the same as writing: [*range(0), *range(1), *range(2), *range(3), *range(4)] which is the same as unpacking all of those ranges into a list. > function(**kw_arguments, **more_arguments) > If the key "key1" is in both dictionaries, more_arguments wins, right? There was some debate and it was decided that duplicate keyword arguments would remain an error (for now at least). If you want to merge the dictionaries with overriding, then you can still do: function(**{**kw_arguments, **more_arguments}) because **-unpacking in dicts overrides as you guessed. ? On Mon, Feb 9, 2015 at 7:12 PM, Donald Stufft wrote: > > On Feb 9, 2015, at 4:06 PM, Neil Girdhar wrote: > > Hello all, > > The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is > implemented now based on some early work by Thomas Wouters (in 2008) and > Florian Hahn (2013) and recently completed by Joshua Landau and me. > > The issue tracker http://bugs.python.org/issue2292 has a working > patch. Would someone be able to review it? > > > I just skimmed over the PEP and it seems like it?s trying to solve a few > different things: > > * Making it easy to combine multiple lists and additional positional args > into a function call > * Making it easy to combine multiple dicts and additional keyword args > into a functional call > * Making it easy to do a single level of nested iterable "flatten". > I would say it's: * making it easy to unpack iterables and mappings in function calls * making it easy to unpack iterables into list and set displays and comprehensions, and * making it easy to unpack mappings into dict displays and comprehensions. > > Looking at the syntax in the PEP I had a hard time detangling what exactly > it was doing even with reading the PEP itself. I wonder if there isn?t a > way to combine simpler more generic things to get the same outcome. > > Looking at the "Making it easy to combine multiple lists and additional > positional args into a function call" aspect of this, why is: > > print(*[1], *[2], 3) better than print(*[1] + [2] + [3])? > > That's already doable in Python right now and doesn't require anything new > to handle it. > Admittedly, this wasn't a great example. But, if [1] and [2] had been iterables, you would have to cast each to list, e.g., accumulator = [] accumulator.extend(a) accumulator.append(b) accumulator.extend(c) print(*accumulator) replaces print(*a, b, *c) where a and c are iterable. The latter version is also more efficient because it unpacks only a onto the stack allocating no auxilliary list. > Looking at the "making it easy to do a single level of nsted iterable > 'flatten'"" aspect of this, the example of: > > >>> ranges = [range(i) for i in range(5)] > >>> [*item for item in ranges] > [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] > > Conceptually a list comprehension like [thing for item in iterable] can be > mapped to a for loop like this: > > result = [] > for item in iterable: > result.append(thing) > > However [*item for item in ranges] is mapped more to something like this: > > result = [] > for item in iterable: > result.extend(*item) > > I feel like switching list comprehensions from append to extend just > because of a * is really confusing and it acts differently than if you just > did *item outside of a list comprehension. I feel like the > itertools.chain() way of doing this is *much* clearer. > > Finally there's the "make it easy to combine multiple dicts into a > function call" aspect of this. This I think is the biggest thing that this > PEP actually adds, however I think it goes around it the wrong way. Sadly > there is nothing like [1] + [2] for dictionaries. The closest thing is: > > kwargs = dict1.copy() > kwargs.update(dict2) > func(**kwargs) > > So what I wonder is if this PEP wouldn't be better off just using the > existing methods for doing the kinds of things that I pointed out above, > and instead defining + or | or some other symbol for something similar to > [1] + [2] but for dictionaries. This would mean that you could simply do: > > func(**dict1 | dict(y=1) | dict2) > > instead of > > dict(**{'x': 1}, y=2, **{'z': 3}) > > I feel like not only does this genericize way better but it limits the > impact and new syntax being added to Python and is a ton more readable. > > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Feb 10 01:30:08 2015 From: larry at hastings.org (Larry Hastings) Date: Mon, 09 Feb 2015 16:30:08 -0800 Subject: [Python-Dev] (no subject) In-Reply-To: <20150210001432.0be7747f@fsol> References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <54D927F1.4090708@stoneleaf.us> <1423519858.3340452.225336093.38A61898@webmail.messagingengine.com> <20150210001432.0be7747f@fsol> Message-ID: <54D95110.9090104@hastings.org> What's an example of a way inspect.signature must change? I thought PEP 448 added new unpacking shortcuts which (for example) change the *caller* side of a function call. I didn't realize it impacted the *callee* side too. //arry/ On 02/09/2015 03:14 PM, Antoine Pitrou wrote: > On Tue, 10 Feb 2015 08:43:53 +1000 > Nick Coghlan wrote: >> For example, the potential for arcane call arguments suggests the need for >> a PEP 8 addition saying "first standalone args, then iterable expansions, >> then mapping expansions", even though syntactically any order would now be >> permitted at call time. > There are other concerns: > > - inspect.signature() must be updated to cover the new call > possibilities > > - function call performance must not be crippled by the new > possibilities > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/larry%40hastings.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Feb 10 01:46:13 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Feb 2015 19:46:13 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: <54D95110.9090104@hastings.org> References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <54D927F1.4090708@stoneleaf.us> <1423519858.3340452.225336093.38A61898@webmail.messagingengine.com> <20150210001432.0be7747f@fsol> <54D95110.9090104@hastings.org> Message-ID: Yes, that's exactly right. It does not affect the callee. Regarding function call performance, nothing has changed for the originally accepted argument lists: the opcodes generated are the same and they are processed in the same way. Also, regarding calling argument order, not any order is allowed. Regular arguments must precede other kinds of arguments. Keyword arguments must precede **-args. *-args must precede **-args. However, I agree with Antoine that PEP 8 should be updated to suggest that *-args should precede any keyword arguments. It is currently allowed to write f(x=2, *args), which is equivalent to f(*args, x=2). Best, Neil On Mon, Feb 9, 2015 at 7:30 PM, Larry Hastings wrote: > > > What's an example of a way inspect.signature must change? I thought PEP > 448 added new unpacking shortcuts which (for example) change the *caller* > side of a function call. I didn't realize it impacted the *callee* side > too. > > > */arry* > > On 02/09/2015 03:14 PM, Antoine Pitrou wrote: > > On Tue, 10 Feb 2015 08:43:53 +1000 > Nick Coghlan wrote: > > For example, the potential for arcane call arguments suggests the need for > a PEP 8 addition saying "first standalone args, then iterable expansions, > then mapping expansions", even though syntactically any order would now be > permitted at call time. > > There are other concerns: > > - inspect.signature() must be updated to cover the new call > possibilities > > - function call performance must not be crippled by the new > possibilities > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing listPython-Dev at python.orghttps://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/larry%40hastings.org > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Feb 10 01:53:39 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 9 Feb 2015 19:53:39 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: > On Feb 9, 2015, at 7:29 PM, Neil Girdhar wrote: > > For some reason I can't seem to reply using Google groups, which is is telling "this is a read-only mirror" (anyone know why?) Anyway, I'm going to answer as best I can the concerns. > > Antoine said: > > To be clear, the PEP will probably be useful for one single line of > Python code every 10000. This is a very weak case for such an intrusive > syntax addition. I would support the PEP if it only added the simple > cases of tuple unpacking, left alone function call conventions, and > didn't introduce **-unpacking. > > To me this is more of a syntax simplification than a syntax addition. For me the **-unpacking is the most useful part. Regarding utility, it seems that a many of the people on r/python were pretty excited about this PEP: http://www.reddit.com/r/Python/comments/2synry/so_8_peps_are_currently_being_proposed_for_python/ > > ? > > Victor noticed that there's a mistake with the code: > > >>> ranges = [range(i) for i in range(5)] > >>> [*item for item in ranges] > [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] > > It should be a range(4) in the code. The "*" applies to only item. It is the same as writing: > > [*range(0), *range(1), *range(2), *range(3), *range(4)] > > which is the same as unpacking all of those ranges into a list. > > > function(**kw_arguments, **more_arguments) > If the key "key1" is in both dictionaries, more_arguments wins, right? > > There was some debate and it was decided that duplicate keyword arguments would remain an error (for now at least). If you want to merge the dictionaries with overriding, then you can still do: > > function(**{**kw_arguments, **more_arguments}) > > because **-unpacking in dicts overrides as you guessed. > > ? > > > > On Mon, Feb 9, 2015 at 7:12 PM, Donald Stufft > wrote: > >> On Feb 9, 2015, at 4:06 PM, Neil Girdhar > wrote: >> >> Hello all, >> >> The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/ ) is implemented now based on some early work by Thomas Wouters (in 2008) and Florian Hahn (2013) and recently completed by Joshua Landau and me. >> >> The issue tracker http://bugs.python.org/issue2292 has a working patch. Would someone be able to review it? >> > > I just skimmed over the PEP and it seems like it?s trying to solve a few different things: > > * Making it easy to combine multiple lists and additional positional args into a function call > * Making it easy to combine multiple dicts and additional keyword args into a functional call > * Making it easy to do a single level of nested iterable "flatten". > > I would say it's: > * making it easy to unpack iterables and mappings in function calls > * making it easy to unpack iterables into list and set displays and comprehensions, and > * making it easy to unpack mappings into dict displays and comprehensions. > > > > Looking at the syntax in the PEP I had a hard time detangling what exactly it was doing even with reading the PEP itself. I wonder if there isn?t a way to combine simpler more generic things to get the same outcome. > > Looking at the "Making it easy to combine multiple lists and additional positional args into a function call" aspect of this, why is: > > print(*[1], *[2], 3) better than print(*[1] + [2] + [3])? > > That's already doable in Python right now and doesn't require anything new to handle it. > > Admittedly, this wasn't a great example. But, if [1] and [2] had been iterables, you would have to cast each to list, e.g., > > accumulator = [] > accumulator.extend(a) > accumulator.append(b) > accumulator.extend(c) > print(*accumulator) > > replaces > > print(*a, b, *c) > > where a and c are iterable. The latter version is also more efficient because it unpacks only a onto the stack allocating no auxilliary list. Honestly that doesn?t seem like the way I?d write it at all, if they might not be lists I?d just cast them to lists: print(*list(a) + [b] + list(c)) But if casting to list really is that big a deal, then perhaps a better solution is to simply make it so that something like ``a_list + an_iterable`` is valid and the iterable would just be consumed and +?d onto the list. That still feels like a more general solution and a far less surprising and easier to read one. > > > Looking at the "making it easy to do a single level of nsted iterable 'flatten'"" aspect of this, the example of: > > >>> ranges = [range(i) for i in range(5)] > >>> [*item for item in ranges] > [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] > > Conceptually a list comprehension like [thing for item in iterable] can be mapped to a for loop like this: > > result = [] > for item in iterable: > result.append(thing) > > However [*item for item in ranges] is mapped more to something like this: > > result = [] > for item in iterable: > result.extend(*item) > > I feel like switching list comprehensions from append to extend just because of a * is really confusing and it acts differently than if you just did *item outside of a list comprehension. I feel like the itertools.chain() way of doing this is *much* clearer. > > Finally there's the "make it easy to combine multiple dicts into a function call" aspect of this. This I think is the biggest thing that this PEP actually adds, however I think it goes around it the wrong way. Sadly there is nothing like [1] + [2] for dictionaries. The closest thing is: > > kwargs = dict1.copy() > kwargs.update(dict2) > func(**kwargs) > > So what I wonder is if this PEP wouldn't be better off just using the existing methods for doing the kinds of things that I pointed out above, and instead defining + or | or some other symbol for something similar to [1] + [2] but for dictionaries. This would mean that you could simply do: > > func(**dict1 | dict(y=1) | dict2) > > instead of > > dict(**{'x': 1}, y=2, **{'z': 3}) > > I feel like not only does this genericize way better but it limits the impact and new syntax being added to Python and is a ton more readable. Honestly the use of * and ** in functions doesn?t bother me a whole lot, though i don?t see much use for it over what?s already available for lists (and I think doing something similarly generic for mapping is a better idea). What really bothers me is these parts: * making it easy to unpack iterables into list and set displays and comprehensions, and * making it easy to unpack mappings into dict displays and comprehensions. I feel like these are super wrong and if they were put in I?d probably end up writing a linter to disallow them in my own code bases. I feel like adding a special case for * in list comprehensions breaks the ?manually expanded? version of those. Switching from append to extend inside of a list comprehension because of a * doesn?t make any sense to me. I can?t seem to construct any for loop that mimics what this PEP proposes as [*item for item in iterable] without fundamentally changing the operation that happens in each loop of the list comprehension. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Feb 10 01:53:48 2015 From: barry at python.org (Barry Warsaw) Date: Mon, 9 Feb 2015 19:53:48 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <54D927F1.4090708@stoneleaf.us> <1423519858.3340452.225336093.38A61898@webmail.messagingengine.com> <20150210001432.0be7747f@fsol> <54D95110.9090104@hastings.org> Message-ID: <20150209195348.7e806741@limelight.wooz.org> On Feb 09, 2015, at 07:46 PM, Neil Girdhar wrote: >Also, regarding calling argument order, not any order is allowed. Regular >arguments must precede other kinds of arguments. Keyword arguments must >precede **-args. *-args must precede **-args. However, I agree with >Antoine that PEP 8 should be updated to suggest that *-args should precede >any keyword arguments. It is currently allowed to write f(x=2, *args), >which is equivalent to f(*args, x=2). But if we have to add a PEP 8 admonition against some syntax that's being newly added, why is this an improvement? I had some more snarky/funny comments to make, but I'll just say -1. The Rationale in the PEP doesn't sell me on it being an improvement to Python. Cheers, -Barry From mistersheik at gmail.com Tue Feb 10 01:56:57 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Feb 2015 19:56:57 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: <20150209195348.7e806741@limelight.wooz.org> References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <54D927F1.4090708@stoneleaf.us> <1423519858.3340452.225336093.38A61898@webmail.messagingengine.com> <20150210001432.0be7747f@fsol> <54D95110.9090104@hastings.org> <20150209195348.7e806741@limelight.wooz.org> Message-ID: The admonition is against syntax that currently exists. On Mon, Feb 9, 2015 at 7:53 PM, Barry Warsaw wrote: > On Feb 09, 2015, at 07:46 PM, Neil Girdhar wrote: > > >Also, regarding calling argument order, not any order is allowed. Regular > >arguments must precede other kinds of arguments. Keyword arguments must > >precede **-args. *-args must precede **-args. However, I agree with > >Antoine that PEP 8 should be updated to suggest that *-args should precede > >any keyword arguments. It is currently allowed to write f(x=2, *args), > >which is equivalent to f(*args, x=2). > > But if we have to add a PEP 8 admonition against some syntax that's being > newly added, why is this an improvement? > > I had some more snarky/funny comments to make, but I'll just say -1. The > Rationale in the PEP doesn't sell me on it being an improvement to Python. > > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Feb 10 02:12:12 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Feb 2015 20:12:12 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <1423517293.3332351.225317905.05ED497F@webmail.messagingengine.com> <54D927F1.4090708@stoneleaf.us> <1423519858.3340452.225336093.38A61898@webmail.messagingengine.com> <20150210001432.0be7747f@fsol> <54D95110.9090104@hastings.org> <20150209195348.7e806741@limelight.wooz.org> Message-ID: Just an FYI: http://www.reddit.com/r/Python/comments/2v8g26/python_350_alpha_1_has_been_released/ 448 was mentioned here (by Python lay people ? not developers). On Mon, Feb 9, 2015 at 7:56 PM, Neil Girdhar wrote: > The admonition is against syntax that currently exists. > > On Mon, Feb 9, 2015 at 7:53 PM, Barry Warsaw wrote: > >> On Feb 09, 2015, at 07:46 PM, Neil Girdhar wrote: >> >> >Also, regarding calling argument order, not any order is allowed. >> Regular >> >arguments must precede other kinds of arguments. Keyword arguments must >> >precede **-args. *-args must precede **-args. However, I agree with >> >Antoine that PEP 8 should be updated to suggest that *-args should >> precede >> >any keyword arguments. It is currently allowed to write f(x=2, *args), >> >which is equivalent to f(*args, x=2). >> >> But if we have to add a PEP 8 admonition against some syntax that's being >> newly added, why is this an improvement? >> >> I had some more snarky/funny comments to make, but I'll just say -1. The >> Rationale in the PEP doesn't sell me on it being an improvement to Python. >> >> Cheers, >> -Barry >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Feb 10 02:14:20 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 10 Feb 2015 02:14:20 +0100 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: 2015-02-10 1:29 GMT+01:00 Neil Girdhar : > For some reason I can't seem to reply using Google groups, which is is > telling "this is a read-only mirror" (anyone know why?) Anyway, I'm going > to answer as best I can the concerns. > > Antoine said: > >> To be clear, the PEP will probably be useful for one single line of >> Python code every 10000. This is a very weak case for such an intrusive >> syntax addition. I would support the PEP if it only added the simple >> cases of tuple unpacking, left alone function call conventions, and >> didn't introduce **-unpacking. > > > To me this is more of a syntax simplification than a syntax addition. For > me the **-unpacking is the most useful part. Regarding utility, it seems > that a many of the people on r/python were pretty excited about this PEP: > http://www.reddit.com/r/Python/comments/2synry/so_8_peps_are_currently_being_proposed_for_python/ I used grep to find how many times dict.update() is used. haypo at selma$ wc -l Lib/*.py (....) 112055 total haypo at selma$ grep '\.update(.\+)' Lib/*.py|wc -l 63 So there are 63 or less (it's a regex, I didn't check each line) calls to dict.update() on a total of 112,055 lines. I found a few numbers of codes using the pattern: "dict1.update(dict2); func(**dict1)". Examples: functools.py: def partial(func, *args, **keywords): def newfunc(*fargs, **fkeywords): newkeywords = keywords.copy() newkeywords.update(fkeywords) return func(*(args + fargs), **newkeywords) ... return newfunc => def partial(func, *args, **keywords): def newfunc(*fargs, **fkeywords): return func(*(args + fargs), **keywords, **fkeywords) ... return newfunc The new code behaves differently since Neil said that an error is raised if fkeywords and keywords have keys in common. By the way, this must be written in the PEP. pdb.py: ns = self.curframe.f_globals.copy() ns.update(self.curframe_locals) code.interact("*interactive*", local=ns) Hum no sorry, ns is not used with ** here. Victor From mistersheik at gmail.com Tue Feb 10 02:34:30 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Feb 2015 20:34:30 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: On Mon, Feb 9, 2015 at 7:53 PM, Donald Stufft wrote: > > On Feb 9, 2015, at 7:29 PM, Neil Girdhar wrote: > > For some reason I can't seem to reply using Google groups, which is is > telling "this is a read-only mirror" (anyone know why?) Anyway, I'm going > to answer as best I can the concerns. > > Antoine said: > > To be clear, the PEP will probably be useful for one single line of >> Python code every 10000. This is a very weak case for such an intrusive >> syntax addition. I would support the PEP if it only added the simple >> cases of tuple unpacking, left alone function call conventions, and >> didn't introduce **-unpacking. > > > To me this is more of a syntax simplification than a syntax addition. For > me the **-unpacking is the most useful part. Regarding utility, it seems > that a many of the people on r/python were pretty excited about this PEP: > http://www.reddit.com/r/Python/comments/2synry/so_8_peps_are_currently_being_proposed_for_python/ > > ? > > Victor noticed that there's a mistake with the code: > > >>> ranges = [range(i) for i in range(5)] >> >>> [*item for item in ranges] >> [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] > > > It should be a range(4) in the code. The "*" applies to only item. It is > the same as writing: > > [*range(0), *range(1), *range(2), *range(3), *range(4)] > > which is the same as unpacking all of those ranges into a list. > > > function(**kw_arguments, **more_arguments) >> If the key "key1" is in both dictionaries, more_arguments wins, right? > > > There was some debate and it was decided that duplicate keyword arguments > would remain an error (for now at least). If you want to merge the > dictionaries with overriding, then you can still do: > > function(**{**kw_arguments, **more_arguments}) > > because **-unpacking in dicts overrides as you guessed. > > ? > > > > On Mon, Feb 9, 2015 at 7:12 PM, Donald Stufft wrote: > >> >> On Feb 9, 2015, at 4:06 PM, Neil Girdhar wrote: >> >> Hello all, >> >> The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/) is >> implemented now based on some early work by Thomas Wouters (in 2008) and >> Florian Hahn (2013) and recently completed by Joshua Landau and me. >> >> The issue tracker http://bugs.python.org/issue2292 has a working >> patch. Would someone be able to review it? >> >> >> I just skimmed over the PEP and it seems like it?s trying to solve a few >> different things: >> >> * Making it easy to combine multiple lists and additional positional args >> into a function call >> * Making it easy to combine multiple dicts and additional keyword args >> into a functional call >> * Making it easy to do a single level of nested iterable "flatten". >> > > I would say it's: > * making it easy to unpack iterables and mappings in function calls > * making it easy to unpack iterables into list and set displays and > comprehensions, and > * making it easy to unpack mappings into dict displays and comprehensions. > > > >> >> Looking at the syntax in the PEP I had a hard time detangling what >> exactly it was doing even with reading the PEP itself. I wonder if there >> isn?t a way to combine simpler more generic things to get the same outcome. >> >> Looking at the "Making it easy to combine multiple lists and additional >> positional args into a function call" aspect of this, why is: >> >> print(*[1], *[2], 3) better than print(*[1] + [2] + [3])? >> >> That's already doable in Python right now and doesn't require anything >> new to handle it. >> > > Admittedly, this wasn't a great example. But, if [1] and [2] had been > iterables, you would have to cast each to list, e.g., > > accumulator = [] > accumulator.extend(a) > accumulator.append(b) > accumulator.extend(c) > print(*accumulator) > > replaces > > print(*a, b, *c) > > where a and c are iterable. The latter version is also more efficient > because it unpacks only a onto the stack allocating no auxilliary list. > > > Honestly that doesn?t seem like the way I?d write it at all, if they might > not be lists I?d just cast them to lists: > > print(*list(a) + [b] + list(c)) > Sure, that works too as long as you put in the missing parentheses. > > But if casting to list really is that big a deal, then perhaps a better > solution is to simply make it so that something like ``a_list + > an_iterable`` is valid and the iterable would just be consumed and +?d onto > the list. That still feels like a more general solution and a far less > surprising and easier to read one. > I understand. However I just want to point out that 448 is more general. There is no binary operator for generators. How do you write (*a, *b, *c)? You need to use itertools.chain(a, b, c). > > > > >> Looking at the "making it easy to do a single level of nsted iterable >> 'flatten'"" aspect of this, the example of: >> >> >>> ranges = [range(i) for i in range(5)] >> >>> [*item for item in ranges] >> [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] >> >> Conceptually a list comprehension like [thing for item in iterable] can >> be mapped to a for loop like this: >> >> result = [] >> for item in iterable: >> result.append(thing) >> >> However [*item for item in ranges] is mapped more to something like this: >> >> result = [] >> for item in iterable: >> result.extend(*item) >> >> I feel like switching list comprehensions from append to extend just >> because of a * is really confusing and it acts differently than if you just >> did *item outside of a list comprehension. I feel like the >> itertools.chain() way of doing this is *much* clearer. >> >> Finally there's the "make it easy to combine multiple dicts into a >> function call" aspect of this. This I think is the biggest thing that this >> PEP actually adds, however I think it goes around it the wrong way. Sadly >> there is nothing like [1] + [2] for dictionaries. The closest thing is: >> >> kwargs = dict1.copy() >> kwargs.update(dict2) >> func(**kwargs) >> >> So what I wonder is if this PEP wouldn't be better off just using the >> existing methods for doing the kinds of things that I pointed out above, >> and instead defining + or | or some other symbol for something similar to >> [1] + [2] but for dictionaries. This would mean that you could simply do: >> >> func(**dict1 | dict(y=1) | dict2) >> >> instead of >> >> dict(**{'x': 1}, y=2, **{'z': 3}) >> >> I feel like not only does this genericize way better but it limits the >> impact and new syntax being added to Python and is a ton more readable. >> > > > Honestly the use of * and ** in functions doesn?t bother me a whole lot, > though i don?t see much use for it over what?s already available for lists > (and I think doing something similarly generic for mapping is a better > idea). What really bothers me is these parts: > > * making it easy to unpack iterables into list and set displays and > comprehensions, and > * making it easy to unpack mappings into dict displays and comprehensions. > > I feel like these are super wrong and if they were put in I?d probably end > up writing a linter to disallow them in my own code bases. > > I feel like adding a special case for * in list comprehensions breaks the > ?manually expanded? version of those. Switching from append to extend > inside of a list comprehension because of a * doesn?t make any sense to me. > I can?t seem to construct any for loop that mimics what this PEP proposes > as [*item for item in iterable] without fundamentally changing the > operation that happens in each loop of the list comprehension. > > I don't know what you mean by this. You can write [*item for item in iterable] in current Python as [it for item in iterable for it in item]. You can unroll that as: a = [] for item in iterable: for it in item: a.append(it) ? or yield for generators or add for sets. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Feb 10 02:48:01 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 9 Feb 2015 20:48:01 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: <6A96EBBA-D2EC-4E2C-B4B7-BD0FF1D42DEF@stufft.io> > On Feb 9, 2015, at 8:34 PM, Neil Girdhar wrote: > > > > On Mon, Feb 9, 2015 at 7:53 PM, Donald Stufft > wrote: > >> On Feb 9, 2015, at 7:29 PM, Neil Girdhar > wrote: >> >> For some reason I can't seem to reply using Google groups, which is is telling "this is a read-only mirror" (anyone know why?) Anyway, I'm going to answer as best I can the concerns. >> >> Antoine said: >> >> To be clear, the PEP will probably be useful for one single line of >> Python code every 10000. This is a very weak case for such an intrusive >> syntax addition. I would support the PEP if it only added the simple >> cases of tuple unpacking, left alone function call conventions, and >> didn't introduce **-unpacking. >> >> To me this is more of a syntax simplification than a syntax addition. For me the **-unpacking is the most useful part. Regarding utility, it seems that a many of the people on r/python were pretty excited about this PEP: http://www.reddit.com/r/Python/comments/2synry/so_8_peps_are_currently_being_proposed_for_python/ >> >> ? >> >> Victor noticed that there's a mistake with the code: >> >> >>> ranges = [range(i) for i in range(5)] >> >>> [*item for item in ranges] >> [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] >> >> It should be a range(4) in the code. The "*" applies to only item. It is the same as writing: >> >> [*range(0), *range(1), *range(2), *range(3), *range(4)] >> >> which is the same as unpacking all of those ranges into a list. >> >> > function(**kw_arguments, **more_arguments) >> If the key "key1" is in both dictionaries, more_arguments wins, right? >> >> There was some debate and it was decided that duplicate keyword arguments would remain an error (for now at least). If you want to merge the dictionaries with overriding, then you can still do: >> >> function(**{**kw_arguments, **more_arguments}) >> >> because **-unpacking in dicts overrides as you guessed. >> >> ? >> >> >> >> On Mon, Feb 9, 2015 at 7:12 PM, Donald Stufft > wrote: >> >>> On Feb 9, 2015, at 4:06 PM, Neil Girdhar > wrote: >>> >>> Hello all, >>> >>> The updated PEP 448 (https://www.python.org/dev/peps/pep-0448/ ) is implemented now based on some early work by Thomas Wouters (in 2008) and Florian Hahn (2013) and recently completed by Joshua Landau and me. >>> >>> The issue tracker http://bugs.python.org/issue2292 has a working patch. Would someone be able to review it? >>> >> >> I just skimmed over the PEP and it seems like it?s trying to solve a few different things: >> >> * Making it easy to combine multiple lists and additional positional args into a function call >> * Making it easy to combine multiple dicts and additional keyword args into a functional call >> * Making it easy to do a single level of nested iterable "flatten". >> >> I would say it's: >> * making it easy to unpack iterables and mappings in function calls >> * making it easy to unpack iterables into list and set displays and comprehensions, and >> * making it easy to unpack mappings into dict displays and comprehensions. >> >> >> >> Looking at the syntax in the PEP I had a hard time detangling what exactly it was doing even with reading the PEP itself. I wonder if there isn?t a way to combine simpler more generic things to get the same outcome. >> >> Looking at the "Making it easy to combine multiple lists and additional positional args into a function call" aspect of this, why is: >> >> print(*[1], *[2], 3) better than print(*[1] + [2] + [3])? >> >> That's already doable in Python right now and doesn't require anything new to handle it. >> >> Admittedly, this wasn't a great example. But, if [1] and [2] had been iterables, you would have to cast each to list, e.g., >> >> accumulator = [] >> accumulator.extend(a) >> accumulator.append(b) >> accumulator.extend(c) >> print(*accumulator) >> >> replaces >> >> print(*a, b, *c) >> >> where a and c are iterable. The latter version is also more efficient because it unpacks only a onto the stack allocating no auxilliary list. > > Honestly that doesn?t seem like the way I?d write it at all, if they might not be lists I?d just cast them to lists: > > print(*list(a) + [b] + list(c)) > > Sure, that works too as long as you put in the missing parentheses. There are no missing parentheses, the * and ** is last in the order of operations (though the parens would likely make that more clear). > > > But if casting to list really is that big a deal, then perhaps a better solution is to simply make it so that something like ``a_list + an_iterable`` is valid and the iterable would just be consumed and +?d onto the list. That still feels like a more general solution and a far less surprising and easier to read one. > > I understand. However I just want to point out that 448 is more general. There is no binary operator for generators. How do you write (*a, *b, *c)? You need to use itertools.chain(a, b, c). I don?t feel like using itertools.chain is a bad thing TBH, it?s extremely clear as to what?s going on, you?re chaining a bunch a bunch of iterables together. I would not however be super upset if the ability to do * and ** multiple times in a function was added, I just don?t think it?s very useful for * (since you can already get that behavior with things I believe are clear-er) and I think getting similar constructs for ** would bring that up to parity. I am really really -1 on the comprehension syntax. > > > >> >> >> Looking at the "making it easy to do a single level of nsted iterable 'flatten'"" aspect of this, the example of: >> >> >>> ranges = [range(i) for i in range(5)] >> >>> [*item for item in ranges] >> [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] >> >> Conceptually a list comprehension like [thing for item in iterable] can be mapped to a for loop like this: >> >> result = [] >> for item in iterable: >> result.append(thing) >> >> However [*item for item in ranges] is mapped more to something like this: >> >> result = [] >> for item in iterable: >> result.extend(*item) >> >> I feel like switching list comprehensions from append to extend just because of a * is really confusing and it acts differently than if you just did *item outside of a list comprehension. I feel like the itertools.chain() way of doing this is *much* clearer. >> >> Finally there's the "make it easy to combine multiple dicts into a function call" aspect of this. This I think is the biggest thing that this PEP actually adds, however I think it goes around it the wrong way. Sadly there is nothing like [1] + [2] for dictionaries. The closest thing is: >> >> kwargs = dict1.copy() >> kwargs.update(dict2) >> func(**kwargs) >> >> So what I wonder is if this PEP wouldn't be better off just using the existing methods for doing the kinds of things that I pointed out above, and instead defining + or | or some other symbol for something similar to [1] + [2] but for dictionaries. This would mean that you could simply do: >> >> func(**dict1 | dict(y=1) | dict2) >> >> instead of >> >> dict(**{'x': 1}, y=2, **{'z': 3}) >> >> I feel like not only does this genericize way better but it limits the impact and new syntax being added to Python and is a ton more readable. > > > Honestly the use of * and ** in functions doesn?t bother me a whole lot, though i don?t see much use for it over what?s already available for lists (and I think doing something similarly generic for mapping is a better idea). What really bothers me is these parts: > > * making it easy to unpack iterables into list and set displays and comprehensions, and > * making it easy to unpack mappings into dict displays and comprehensions. > > I feel like these are super wrong and if they were put in I?d probably end up writing a linter to disallow them in my own code bases. > > I feel like adding a special case for * in list comprehensions breaks the ?manually expanded? version of those. Switching from append to extend inside of a list comprehension because of a * doesn?t make any sense to me. I can?t seem to construct any for loop that mimics what this PEP proposes as [*item for item in iterable] without fundamentally changing the operation that happens in each loop of the list comprehension. > > > I don't know what you mean by this. You can write [*item for item in iterable] in current Python as [it for item in iterable for it in item]. You can unroll that as: > a = [] > for item in iterable: > for it in item: > a.append(it) > > ? or yield for generators or add for sets. I don?t think * means ?loop? anywhere else in Python and I would never ?guess? that [*item for item in iterable] meant that. It?s completely non intuitive. Anywhere else you see *foo it?s unpacking a tuple not making an inner loop. That means that anywhere else in Python *item is the same thing as item[0], item[1], item[2], ?, but this PEP makes it so just inside of a comprehension it actually means ?make a second, inner loop? instead of what I think anyone who has learned that syntax would expect, which is it should be equivalent to [(item[0], item[1], item[2], ?) for item in iterable]. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Feb 10 03:06:02 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 09 Feb 2015 18:06:02 -0800 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: <54D9678A.1070108@stoneleaf.us> On 02/09/2015 05:14 PM, Victor Stinner wrote: > > def partial(func, *args, **keywords): > def newfunc(*fargs, **fkeywords): > return func(*(args + fargs), **keywords, **fkeywords) > ... > return newfunc > > The new code behaves differently since Neil said that an error is > raised if fkeywords and keywords have keys in common. By the way, this > must be written in the PEP. That line should read return func(*(args + fargs), **{**keywords, **fkeywords}) to avoid the duplicate key error and keep the original functionality. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Tue Feb 10 03:10:36 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 09 Feb 2015 18:10:36 -0800 Subject: [Python-Dev] (no subject) In-Reply-To: <6A96EBBA-D2EC-4E2C-B4B7-BD0FF1D42DEF@stufft.io> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <6A96EBBA-D2EC-4E2C-B4B7-BD0FF1D42DEF@stufft.io> Message-ID: <54D9689C.2030307@stoneleaf.us> On 02/09/2015 05:48 PM, Donald Stufft wrote: > > I don?t think * means ?loop? anywhere else in Python and I would never ?guess? that > > [*item for item in iterable] > > meant that. It?s completely non intuitive. Anywhere else you see *foo it?s unpacking a tuple not making an inner loop. That > means that anywhere else in Python *item is the same thing as item[0], item[1], item[2], ?, but this PEP makes it so > just inside of a comprehension it actually means ?make a second, inner loop? instead of what I think anyone who has > learned that syntax would expect, which is it should be equivalent to [(item[0], item[1], item[2], ?) for item in iterable]. I agree with Donald. I would expect a list of lists from that syntax... or maybe a list of tuples? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From greg.ewing at canterbury.ac.nz Tue Feb 10 06:47:19 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 10 Feb 2015 18:47:19 +1300 Subject: [Python-Dev] (no subject) In-Reply-To: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: <54D99B67.8040905@canterbury.ac.nz> Donald Stufft wrote: > > why is: > > print(*[1], *[2], 3) better than print(*[1] + [2] + [3])? It could potentially be a little more efficient by eliminating the construction of an intermediate list. > defining + or | or some other symbol for something similar > to [1] + [2] but for dictionaries. This would mean that you could simply do: > > func(**dict1 | dict(y=1) | dict2) Same again, multiple ** avoids construction of an itermediate dict. -- Greg From greg.ewing at canterbury.ac.nz Tue Feb 10 06:55:20 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 10 Feb 2015 18:55:20 +1300 Subject: [Python-Dev] (no subject) In-Reply-To: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: <54D99D48.7020700@canterbury.ac.nz> Donald Stufft wrote: > However [*item for item in ranges] is mapped more to something like this: > > result = [] > for item in iterable: > result.extend(*item) Actually it would be result.extend(item) But if that bothers you, you could consider the expansion to be result = [] for item in iterable: for item1 in item: result.append(item) In other words, the * is shorthand for an extra level of looping. > and it acts differently than if you > just did *item outside of a list comprehension. Not sure what you mean by that. It seems closely analogous to the use of * in a function call to me. -- Greg From greg.ewing at canterbury.ac.nz Tue Feb 10 07:04:03 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 10 Feb 2015 19:04:03 +1300 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: <54D99F53.90506@canterbury.ac.nz> Donald Stufft wrote: > > perhaps a better > solution is to simply make it so that something like ``a_list + > an_iterable`` is valid and the iterable would just be consumed and +?d > onto the list. I don't think I like the asymmetry that this would introduce into + on lists. Currently [1, 2, 3] + (4, 5, 6) is an error because it's not clear whether the programmer intended the result to be a list or a tuple. I think that's a good thing. Also, it would mean that [1, 2, 3] + foo == [1, 2, 3, "f", "o", "o"] which would be surprising and probably not what was intended. -- Greg From donald at stufft.io Tue Feb 10 07:31:30 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 10 Feb 2015 01:31:30 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: <54D99D48.7020700@canterbury.ac.nz> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D99D48.7020700@canterbury.ac.nz> Message-ID: > On Feb 10, 2015, at 12:55 AM, Greg Ewing wrote: > > Donald Stufft wrote: >> However [*item for item in ranges] is mapped more to something like this: >> result = [] >> for item in iterable: >> result.extend(*item) > > Actually it would be > > result.extend(item) > > But if that bothers you, you could consider the expansion > to be > > result = [] > for item in iterable: > for item1 in item: > result.append(item) > > In other words, the * is shorthand for an extra level > of looping. > >> and it acts differently than if you just did *item outside of a list comprehension. > > Not sure what you mean by that. It seems closely > analogous to the use of * in a function call to > me. > Putting aside the proposed syntax the current two statements are currently true: 1. The statement *item is roughly the same thing as (item[0], item[1], item[n]) 2. The statement [something for thing in iterable] is roughly the same as: result = [] for thing in iterable: result.append(something) This is a single loop where an expression is ran for each iteration of the loop, and the return result of that expression is appended to the result. If you combine these two things, the "something" in #2 becuase *item, and since *item is roughly the same thing as (item[0], item[1], item[n]) what you end up with is something that *should* behave like: result = [] for thing in iterable: result.append((thing[0], thing[1], thing[n])) Or to put it another way: >>> [*item for item in [[1, 2, 3], [4, 5, 6]] [(1, 2, 3), (4, 5, 6)] Is a lot more consistent with what *thing and list comprehensions already mean in Python than for the answer to be [1, 2, 3, 4, 5, 6]. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From mistersheik at gmail.com Tue Feb 10 07:49:36 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 10 Feb 2015 01:49:36 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D99D48.7020700@canterbury.ac.nz> Message-ID: On Tue, Feb 10, 2015 at 1:31 AM, Donald Stufft wrote: > > > On Feb 10, 2015, at 12:55 AM, Greg Ewing > wrote: > > > > Donald Stufft wrote: > >> However [*item for item in ranges] is mapped more to something like > this: > >> result = [] > >> for item in iterable: > >> result.extend(*item) > > > > Actually it would be > > > > result.extend(item) > > > > But if that bothers you, you could consider the expansion > > to be > > > > result = [] > > for item in iterable: > > for item1 in item: > > result.append(item) > > > > In other words, the * is shorthand for an extra level > > of looping. > > > >> and it acts differently than if you just did *item outside of a list > comprehension. > > > > Not sure what you mean by that. It seems closely > > analogous to the use of * in a function call to > > me. > > > > Putting aside the proposed syntax the current two statements are currently > true: > > 1. The statement *item is roughly the same thing as (item[0], item[1], > item[n]) > 2. The statement [something for thing in iterable] is roughly the same as: > result = [] > for thing in iterable: > result.append(something) > This is a single loop where an expression is ran for each iteration of > the > loop, and the return result of that expression is appended to the > result. > > If you combine these two things, the "something" in #2 becuase *item, and > since > *item is roughly the same thing as (item[0], item[1], item[n]) what you end > up with is something that *should* behave like: > > result = [] > for thing in iterable: > result.append((thing[0], thing[1], thing[n])) > That is what [list(something) for thing in iterable] does. The iterable unpacking rule might have been better explained as follows: ???? On the left of assignment * is packing, e.g. a, b, *cs = iterable On the right of an assignment, * is an unpacking, e.g. xs = a, b, *cs ???? In either case, the elements of "cs" are treated the same as a and b. Do you agree that [*x for x in [as, bs, cs]] === [*as, *bs, *cs] ? Then the elements of *as are unpacked into the list, the same way that those elements are currently unpacked in a regular function call f(*as) === f(as[0], as[1], ...) Similarly, [*as, *bs, *cs] === [as[0], as[1], ?, bs[0], bs[1], ?, cs[0], cs[1], ?] The rule for function calls is analogous: ???? In a function definition, * is a packing, collecting extra positional argument in a list. E.g., def f(*args): In a function call, * is an unpacking, expanding an iterable to populate positional arguments. E.g., f(*args) ? PEP 448 proposes having arbitrary numbers of unpackings in arbitrary positions. I will be updating the PEP this week if I can find the time. > > Or to put it another way: > > >>> [*item for item in [[1, 2, 3], [4, 5, 6]] > [(1, 2, 3), (4, 5, 6)] > > > Is a lot more consistent with what *thing and list comprehensions already > mean > in Python than for the answer to be [1, 2, 3, 4, 5, 6]. > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Feb 10 08:08:48 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 10 Feb 2015 08:08:48 +0100 Subject: [Python-Dev] (no subject) In-Reply-To: <54D9678A.1070108@stoneleaf.us> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D9678A.1070108@stoneleaf.us> Message-ID: Le 10 f?vr. 2015 03:07, "Ethan Furman" a ?crit : > That line should read > > return func(*(args + fargs), **{**keywords, **fkeywords}) > > to avoid the duplicate key error and keep the original functionality. To me, this is just ugly. It prefers the original code which use .update(). Maybe the PEP should be changed to behave as .update()? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Feb 10 08:16:40 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 10 Feb 2015 02:16:40 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D9678A.1070108@stoneleaf.us> Message-ID: On Tue, Feb 10, 2015 at 2:08 AM, Victor Stinner wrote: > > Le 10 f?vr. 2015 03:07, "Ethan Furman" a ?crit : > > That line should read > > > > return func(*(args + fargs), **{**keywords, **fkeywords}) > > > > to avoid the duplicate key error and keep the original functionality. > > To me, this is just ugly. It prefers the original code which use .update(). > > Maybe the PEP should be changed to behave as .update()? > > Victor > > Just for clarity, Ethan is right, but it could also be written: return func(*args, *fargs, **{**keywords, **fkeywords}) Best, Neil > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Feb 10 08:20:06 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 10 Feb 2015 08:20:06 +0100 Subject: [Python-Dev] (no subject) In-Reply-To: <54D9689C.2030307@stoneleaf.us> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <6A96EBBA-D2EC-4E2C-B4B7-BD0FF1D42DEF@stufft.io> <54D9689C.2030307@stoneleaf.us> Message-ID: To be logic, I expect [(*item) for item in mylist] to simply return mylist. [*(item for item in mylist] with mylist=[(1, 2), (3,)] could return [1, 2, 3], as just [*mylist], so "unpack" mylist. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Feb 10 08:29:02 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 10 Feb 2015 02:29:02 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <6A96EBBA-D2EC-4E2C-B4B7-BD0FF1D42DEF@stufft.io> <54D9689C.2030307@stoneleaf.us> Message-ID: On Tue, Feb 10, 2015 at 2:20 AM, Victor Stinner wrote: > To be logic, I expect [(*item) for item in mylist] to simply return mylist. > If you want simply mylist as a list, that is [*mylist] > [*(item) for item in mylist] with mylist=[(1, 2), (3,)] could return [1, > 2, 3], > right > as just [*mylist], so "unpack" mylist. > [*mylist] remains equivalent list(mylist), just as it is now. In one case, you're unpacking the elements of the list, in the other you're unpacking the list itself. Victor > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Feb 10 08:31:35 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 10 Feb 2015 02:31:35 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <6A96EBBA-D2EC-4E2C-B4B7-BD0FF1D42DEF@stufft.io> <54D9689C.2030307@stoneleaf.us> Message-ID: ah, sorry? forget that I said "just as it is now" ? I am losing track of what's allowed in Python now! On Tue, Feb 10, 2015 at 2:29 AM, Neil Girdhar wrote: > > > On Tue, Feb 10, 2015 at 2:20 AM, Victor Stinner > wrote: > >> To be logic, I expect [(*item) for item in mylist] to simply return >> mylist. >> > > If you want simply mylist as a list, that is [*mylist] > >> [*(item) for item in mylist] with mylist=[(1, 2), (3,)] could return [1, >> 2, 3], >> > right > >> as just [*mylist], so "unpack" mylist. >> > > [*mylist] remains equivalent list(mylist), just as it is now. In one > case, you're unpacking the elements of the list, in the other you're > unpacking the list itself. > > Victor >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Feb 10 09:11:22 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 10 Feb 2015 09:11:22 +0100 Subject: [Python-Dev] (no subject) In-Reply-To: <54D99B67.8040905@canterbury.ac.nz> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D99B67.8040905@canterbury.ac.nz> Message-ID: Le 10 f?vr. 2015 06:48, "Greg Ewing" a ?crit : > It could potentially be a little more efficient by > eliminating the construction of an intermediate list. Is it the case in the implementation? If it has to create a temporary list/tuple, I will prefer to not use it. After long years of development, I chose to limit myself to one instruction per line. It is for different reason: - I spend more time to read code than to write code, readability matters - it's easier to debug: most debugger work line by line, or at least it's a convinient way to use them. If you use instruction per instruction, usually you have to read assember/bytecode to get the current instruction - profilers computes statistics per line,not per instruction (it's also the case for tracemalloc) - tracebacks only give the line number, not the column - etc. So I now prefer more verbise code even it is longer to write and may look less efficient. > Same again, multiple ** avoids construction of an > itermediate dict. Again, is it the case in the implementation. It may be possible to modify CPython to really avoid a temporary dict (at least for some kind of Python functions), but it would a large refactoring. Usually, if an operation is not efficient, it's not implement, so users don't try to use it and may even try to write their code differently (to avoid the performance issue). (But slow operations exist like list.remove.) Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Feb 10 09:11:34 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 10 Feb 2015 10:11:34 +0200 Subject: [Python-Dev] (no subject) In-Reply-To: <54D9678A.1070108@stoneleaf.us> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D9678A.1070108@stoneleaf.us> Message-ID: On 10.02.15 04:06, Ethan Furman wrote: > return func(*(args + fargs), **{**keywords, **fkeywords}) We don't use [*args, *fargs] for concatenating lists, but args + fargs. Why not use "+" or "|" operators for merging dicts? From solipsis at pitrou.net Tue Feb 10 09:27:50 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Feb 2015 09:27:50 +0100 Subject: [Python-Dev] (no subject) References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D99F53.90506@canterbury.ac.nz> Message-ID: <20150210092750.760fb2df@fsol> On Tue, 10 Feb 2015 19:04:03 +1300 Greg Ewing wrote: > Donald Stufft wrote: > > > > perhaps a better > > solution is to simply make it so that something like ``a_list + > > an_iterable`` is valid and the iterable would just be consumed and +?d > > onto the list. > > I don't think I like the asymmetry that this would > introduce into + on lists. Currently > > [1, 2, 3] + (4, 5, 6) > > is an error because it's not clear whether the > programmer intended the result to be a list or > a tuple. >>> bytearray(b"a") + b"bc" bytearray(b'abc') >>> b"a" + bytearray(b"bc") b'abc' It's quite convenient. In many contexts lists and tuples are quite interchangeable (for example when unpacking). Regards Antoine. From solipsis at pitrou.net Tue Feb 10 09:30:02 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Feb 2015 09:30:02 +0100 Subject: [Python-Dev] (no subject) References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D9678A.1070108@stoneleaf.us> Message-ID: <20150210093002.5021f779@fsol> On Mon, 09 Feb 2015 18:06:02 -0800 Ethan Furman wrote: > On 02/09/2015 05:14 PM, Victor Stinner wrote: > > > > def partial(func, *args, **keywords): > > def newfunc(*fargs, **fkeywords): > > return func(*(args + fargs), **keywords, **fkeywords) > > ... > > return newfunc > > > > The new code behaves differently since Neil said that an error is > > raised if fkeywords and keywords have keys in common. By the way, this > > must be written in the PEP. > > > That line should read > > return func(*(args + fargs), **{**keywords, **fkeywords}) > > to avoid the duplicate key error and keep the original functionality. While losing readability. What's the point exactly? One line over 112055 (as shown by Victor) can be collapsed away? Wow, that's sure gonna change Python programming in a massively beneficial way... Regards Antoine. From tjreedy at udel.edu Tue Feb 10 10:22:48 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 10 Feb 2015 04:22:48 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: On 2/9/2015 7:29 PM, Neil Girdhar wrote: > For some reason I can't seem to reply using Google groups, which is is > telling "this is a read-only mirror" (anyone know why?) I presume spam prevention. Most spam on python-list comes from the read-write GG mirror. -- Terry Jan Reedy From p.f.moore at gmail.com Tue Feb 10 10:33:42 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 09:33:42 +0000 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: On 10 February 2015 at 00:29, Neil Girdhar wrote: >> > function(**kw_arguments, **more_arguments) >> If the key "key1" is in both dictionaries, more_arguments wins, right? > > > There was some debate and it was decided that duplicate keyword arguments > would remain an error (for now at least). If you want to merge the > dictionaries with overriding, then you can still do: > > function(**{**kw_arguments, **more_arguments}) > > because **-unpacking in dicts overrides as you guessed. Eww. Seriously, function(**{**kw_arguments, **more_arguments}) feels more like a Perl "executable line noise" construct than anything I'd ever want to see in Python. And taking something that doesn't work and saying you can make it work by wrapping **{...} round it just seems wrong. Paul From p.f.moore at gmail.com Tue Feb 10 10:39:23 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 09:39:23 +0000 Subject: [Python-Dev] (no subject) In-Reply-To: <6A96EBBA-D2EC-4E2C-B4B7-BD0FF1D42DEF@stufft.io> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <6A96EBBA-D2EC-4E2C-B4B7-BD0FF1D42DEF@stufft.io> Message-ID: On 10 February 2015 at 01:48, Donald Stufft wrote: > I am really really -1 on the comprehension syntax. [... omitting because gmail seems to have messed up the quoting ...] > I don?t think * means ?loop? anywhere else in Python and I would never > ?guess? that [*item for item in iterable] meant that. It?s completely non > intuitive. Anywhere else you see *foo it?s unpacking a tuple not making an > inner loop. That means that anywhere else in Python *item is the same thing > as item[0], item[1], item[2], ?, but this PEP makes it so just inside of a > comprehension it actually means ?make a second, inner loop? instead of what > I think anyone who has learned that syntax would expect, which is it should > be equivalent to [(item[0], item[1], item[2], ?) for item in iterable]. I agree completely with Donald here. The comprehension syntax has consistently been the part of the proposal that has resulted in confused questions from reviewers, and I don't think it's at all intuitive. Is it allowable to vote on parts of the PEP separately? If not, then the comprehension syntax is enough for me to reject the whole proposal. If we can look at parts in isolation, I'm OK with saying -1 to the comprehension syntax and then we can look at whether the other parts of the PEP add enough to be worth it (the comprehension side is enough of a distraction that I haven't really considered the other bits yet). Paul From p.f.moore at gmail.com Tue Feb 10 13:38:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 12:38:25 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows Message-ID: tl;dr - Having a shared per-user scripts directory on Windows (%APPDATA%/Python/Scripts) causes a number of potential issues when users install the same package in multiple Python versions. I'd like to suggest that this be changed to a versioned directory (%APPDATA%/PythonXY/Scripts) to match all other Python directories on Windows. This issue originally started as a discussion on the pip tracker around making --user the default for pip. See https://github.com/pypa/pip/issues/1668 for details. However, because the user scripts directory is shared across all Python versions, this poses a number of issues: 1. If we "pip install" something like pytest, which version gets the name "py.test.exe" works on a "last install wins" basis, which is not ideal. 2. The above behaviour (and more generally the shared site scripts directory) does not respect the user's "Make this my default Python" choice from the installer, in potentially surprising ways. 3. Uninstalling a package which has been installed in 2 pythons will likely fail, because either the checksum of an installed script does not match what's in the RECORD file, or because a file that should be present has already been uninstalled via a different Python version [1]. I suggest that the per-user scripts directory should be versioned, in exactly the same way as every other Python-related directory on Windows. So we would have %APPDATA%\PythonXY\Scripts alongside %APPDATA%\PythonXY\site-packages. This is a familiar pattern for Windows users, and should not cause significant confusion. It would be good to get this added for Python 3.5, *before* making user installs the default (needed for 3.5 as the default install location for 3.5 will need admin privileges to write). Some additional points: 1. This is a Windows-specific proposal, because Unix has always had scripts installed in a common location (e.g., /usr/local/bin) and commonly-understood solutions already exist on that platform (versioned command names, python -m, system package management, etc). However, these solutions would be a significant change for Windows users. 2. Packages which recommend a user install specifically so that their commands are available regardless of which Python is on PATH would have to find an alternative solution. I doubt there are many such packages, but this should be considered. 3. This proposal does not affect the Python launcher for Windows. People using the launcher exclusively may be currently using "py -m pip" (or whatever) to launch applications, in which case the change of user script directory won't affect them. 4. I am not aware of any tools that rely on the location of the user scripts directory. Such scripts should be using sysconfig in any case. 5. The problems mentioned exist in current Python versions, but haven't been flagged up, presumably because very few people use the user install location - precisely because it's not the default in pip. Changing the pip default would cause more people to encounter the issues, but I don't propose backporting this change, as it's not a bug fix. If it becomes a major issue for pip users on older versions, we can either reconsider this, or pip can decide on its own workaround. Comments? If this is acceptable, I would be willing to prepare a patch for Python 3.5 implementing this. Paul [1] Checking of checksums is not currently implemented, AFAIK, but it is the purpose of the checksums in the RECORD file, and could be implemented in future versions of pip. From ncoghlan at gmail.com Tue Feb 10 14:16:38 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Feb 2015 23:16:38 +1000 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: On 10 Feb 2015 19:24, "Terry Reedy" wrote: > > On 2/9/2015 7:29 PM, Neil Girdhar wrote: >> >> For some reason I can't seem to reply using Google groups, which is is >> telling "this is a read-only mirror" (anyone know why?) > > > I presume spam prevention. Most spam on python-list comes from the read-write GG mirror. There were also problems with Google Groups getting the reply-to headers wrong (so if someone flipped the mirror to read-only: thank you!) With any luck, we'll have a native web gateway later this year after Mailman 3 is released, so posting through Google Groups will be less desirable. Cheers, Nick. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Tue Feb 10 14:21:08 2015 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 10 Feb 2015 05:21:08 -0800 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: On Tue, Feb 10, 2015 at 1:33 AM, Paul Moore wrote: > On 10 February 2015 at 00:29, Neil Girdhar wrote: > >> > function(**kw_arguments, **more_arguments) > >> If the key "key1" is in both dictionaries, more_arguments wins, right? > > > > > > There was some debate and it was decided that duplicate keyword arguments > > would remain an error (for now at least). If you want to merge the > > dictionaries with overriding, then you can still do: > > > > function(**{**kw_arguments, **more_arguments}) > > > > because **-unpacking in dicts overrides as you guessed. > > Eww. Seriously, function(**{**kw_arguments, **more_arguments}) feels > more like a Perl "executable line noise" construct than anything I'd > ever want to see in Python. And taking something that doesn't work and > saying you can make it work by wrapping **{...} round it just seems > wrong. > +1 to this and similar reasoning I find the syntax proposed in PEP 448 incredibly obtuse, and I don't think it's worth it. Python has never placed terseness of expression as its primary goal, but this is mainly what the PEP is aiming at. -1 on the PEP for me, at least in its current form. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Feb 10 14:23:43 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Feb 2015 14:23:43 +0100 Subject: [Python-Dev] (no subject) References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: <20150210142343.5e66e5c0@fsol> On Tue, 10 Feb 2015 23:16:38 +1000 Nick Coghlan wrote: > On 10 Feb 2015 19:24, "Terry Reedy" wrote: > > > > On 2/9/2015 7:29 PM, Neil Girdhar wrote: > >> > >> For some reason I can't seem to reply using Google groups, which is is > >> telling "this is a read-only mirror" (anyone know why?) > > > > > > I presume spam prevention. Most spam on python-list comes from the > read-write GG mirror. > > There were also problems with Google Groups getting the reply-to headers > wrong (so if someone flipped the mirror to read-only: thank you!) > > With any luck, we'll have a native web gateway later this year after > Mailman 3 is released, so posting through Google Groups will be less > desirable. There is already a Web and NNTP gateway with Gmane: http://news.gmane.org/gmane.comp.python.devel No need to rely on Google's mediocre services. Regards Antoine. From ncoghlan at gmail.com Tue Feb 10 14:26:19 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Feb 2015 23:26:19 +1000 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <6A96EBBA-D2EC-4E2C-B4B7-BD0FF1D42DEF@stufft.io> Message-ID: On 10 Feb 2015 19:41, "Paul Moore" wrote: > > I agree completely with Donald here. The comprehension syntax has > consistently been the part of the proposal that has resulted in > confused questions from reviewers, and I don't think it's at all > intuitive. > > Is it allowable to vote on parts of the PEP separately? If not, then > the comprehension syntax is enough for me to reject the whole > proposal. If we can look at parts in isolation, I'm OK with saying -1 > to the comprehension syntax and then we can look at whether the other > parts of the PEP add enough to be worth it (the comprehension side is > enough of a distraction that I haven't really considered the other > bits yet). It occurs to me that the PEP effectively changes the core of a generator expression from "yield x" to "yield from x" if the tuple expansion syntax is used. If we rejected the "yield *x" syntax for standalone yield expressions, I don't think it makes sense to now add it for generator expressions. So I guess that adds me to the -1 camp on the comprehension/generator expression part of the story - it doesn't make things all that much easier to write than the relevant nested loop, and it makes them notably harder to read. I haven't formed an opinion on the rest of the PEP yet, as it's been a while since I read the full text. I'll read through the latest version tomorrow. Regards, Nick. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 10 14:34:57 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Feb 2015 23:34:57 +1000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: On 10 Feb 2015 22:40, "Paul Moore" wrote: > > tl;dr - Having a shared per-user scripts directory on Windows > (%APPDATA%/Python/Scripts) causes a number of potential issues when > users install the same package in multiple Python versions. I'd like > to suggest that this be changed to a versioned directory > (%APPDATA%/PythonXY/Scripts) to match all other Python directories on > Windows. > > This issue originally started as a discussion on the pip tracker > around making --user the default for pip. See > https://github.com/pypa/pip/issues/1668 for details. For the reasons Paul gives, I'm personally +1 on making the change. I was actually surprised it didn't already work that way, given that almost everything else on Windows is version specific. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Tue Feb 10 14:37:59 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 10 Feb 2015 13:37:59 +0000 Subject: [Python-Dev] (no subject) In-Reply-To: <20150210142343.5e66e5c0@fsol> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <20150210142343.5e66e5c0@fsol> Message-ID: On 10/02/2015 13:23, Antoine Pitrou wrote: > On Tue, 10 Feb 2015 23:16:38 +1000 > Nick Coghlan wrote: >> On 10 Feb 2015 19:24, "Terry Reedy" wrote: >>> >>> On 2/9/2015 7:29 PM, Neil Girdhar wrote: >>>> >>>> For some reason I can't seem to reply using Google groups, which is is >>>> telling "this is a read-only mirror" (anyone know why?) >>> >>> >>> I presume spam prevention. Most spam on python-list comes from the >> read-write GG mirror. >> >> There were also problems with Google Groups getting the reply-to headers >> wrong (so if someone flipped the mirror to read-only: thank you!) >> >> With any luck, we'll have a native web gateway later this year after >> Mailman 3 is released, so posting through Google Groups will be less >> desirable. > > There is already a Web and NNTP gateway with Gmane: > http://news.gmane.org/gmane.comp.python.devel > > No need to rely on Google's mediocre services. > > Regards > > Antoine. > Highly recommended as effectively zero spam. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From p.f.moore at gmail.com Tue Feb 10 14:40:06 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 13:40:06 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: On 10 February 2015 at 13:34, Nick Coghlan wrote: > I was actually surprised it didn't already work that way, given that almost > everything else on Windows is version specific. That's actually a point I missed making. While I wouldn't class myself as "typical", when I went looking for the user-scripts directory on my machine, I automatically looked in %APPDATA%\PythonXY. I suspect that many people would assume this was how it worked anyway, and be surprised by the current behaviour. Paul From Steve.Dower at microsoft.com Tue Feb 10 15:13:52 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 10 Feb 2015 14:13:52 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: , Message-ID: I was also surprised as I was working on the installer, so +1 on changing it. The installer will also need some changes to add it to PATH, which should be fairly straightforward, but ping me if you need/want pointers. (It's checked in now, so I consider it fair game for anyone who wants to change it.) Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Paul Moore Sent: ?2/?10/?2015 5:40 To: Nick Coghlan Cc: Python Dev Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows On 10 February 2015 at 13:34, Nick Coghlan wrote: > I was actually surprised it didn't already work that way, given that almost > everything else on Windows is version specific. That's actually a point I missed making. While I wouldn't class myself as "typical", when I went looking for the user-scripts directory on my machine, I automatically looked in %APPDATA%\PythonXY. I suspect that many people would assume this was how it worked anyway, and be surprised by the current behaviour. Paul _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 10 15:41:43 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 14:41:43 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: On 10 February 2015 at 14:13, Steve Dower wrote: > The installer will also need some changes to add it to PATH, which should be > fairly straightforward, but ping me if you need/want pointers. (It's checked > in now, so I consider it fair game for anyone who wants to change it.) I was sort of hoping you'd be doing that independently, but never mind :-) I'll give it a go and shout if I get stuck. Paul From p.f.moore at gmail.com Tue Feb 10 15:43:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 14:43:33 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: On 10 February 2015 at 14:13, Steve Dower wrote: > I was also surprised as I was working on the installer, so +1 on changing > it. On a procedural note, does this require a change to the PEP (assuming it's generally acceptable)? Or is a patch to the code and docs sufficient? Paul From Steve.Dower at microsoft.com Tue Feb 10 16:41:45 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 10 Feb 2015 15:41:45 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: , Message-ID: One of my main engineering concerns is lack of shared knowledge a.k.a. the truck factor (not CPython specific, btw, just every project I work on). So I'm always keen to get multiple people working on new code asap. I also prefer installers to be treated as part of the product and updated with whatever changes affect it, though that basically never happens in my experience :) I assume the PEP will need updating if it specifies the exact path, and the installer readme may have a reference too. Presumably it'll only affect 3.5, as previous versions wouldn't use the directory for anything other than installing scripts. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Paul Moore Sent: ?2/?10/?2015 6:41 To: Steve Dower Cc: Nick Coghlan; Python Dev Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows On 10 February 2015 at 14:13, Steve Dower wrote: > The installer will also need some changes to add it to PATH, which should be > fairly straightforward, but ping me if you need/want pointers. (It's checked > in now, so I consider it fair game for anyone who wants to change it.) I was sort of hoping you'd be doing that independently, but never mind :-) I'll give it a go and shout if I get stuck. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 10 17:15:18 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 16:15:18 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: On 10 February 2015 at 12:38, Paul Moore wrote: > tl;dr - Having a shared per-user scripts directory on Windows > (%APPDATA%/Python/Scripts) causes a number of potential issues when > users install the same package in multiple Python versions. I'd like > to suggest that this be changed to a versioned directory > (%APPDATA%/PythonXY/Scripts) to match all other Python directories on > Windows. Making the full detail explicit (based on a comment made on the pip issue), the default PATH should match the behaviour of sys.path (so that "pip" and "python -m pip" always execute the same thing). So we have C:\PythonXY before %APPDATA%\PythonXY\Scripts before C:\PythonXY\Scripts (matching stdlib before user site-packages before system site-packages). And user site-packages is versioned because everything else is. The PATH setting is purely an installer change, of course - only versioning the user scripts directory affects the core. Paul From martin at v.loewis.de Tue Feb 10 17:17:31 2015 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 10 Feb 2015 17:17:31 +0100 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: , Message-ID: <54DA2F1B.6040605@v.loewis.de> Am 10.02.15 um 16:41 schrieb Steve Dower: > One of my main engineering concerns is lack of shared knowledge a.k.a. the truck factor (not CPython specific, btw, just every project I work on). Wrt. the installer, I think this is a lost cause. IIUC, you aren't really getting familiar with msi.py, and you are the one who has the highest chances - so I was the only real maintainer of it. Likewise, the Wise installer was only maintained by Tim Peters. So chances are really high that you will be the only one making substantial changes to the WiX files, although some people might learn how to build the installer. Regards, Martin From Steve.Dower at microsoft.com Tue Feb 10 17:27:30 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 10 Feb 2015 16:27:30 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <54DA2F1B.6040605@v.loewis.de> References: , <54DA2F1B.6040605@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Am 10.02.15 um 16:41 schrieb Steve Dower: >> One of my main engineering concerns is lack of shared knowledge a.k.a. the >> truck factor (not CPython specific, btw, just every project I work on). > > Wrt. the installer, I think this is a lost cause. IIUC, you aren't really > getting familiar with msi.py, and you are the one who has the highest chances - > so I was the only real maintainer of it. > Likewise, the Wise installer was only maintained by Tim Peters. > > So chances are really high that you will be the only one making substantial > changes to the WiX files, although some people might learn how to build the > installer. Aww, come on. Let me have a little bit of hope ;) > Regards, > Martin From martin at v.loewis.de Tue Feb 10 17:19:55 2015 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 10 Feb 2015 17:19:55 +0100 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: <54DA2FAB.6050906@v.loewis.de> Am 10.02.15 um 15:43 schrieb Paul Moore: > On 10 February 2015 at 14:13, Steve Dower wrote: >> I was also surprised as I was working on the installer, so +1 on changing >> it. > > On a procedural note, does this require a change to the PEP (assuming > it's generally acceptable)? Or is a patch to the code and docs > sufficient? There should be a PEP if the change is likely controversial. In any case, there should be documentation of the rationale for the change, and the docs itself is the wrong place for it. The commit message could be the right place, but if the rationale is a bit more verbose, a bug tracker item would be right. All IMO, of course. Martin From ethan at stoneleaf.us Tue Feb 10 18:20:59 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 Feb 2015 09:20:59 -0800 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: <54DA3DFB.1010203@stoneleaf.us> On 02/10/2015 04:38 AM, Paul Moore wrote: > > tl;dr - Having a shared per-user scripts directory on Windows > (%APPDATA%/Python/Scripts) causes a number of potential issues when > users install the same package in multiple Python versions. I'd like > to suggest that this be changed to a versioned directory > (%APPDATA%/PythonXY/Scripts) to match all other Python directories on > Windows. +1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From Steve.Dower at microsoft.com Tue Feb 10 18:45:49 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 10 Feb 2015 17:45:49 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <54DA3DFB.1010203@stoneleaf.us> References: , <54DA3DFB.1010203@stoneleaf.us> Message-ID: <1423590351443.13167@microsoft.com> So I've enumerated the problems with PATH on Windows before (and my subsequent dislike for modifying it through the installer), but here's the dot-point summary for those who haven't seen it. * System-wide values always precede per-user values * Most-recently-installed values always precede existing values (within the system/user separation above) * It's impossible to specify per-user values from a system-wide installer * The system-wide value has a length limit that silently breaks PATH if you exceed it * It's difficult for users to modify PATH themselves (mostly due to bad UI in Windows) The py.exe launcher solves many of these problems, but does not help with installed scripts. It's trivially easy to end up in a place where this would fail: pip install --user spam spam The *only* case where this will work reliably (without manual user configuration) is where Python 3.5 is the only installed version on the machine, and it was installed by the current user just for them. As we've seen from earlier discussions, the main beneficiaries of having Python on PATH are those using the command-line. Most scripts are going to make assumptions or work unnecessarily hard to find the actual location of the Python version they need. My best idea for dealing with this is to add another script alongside the py.exe launcher called `activate-py`.[1] This script would take the same command-line version argument as py.exe and use it to update the current prompt's PATH correctly. Specifically, I expect the script would actually use py.exe to invoke the requested Python and get the paths directly from sysconfig, which will ensure that the order of precedence always matches. Users would need to run `activate-py`, `activate-py -2` or `activate-py -3.5` (etc.) when they start their session.[2] This parallels venv's `activate` command, though because this would be a global command I believe the `-py` suffix is necessary. The example above becomes: activate-py -3.5 pip install --user spam spam A `deactivate-py` command would also be added to revert the changes, though I don't expect that to be part of most workflows. Another benefit is that shell scripts could call `activate-py` rather than trying to detect whether Python is already on the path or requiring a PYTHON variable to be set. (I seem to write these scripts all the time and know that I'd benefit from it.) This is yet another attempt to try and change user behaviour, which I'm not thrilled about, but I'm really starting to feel that this is the best way out of a bad situation. It differs from the solutions used on Linux, though there may be some value in this approach there too. Thought/votes/suggestions? Cheers, Steve [1]: Actually multiple scripts to handle cmd.exe/PowerShell, but this is an implementation detail. Users would simply type `activate-py` and the shells know which one to use. [2]: We can easily add Start Menu shortcuts (e.g. "Python 3.5 (32-bit) Command Prompt") and run it for them, though that is not a critical part of this proposal. From p.f.moore at gmail.com Tue Feb 10 21:47:49 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 20:47:49 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: On 10 February 2015 at 12:38, Paul Moore wrote: > Comments? If this is acceptable, I would be willing to prepare a patch > for Python 3.5 implementing this. See http://bugs.python.org/issue23437 As yet untested, as I just realised I need to get Visual Studio 2015 installed to be able to build Python 3.5. I'll try to get that sorted out, but I thought it would be worth putting the patch up anyway - it's pretty simple. Paul From p.f.moore at gmail.com Tue Feb 10 21:50:22 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 20:50:22 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <1423590351443.13167@microsoft.com> References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: On 10 February 2015 at 17:45, Steve Dower wrote: > This is yet another attempt to try and change user behaviour, which I'm not thrilled > about, but I'm really starting to feel that this is the best way out of a bad situation. > It differs from the solutions used on Linux, though there may be some value in this > approach there too. Thought/votes/suggestions? I have to admit, I agree with you. But before the "Add Python to PATH" option was added to the installer, we repeatedly got requests for it and it was apparently a constant source of confusion for beginners. If we're going to revert that change, we'll need to be very careful how we present it. From p.f.moore at gmail.com Tue Feb 10 21:58:45 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Feb 2015 20:58:45 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: On 10 February 2015 at 20:50, Paul Moore wrote: > On 10 February 2015 at 17:45, Steve Dower wrote: >> This is yet another attempt to try and change user behaviour, which I'm not thrilled >> about, but I'm really starting to feel that this is the best way out of a bad situation. >> It differs from the solutions used on Linux, though there may be some value in this >> approach there too. Thought/votes/suggestions? > > I have to admit, I agree with you. But before the "Add Python to PATH" > option was added to the installer, we repeatedly got requests for it > and it was apparently a constant source of confusion for beginners. If > we're going to revert that change, we'll need to be very careful how > we present it. Drat - hit "Send" too soon. Also, the proposed scripts won't be on PATH, so users will have to run them using explicit paths, which is both inconvenient and poses a discoverability issue (Tools/scripts/win_add2path.py has been round for ages, but AFAIK hardly anyone uses it). I wonder if there's a way to get the launcher (py.exe) to help? For powershell, we could have "py --setpath" write the appropriate path setting command to stdout, and users could use "Invoke-Expression $(py --setpath)", for example. That's a bit clumsy, and doesn't work for cmd.exe, but maybe there's possibilities... Essentially, py.exe is the only command guaranteed to be on PATH for any install (or is it? for a user install, is it stuck somewhere in the user's APPDATA?) It's fair enough to say that installers shouldn't be in the business of messing with PATH, but nevertheless, people expect to run an installer and have the installed programs available for use. Command line utilities like Mercurial do it, so people will expect Python to. Unfortunately. Sorry, all problems, no real answers here. Paul From breamoreboy at yahoo.co.uk Tue Feb 10 22:04:58 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 10 Feb 2015 21:04:58 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: On 10/02/2015 20:47, Paul Moore wrote: > On 10 February 2015 at 12:38, Paul Moore wrote: >> Comments? If this is acceptable, I would be willing to prepare a patch >> for Python 3.5 implementing this. > > See http://bugs.python.org/issue23437 > > As yet untested, as I just realised I need to get Visual Studio 2015 > installed to be able to build Python 3.5. I'll try to get that sorted > out, but I thought it would be worth putting the patch up anyway - > it's pretty simple. > > Paul > Visual Studio 2013 is fine for building 3.5. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From Steve.Dower at microsoft.com Tue Feb 10 22:09:30 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 10 Feb 2015 21:09:30 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> , Message-ID: <1423602569971.8212@microsoft.com> Paul Moore wrote: > On 10 February 2015 at 20:50, Paul Moore wrote: >> On 10 February 2015 at 17:45, Steve Dower wrote: >>> This is yet another attempt to try and change user behaviour, which I'm not thrilled >>> about, but I'm really starting to feel that this is the best way out of a bad situation. >>> It differs from the solutions used on Linux, though there may be some value in this >>> approach there too. Thought/votes/suggestions? >> >> I have to admit, I agree with you. But before the "Add Python to PATH" >> option was added to the installer, we repeatedly got requests for it >> and it was apparently a constant source of confusion for beginners. If >> we're going to revert that change, we'll need to be very careful how >> we present it. > > Drat - hit "Send" too soon. > > Also, the proposed scripts won't be on PATH, so users will have to run > them using explicit paths, which is both inconvenient and poses a > discoverability issue (Tools/scripts/win_add2path.py has been round > for ages, but AFAIK hardly anyone uses it). I wonder if there's a way > to get the launcher (py.exe) to help? For powershell, we could have > "py --setpath" write the appropriate path setting command to stdout, > and users could use "Invoke-Expression $(py --setpath)", for example. > That's a bit clumsy, and doesn't work for cmd.exe, but maybe there's > possibilities... Essentially, py.exe is the only command guaranteed to > be on PATH for any install (or is it? for a user install, is it stuck > somewhere in the user's APPDATA?) (For a user install, I've guaranteed that the launcher will be added to the user's PATH. But a system-wide launcher will win.) win_add2path.py does the same thing as the installer, so it isn't a useful solution. The `py --setpath` is basically my idea with `activate-py`, though with the "Invoke-Expression" part handled in the script (and not actually using Invoke-Expression, but that's implementation details). If these are part of the launcher, then they'll always be on PATH, and if they're just thin wrappers around py.exe then they can support all the same versions of Python (in effect, all of them, though with the 32/64-bit per-user collisions that existed pre-3.5). > It's fair enough to say that installers shouldn't be in the business > of messing with PATH, but nevertheless, people expect to run an > installer and have the installed programs available for use. Command > line utilities like Mercurial do it, so people will expect Python to. > Unfortunately. And Python does, as long as that installed program is "py". The problem is that people expect "python" to be there and be the correct one (for a totally arbitrary definition of "correct") as well as pip, easy_install and anything installed by those two tools, whether they were installed for all users or just the current one. It's far more complex than a single app (especially when you realise that a system-wide install is effectively trying to make Python look like part of the OS - we only get that about 50% right now...). > Sorry, all problems, no real answers here. > Paul Yeah, same. `activate-py` is the best I've got. Cheers, Steve From ncoghlan at gmail.com Wed Feb 11 00:20:45 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 11 Feb 2015 09:20:45 +1000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <1423590351443.13167@microsoft.com> References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: On 11 Feb 2015 03:47, "Steve Dower" wrote: > > This is yet another attempt to try and change user behaviour, which I'm not thrilled about, but I'm really starting to feel that this is the best way out of a bad situation. It differs from the solutions used on Linux, though there may be some value in this approach there too. Thought/votes/suggestions? I don't think we have this problem solved on Linux either, we just have a variety of workarounds in place of varying degrees of clunkiness :) The thing I find attractive about this specific proposal is that we could potentially expand it to Linux as well through appropriate use of environment modules, and I expect that would also cover other POSIX systems. So if you wanted to draft up an initial PEP with a concrete proposal covering Windows systems, I'd be happy to tackle the task of getting the same design to work on Linux (and hopefully POSIX in general) using environment modules. Cheers, Nick. > > Cheers, > Steve > > > [1]: Actually multiple scripts to handle cmd.exe/PowerShell, but this is an implementation detail. Users would simply type `activate-py` and the shells know which one to use. > > [2]: We can easily add Start Menu shortcuts (e.g. "Python 3.5 (32-bit) Command Prompt") and run it for them, though that is not a critical part of this proposal. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Feb 11 05:49:01 2015 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 10 Feb 2015 23:49:01 -0500 Subject: [Python-Dev] [Python-checkins] cpython: Update copyright. In-Reply-To: <20150211043727.29530.29807@psf.io> References: <20150211043727.29530.29807@psf.io> Message-ID: <1423630141.237881.225973753.24ED9A94@webmail.messagingengine.com> On Tue, Feb 10, 2015, at 23:37, raymond.hettinger wrote: > https://hg.python.org/cpython/rev/7d826a6b92a1 > changeset: 94582:7d826a6b92a1 > user: Raymond Hettinger > date: Tue Feb 10 22:37:22 2015 -0600 > summary: > Update copyright. > > files: > Modules/_collectionsmodule.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c > --- a/Modules/_collectionsmodule.c > +++ b/Modules/_collectionsmodule.c > @@ -3,7 +3,7 @@ > > /* collections module implementation of a deque() datatype > Written and maintained by Raymond D. Hettinger > - Copyright (c) 2004-2014 Python Software Foundation. > + Copyright (c) 2004-2015 Python Software Foundation. > All rights reserved. > */ How about simply removing this line? The copyight years are well documented in the LICENSE and README.txt, and it would save you the trouble of keeping this up to date. From greg.ewing at canterbury.ac.nz Wed Feb 11 06:17:31 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 11 Feb 2015 18:17:31 +1300 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D99D48.7020700@canterbury.ac.nz> Message-ID: <54DAE5EB.2040509@canterbury.ac.nz> Donald Stufft wrote: > 1. The statement *item is roughly the same thing as (item[0], item[1], item[n]) No, it's not -- that would make it equivalent to tuple(item), which is not what it means in any of its existing usages. What it *is* roughly equivalent to is item[0], item[1], item[n] i.e. *without* the parens, whatever that means in the context concerned. In the context of a function call, it has the effect of splicing the sequence in as if you had written each item out as a separate expression. You do have a valid objection insofar as this currently has no meaning at all in a comprehension, i.e. this is a syntax error: [item[0], item[1], item[n] for item in items] So we would be giving a meaning to something that doesn't currently have a meaning, rather than changing an existing meaning, if you see what I mean. -- Greg From dcrs6000 at aceweb.com Wed Feb 11 06:19:43 2015 From: dcrs6000 at aceweb.com (Dwight) Date: Tue, 10 Feb 2015 21:19:43 -0800 Subject: [Python-Dev] Problem running ./python -m test -v test_whatever Message-ID: <54DAE66F.7090505@aceweb.com> An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Feb 11 06:34:15 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 11 Feb 2015 18:34:15 +1300 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D99B67.8040905@canterbury.ac.nz> Message-ID: <54DAE9D7.6010903@canterbury.ac.nz> Victor Stinner wrote: > Le 10 f?vr. 2015 06:48, "Greg Ewing" > a ?crit : > > It could potentially be a little more efficient by > > eliminating the construction of an intermediate list. > > Is it the case in the implementation? If it has to create a temporary > list/tuple, I will prefer to not use it. The function call machinery will create a new tuple for the positional args in any case. But if you manually combine your * args into a tuple before calling, there are *two* tuple allocations being done. Passing all the * args directly into the call would allow one of them to be avoided. Similarly for dicts and ** args. -- Greg From ianlee1521 at gmail.com Wed Feb 11 06:35:22 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Tue, 10 Feb 2015 21:35:22 -0800 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D9678A.1070108@stoneleaf.us> Message-ID: +1 for adding "+" or "|" operator for merging dicts. To me this operation: >>> {'x': 1, 'y': 2} + {'z': 3} {'x': 1, 'y': 2, 'z': 3} Is very clear. The only potentially non obvious case I can see then is when there are duplicate keys, in which case the syntax could just be defined that last setter wins, e.g.: >>> {'x': 1, 'y': 2} + {'x': 3} {'x': 3, 'y': 2} Which is analogous to the example: new_dict = dict1.copy() new_dict.update(dict2) ~ Ian Lee On Tue, Feb 10, 2015 at 12:11 AM, Serhiy Storchaka wrote: > On 10.02.15 04:06, Ethan Furman wrote: > >> return func(*(args + fargs), **{**keywords, **fkeywords}) >> > > We don't use [*args, *fargs] for concatenating lists, but args + fargs. > Why not use "+" or "|" operators for merging dicts? > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > ianlee1521%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Feb 11 06:45:40 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 11 Feb 2015 18:45:40 +1300 Subject: [Python-Dev] (no subject) In-Reply-To: <20150210092750.760fb2df@fsol> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D99F53.90506@canterbury.ac.nz> <20150210092750.760fb2df@fsol> Message-ID: <54DAEC84.2010501@canterbury.ac.nz> Antoine Pitrou wrote: >>>>bytearray(b"a") + b"bc" > > bytearray(b'abc') > >>>>b"a" + bytearray(b"bc") > > b'abc' > > It's quite convenient. It's a bit disconcerting that the left operand wins, rather than one of them being designated as the "wider" type, as occurs with many other operations on mixed types, e.g. int + float. In any case, these seem to be special-case combinations. It's not so promiscuous as to accept any old iterable on the right: >>> b"a" + [1,2,3] Traceback (most recent call last): File "", line 1, in TypeError: can't concat bytes to list >>> [1,2,3] + b"a" Traceback (most recent call last): File "", line 1, in TypeError: can only concatenate list (not "bytes") to list -- Greg From ncoghlan at gmail.com Wed Feb 11 07:12:40 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 11 Feb 2015 16:12:40 +1000 Subject: [Python-Dev] Binary concatenation (was Re: (no subject)) Message-ID: On 11 February 2015 at 15:45, Greg Ewing wrote: > Antoine Pitrou wrote: >>>>> >>>>> bytearray(b"a") + b"bc" >> >> >> bytearray(b'abc') >> >>>>> b"a" + bytearray(b"bc") >> >> >> b'abc' >> >> It's quite convenient. > > > It's a bit disconcerting that the left operand wins, > rather than one of them being designated as the > "wider" type, as occurs with many other operations on > mixed types, e.g. int + float. > > In any case, these seem to be special-case combinations. > It's not so promiscuous as to accept any old iterable > on the right: Binary concatenation accepts any bytes-like object (aka buffer API implementing type), and it's the fact that both bytes and bytearray interoperate with any bytes-like object that results in the "left operand wins" behaviour when you use them together. I don't believe we deliberately designed it that way, it's just an inherent consequence of having both types implement concatenation based on the buffer protocol rather than using any kind of type check. If we *had* tried to define one of the two as encompassing the other, then we would have also run into http://bugs.python.org/issue11477 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Wed Feb 11 12:22:22 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Feb 2015 12:22:22 +0100 Subject: [Python-Dev] (no subject) References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D99F53.90506@canterbury.ac.nz> <20150210092750.760fb2df@fsol> <54DAEC84.2010501@canterbury.ac.nz> Message-ID: <20150211122222.35a02ca0@fsol> On Wed, 11 Feb 2015 18:45:40 +1300 Greg Ewing wrote: > Antoine Pitrou wrote: > >>>>bytearray(b"a") + b"bc" > > > > bytearray(b'abc') > > > >>>>b"a" + bytearray(b"bc") > > > > b'abc' > > > > It's quite convenient. > > It's a bit disconcerting that the left operand wins, > rather than one of them being designated as the > "wider" type, as occurs with many other operations on > mixed types, e.g. int + float. There is no "wider" type here. This behaviour is perfectly logical. > In any case, these seem to be special-case combinations. No: >>> b"abc" + array.array("b", b"def") b'abcdef' >>> bytearray(b"abc") + array.array("b", b"def") bytearray(b'abcdef') Regards Antoine. From brett at python.org Wed Feb 11 15:04:53 2015 From: brett at python.org (Brett Cannon) Date: Wed, 11 Feb 2015 14:04:53 +0000 Subject: [Python-Dev] Problem running ./python -m test -v test_whatever References: <54DAE66F.7090505@aceweb.com> Message-ID: You might want to try asking on python-list at python.org to get a wider audience as you might find a fellow AIX user there who can help you out. On Wed Feb 11 2015 at 12:29:56 AM Dwight wrote: > Hi, > I am primarily a user; but since I can not get a newer version > of firefox for my system I have begun the very long process of > trying to build a newer version of firefox for my system. > I am using an IBM pSeries system running AIX 7.1. > I am using gcc and IBM ld. > All the modules I have built are being installed in a directory > called /opt/alinux. A lot of linux routines are stored in a directory > called /opt/freeware and of course IBM has some version of their > own which are installed in /usr... Currently there is only one thing > installed in /usr/local and that is clamscan. > I have built and installed the tcl.8.6.3, tkl.8.6.3 and python 2.7.9 > into > /opt/aluinux. > I am now trying to build and install python 3.4.2. So far I > have found a way to compile python successfully. There are > only three features missing (_sqlite3, ossaudiodev and spwd). > The configure command I ran was: > ./configure --prefix=/opt/alinux --exec-prefix=/opt/alinux > --enable-shared --with-system-ffi --enable-ipv6 \ > --with-tcltk-includes='-I/opt/alinux/include' > --with-tcltk-libs='-L/opt/alinux/lib' | tee MYconfig.log > > After running gmake test I found: > Ran 509 tests in 47.407s > FAILED (errors=2, skipped=8) > Ran 49 tests in 0.065s > FAILED (failures=2, skipped=1) > Ran 34 tests in 0.320s > FAILED (errors=2, skipped=6) > Ran 80 tests in 1.040s > FAILED (errors=2, skipped=20) > Ran 10 tests in 0.366s > FAILED (failures=1, skipped=2) > Ran 506 tests in 28.860s > FAILED (failures=6, errors=5, skipped=83) > Ran 97 tests in 21.921s > FAILED (failures=9, skipped=3) > I then tried to run ./python -m test -v test_whatever > and got the following error: > $ pwd > /home/surfer/DownLoadLFNs/HTML/NEWS/BuildFirefox/Python-3.4.2 > $ ls -la lib* > -rw-r--r-- 1 surfer Internet 19562608 Feb 10 20:02 libpython3.4m.a > -rwxr-xr-x 1 surfer Internet 13331408 Feb 10 20:02 libpython3.4m.so > $ ./python -m test -v test_ssl > exec(): 0509-036 Cannot load program ./python because of the following > errors: > 0509-150 Dependent module libpython3.4m.so could not be loaded. > 0509-022 Cannot load module libpython3.4m.so. > 0509-026 System error: A file or directory in the path name does > not exist. > I would really appreciate some help in determining what I am doing > wrong. > As I said in the beginning I am primarily a user and not a developer. > I can solve > some fairly simple problems; but that about it. > I am guessing that there is some kind of linking problem; but I do not > know > how to solve this problem. > I tried this: > $ > LDFLAGS=-L/home/surfer/DownLoadLFNs/HTML/NEWS/BuildFirefox/Python-3.4.2 > > $ export LDFLAGS > and got the same results. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokoproject at gmail.com Wed Feb 11 07:35:17 2015 From: gokoproject at gmail.com (John Wong) Date: Wed, 11 Feb 2015 01:35:17 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D9678A.1070108@stoneleaf.us> Message-ID: On Wed, Feb 11, 2015 at 12:35 AM, Ian Lee wrote: > +1 for adding "+" or "|" operator for merging dicts. To me this operation: > > >>> {'x': 1, 'y': 2} + {'z': 3} > {'x': 1, 'y': 2, 'z': 3} > > Is very clear. The only potentially non obvious case I can see then is > when there are duplicate keys, in which case the syntax could just be > defined that last setter wins, e.g.: > > >>> {'x': 1, 'y': 2} + {'x': 3} > {'x': 3, 'y': 2} > > Which is analogous to the example: > > new_dict = dict1.copy() > new_dict.update(dict2) > > > Well looking at just list a + b yields new list a += b yields modified a then there is also .extend in list. etc. so do we want to follow list's footstep? I like + because + is more natural to read. Maybe this needs to be a separate thread. I am actually amazed to remember dict + dict is not possible... there must be a reason (performance??) for this... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianlee1521 at gmail.com Wed Feb 11 08:24:33 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Tue, 10 Feb 2015 23:24:33 -0800 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D9678A.1070108@stoneleaf.us> Message-ID: I split off a separate thread on python-ideas [1] specific to the idea of introducing "+" and "+=" operators on a dict. [1] https://mail.python.org/pipermail/python-ideas/2015-February/031748.html ~ Ian Lee On Tue, Feb 10, 2015 at 10:35 PM, John Wong wrote: > > > On Wed, Feb 11, 2015 at 12:35 AM, Ian Lee wrote: > >> +1 for adding "+" or "|" operator for merging dicts. To me this operation: >> >> >>> {'x': 1, 'y': 2} + {'z': 3} >> {'x': 1, 'y': 2, 'z': 3} >> >> Is very clear. The only potentially non obvious case I can see then is >> when there are duplicate keys, in which case the syntax could just be >> defined that last setter wins, e.g.: >> >> >>> {'x': 1, 'y': 2} + {'x': 3} >> {'x': 3, 'y': 2} >> >> Which is analogous to the example: >> >> new_dict = dict1.copy() >> new_dict.update(dict2) >> >> >> Well looking at just list > a + b yields new list > a += b yields modified a > then there is also .extend in list. etc. > > so do we want to follow list's footstep? I like + because + is more > natural to read. Maybe this needs to be a separate thread. I am actually > amazed to remember dict + dict is not possible... there must be a reason > (performance??) for this... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Wed Feb 11 20:38:40 2015 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 11 Feb 2015 20:38:40 +0100 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: On 02/10/2015 10:33 AM, Paul Moore wrote: > On 10 February 2015 at 00:29, Neil Girdhar wrote: >>> > function(**kw_arguments, **more_arguments) >>> If the key "key1" is in both dictionaries, more_arguments wins, right? >> >> >> There was some debate and it was decided that duplicate keyword arguments >> would remain an error (for now at least). If you want to merge the >> dictionaries with overriding, then you can still do: >> >> function(**{**kw_arguments, **more_arguments}) >> >> because **-unpacking in dicts overrides as you guessed. > > Eww. Seriously, function(**{**kw_arguments, **more_arguments}) feels > more like a Perl "executable line noise" construct than anything I'd > ever want to see in Python. And taking something that doesn't work and > saying you can make it work by wrapping **{...} round it just seems > wrong. I don't think people would want to write the above. I like the "sequence and dict flattening" part of the PEP, mostly because it is consistent and should be easy to understand, but the comprehension syntax enhancements seem to be bad for readability and "comprehending" what the code does. The call syntax part is a mixed bag: on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above). Georg From greg.ewing at canterbury.ac.nz Wed Feb 11 20:39:42 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 12 Feb 2015 08:39:42 +1300 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54D9678A.1070108@stoneleaf.us> Message-ID: <54DBAFFE.2050902@canterbury.ac.nz> John Wong wrote: > > I am actually > amazed to remember dict + dict is not possible... there must be a reason > (performance??) for this... I think it's mainly because there is no obviously correct answer to the question of what to do about duplicate keys. -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 11 21:15:46 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 12 Feb 2015 09:15:46 +1300 Subject: [Python-Dev] (no subject) In-Reply-To: References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> Message-ID: <54DBB872.2080409@canterbury.ac.nz> Georg Brandl wrote: > The call syntax part is a mixed bag: on the one hand it is nice to be > consistent with the extended possibilities in literals (flattening), > but on the other hand there would be small but annoying inconsistencies > anyways (e.g. the duplicate kwarg case above). That inconsistency already exists -- duplicate keys are allowed in dict literals but not calls: >>> {'a':1, 'a':2} {'a': 2} -- Greg From theller at ctypes.org Thu Feb 12 09:05:46 2015 From: theller at ctypes.org (Thomas Heller) Date: Thu, 12 Feb 2015 09:05:46 +0100 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <1423590351443.13167@microsoft.com> References: , <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: Am 10.02.2015 um 18:45 schrieb Steve Dower: > As we've seen from earlier discussions, the main beneficiaries of > having Python on PATH are those using the command-line. Most scripts > are going to make assumptions or work unnecessarily hard to find the > actual location of the Python version they need. Maybe I'm more or less alone with the way I work, but I don't like python.exe on my PATH (and py.exe alloes me to do this). I start python scripts from the command line either with 'script.py' or 'py -3.4 script.py' or 'py -2.7 script.py'. I even like running scripts in this way: 'py -2.7 -m pip install whatever'. However, this only works when the script also can be started as package. Some packages allow this, some don't. Could not py.exe be extended so that it allows starting scripts in a somewhat similar way? 'py-script -2.7 myscript foo bar baz' ??? Which would execute the script myscript.exe, myscript.bat, myscript.py, myscript.cmd or whatever is in the Scripts directory on the Python 2.7 installation, without changing PATH presistently? Thanks, Thomas From Steve.Dower at microsoft.com Thu Feb 12 15:26:55 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 12 Feb 2015 14:26:55 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: , <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com>, Message-ID: You're not alone, that's exactly how I do it too. I'm writing up a PEP for the activate-py command I suggested earlier that will temporarily set your PATH. If that goes well then we'll be able to ship that with the py.exe launcher (I'm particularly excited for batch files to be able to use it). Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Thomas Heller Sent: ?2/?12/?2015 0:03 To: python-dev at python.org Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows Am 10.02.2015 um 18:45 schrieb Steve Dower: > As we've seen from earlier discussions, the main beneficiaries of > having Python on PATH are those using the command-line. Most scripts > are going to make assumptions or work unnecessarily hard to find the > actual location of the Python version they need. Maybe I'm more or less alone with the way I work, but I don't like python.exe on my PATH (and py.exe alloes me to do this). I start python scripts from the command line either with 'script.py' or 'py -3.4 script.py' or 'py -2.7 script.py'. I even like running scripts in this way: 'py -2.7 -m pip install whatever'. However, this only works when the script also can be started as package. Some packages allow this, some don't. Could not py.exe be extended so that it allows starting scripts in a somewhat similar way? 'py-script -2.7 myscript foo bar baz' ??? Which would execute the script myscript.exe, myscript.bat, myscript.py, myscript.cmd or whatever is in the Scripts directory on the Python 2.7 installation, without changing PATH presistently? Thanks, Thomas _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Feb 12 15:46:50 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Feb 2015 14:46:50 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: On 12 February 2015 at 08:05, Thomas Heller wrote: > Maybe I'm more or less alone with the way I work, but I don't like > python.exe on my PATH (and py.exe alloes me to do this). > I start python scripts from the command line either with 'script.py' > or 'py -3.4 script.py' or 'py -2.7 script.py'. This is how I worked for a long time after py.exe became available. But then I discovered virtualenv, and started using that, and needing to use 2 completely different command sets for "when I'm in a virtualenv" vs "when I'm not" became completely unusable. If py.exe detected when the environment variable VIRTUAL_ENV was set, and used that virtualenv as the default Python rather than the "system" python it normally used, this would be perfect. I think I'll write a PEP for this. Do people think it's a reasonable idea? (On that note, is there any relationship between the py.exe shipped with Python and the standalone version Vinay used to maintain but which now appears to be hosted at https://bitbucket.org/pypa/pylauncher? The two seem to be different, for example the pypa version registers pyz/pyzw extensions). Paul From Steve.Dower at microsoft.com Thu Feb 12 16:37:24 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 12 Feb 2015 15:37:24 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> , Message-ID: If venv's activate script sets it, I say go ahead and write it up. If it's just virtualenv, I'd rather not explicitly depend on it, especially since PEP 397's stated aim is file associations and not command line. I've been making changes to py.exe in hg.p.o, so I hope the standalone one is tracking. The msi for it as part of the official build can also standalone, so maybe we should merge the two? Top-posted from my Windows Phone ________________________________ From: Paul Moore Sent: ?2/?12/?2015 6:48 To: Thomas Heller Cc: Python Dev Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows On 12 February 2015 at 08:05, Thomas Heller wrote: > Maybe I'm more or less alone with the way I work, but I don't like > python.exe on my PATH (and py.exe alloes me to do this). > I start python scripts from the command line either with 'script.py' > or 'py -3.4 script.py' or 'py -2.7 script.py'. This is how I worked for a long time after py.exe became available. But then I discovered virtualenv, and started using that, and needing to use 2 completely different command sets for "when I'm in a virtualenv" vs "when I'm not" became completely unusable. If py.exe detected when the environment variable VIRTUAL_ENV was set, and used that virtualenv as the default Python rather than the "system" python it normally used, this would be perfect. I think I'll write a PEP for this. Do people think it's a reasonable idea? (On that note, is there any relationship between the py.exe shipped with Python and the standalone version Vinay used to maintain but which now appears to be hosted at https://bitbucket.org/pypa/pylauncher? The two seem to be different, for example the pypa version registers pyz/pyzw extensions). Paul _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Feb 12 17:22:30 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Feb 2015 16:22:30 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: On 12 February 2015 at 15:37, Steve Dower wrote: > If venv's activate script sets it, I say go ahead and write it up. If it's > just virtualenv, I'd rather not explicitly depend on it, especially since > PEP 397's stated aim is file associations and not command line. Yep, venv uses it too (see https://hg.python.org/cpython/file/9e10c4255277/Lib/venv/scripts/nt/activate.bat). As people like Thomas (and me, until this issue stopped me :-)) are using the launcher for command line use, I think it's fair to broaden the scope to make command line usage more convenient. I agree that PEP 397 was originally focused mainly on file associations, but I think it's worth looking beyond that now. > I've been making changes to py.exe in hg.p.o, so I hope the standalone one > is tracking. The msi for it as part of the official build can also > standalone, so maybe we should merge the two? Hmm, sadly I don't think it is. Originally the standalone one was (I believe) provided by Vinay for people using Pythons that didn't have it bundled, and to add features (such as the new file extensions) on a quicker timescale than Python releases. But since he passed it to the pypa umbrella I don't think he's been keeping the two in sync. I've copied him in case I'm completely wrong on this. Personally if we now have a standalone launcher MSI, I'd like to discontinue the external one completely, and officially publish the standalone launcher MSI on python.org as a service for users of older Pythons. There seems little reason to maintain 2 repos if we don't have to. We could merge in changes from the external repo before discontinuing it, but I'm not sure how controversial that would be (for example, would it need a PEP to include the 2 new extensions?) Paul From martin at v.loewis.de Thu Feb 12 17:29:04 2015 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 12 Feb 2015 17:29:04 +0100 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <1423590351443.13167@microsoft.com> References: , <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: <54DCD4D0.6020507@v.loewis.de> Am 10.02.15 um 18:45 schrieb Steve Dower: > So I've enumerated the problems with PATH on Windows before (and my subsequent dislike for modifying it through the installer) It's quite comforting to hear this - I was arguing against that addition for years (to the point of refusing to implement it). So you are certainly not alone here, either. Regards, Martin From Steve.Dower at microsoft.com Thu Feb 12 17:42:58 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 12 Feb 2015 16:42:58 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: Paul Moore wrote: > > On 12 February 2015 at 15:37, Steve Dower wrote: >> If venv's activate script sets it, I say go ahead and write it up. If >> it's just virtualenv, I'd rather not explicitly depend on it, >> especially since PEP 397's stated aim is file associations and not command > line. > > Yep, venv uses it too (see > https://hg.python.org/cpython/file/9e10c4255277/Lib/venv/scripts/nt/activate.bat). > > As people like Thomas (and me, until this issue stopped me :-)) are using the > launcher for command line use, I think it's fair to broaden the scope to make > command line usage more convenient. I agree that PEP > 397 was originally focused mainly on file associations, but I think it's worth > looking beyond that now. That's roughly the premise behind my activate-py proposal. Use `activate` to make a virtualenv/venv the default, and `activate-py [-x[.y[-32]]]` to make a non-venv the default. I have proof of concept implementations already for cmd.exe and PowerShell and it's fairly simple, but writing up the rationale is time consuming (and also the main point - it's worth having all the problems with PATH documented in one place). >> I've been making changes to py.exe in hg.p.o, so I hope the standalone >> one is tracking. The msi for it as part of the official build can also >> standalone, so maybe we should merge the two? > > Hmm, sadly I don't think it is. Originally the standalone one was (I > believe) provided by Vinay for people using Pythons that didn't have it bundled, > and to add features (such as the new file extensions) on a quicker timescale > than Python releases. But since he passed it to the pypa umbrella I don't think > he's been keeping the two in sync. I've copied him in case I'm completely wrong > on this. > > Personally if we now have a standalone launcher MSI, I'd like to discontinue the > external one completely, and officially publish the standalone launcher MSI on > python.org as a service for users of older Pythons. There seems little reason to > maintain 2 repos if we don't have to. We could merge in changes from the > external repo before discontinuing it, but I'm not sure how controversial that > would be (for example, would it need a PEP to include the 2 new extensions?) None of my installer changes so far have had a PEP, and only a few people have complained about that :) (it does have more documentation than I've ever written for an installer before though) IIRC, there was a PEP for executing ZIP files directly (2.6-era?), which I believe are the purpose of those extensions. If "py.exe spam.pyz" already works, I don't see any need for a PEP to add the association in the installer. Cheers, Steve > Paul From theller at ctypes.org Thu Feb 12 18:13:19 2015 From: theller at ctypes.org (Thomas Heller) Date: Thu, 12 Feb 2015 18:13:19 +0100 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: Am 12.02.2015 um 15:46 schrieb Paul Moore: > On 12 February 2015 at 08:05, Thomas Heller wrote: >> Maybe I'm more or less alone with the way I work, but I don't like >> python.exe on my PATH (and py.exe alloes me to do this). >> I start python scripts from the command line either with 'script.py' >> or 'py -3.4 script.py' or 'py -2.7 script.py'. > > This is how I worked for a long time after py.exe became available. > But then I discovered virtualenv, and started using that, and needing > to use 2 completely different command sets for "when I'm in a > virtualenv" vs "when I'm not" became completely unusable. I'm normally not using virtual environments, so my understanding might be wrong. Isn't it the case that in virtual envs you don't have these problems at all? The 'correct' python.exe is on the PATH as well as the Scripts directory: this is what activate.bat does. Thomas From ethan at stoneleaf.us Thu Feb 12 18:39:58 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Feb 2015 09:39:58 -0800 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: , <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: <54DCE56E.2080408@stoneleaf.us> On 02/12/2015 12:05 AM, Thomas Heller wrote: > Could not py.exe be extended so that it allows starting scripts in a > somewhat similar way? 'py-script -2.7 myscript foo bar baz' ??? > Which would execute the script myscript.exe, myscript.bat, myscript.py, > myscript.cmd or whatever is in the Scripts directory on the Python 2.7 > installation, without changing PATH presistently? I'm pretty sure py.exe should only start .py files, not .bat, .cmd, or .anything-else. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Thu Feb 12 19:51:56 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Feb 2015 18:51:56 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: On 12 February 2015 at 17:13, Thomas Heller wrote: >> This is how I worked for a long time after py.exe became available. >> But then I discovered virtualenv, and started using that, and needing >> to use 2 completely different command sets for "when I'm in a >> virtualenv" vs "when I'm not" became completely unusable. > > I'm normally not using virtual environments, so my understanding might > be wrong. Isn't it the case that in virtual envs you don't have these > problems at all? The 'correct' python.exe is on the PATH as well as the > Scripts directory: this is what activate.bat does. Yes, but that's essentially the problem. I don't want to use "py" to start Python, but when I'm in a virtualenv, have to switch to using "python". That was what I used to do, and I found that I was forever running "py" in a virtualenv and wondering why the packages I'd installed weren't available... Paul From p.f.moore at gmail.com Thu Feb 12 19:52:09 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Feb 2015 18:52:09 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <54DCE56E.2080408@stoneleaf.us> References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> Message-ID: On 12 February 2015 at 17:39, Ethan Furman wrote: > I'm pretty sure py.exe should only start .py files, not .bat, .cmd, or .anything-else. Agreed. Paul From p.f.moore at gmail.com Thu Feb 12 20:44:58 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Feb 2015 19:44:58 +0000 Subject: [Python-Dev] PEP 486: Make the Python Launcher aware of virtual environments Message-ID: On 12 February 2015 at 14:46, Paul Moore wrote: > If py.exe detected when the environment variable VIRTUAL_ENV was set, > and used that virtualenv as the default Python rather than the > "system" python it normally used, this would be perfect. > > I think I'll write a PEP for this. Do people think it's a reasonable idea? OK, here we go. https://www.python.org/dev/peps/pep-0486/ Reproduced inline below for ease of reading / commenting. It's also available on github at https://github.com/pfmoore/peps should people want to submit corrections or anything there. PEP: 486 Title: Make the Python Launcher aware of virtual environments Version: $Revision$ Last-Modified: $Date$ Author: Paul Moore Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Feb-2015 Python-Version: 3.5 Post-History: Abstract ======== The Windows installers for Python include a launcher that locates the correct Python interpreter to run (see PEP 397). However, the launcher is not aware of virtual environments (virtualenv [1]_ or PEP 405 based), and so cannot be used to run commands from the active virtualenv. This PEP proposes making the launcher "virtualenv aware". This means that when run without specifying an explicit Python interpreter to use, the launcher will use the currently active virtualenv, if any, before falling back to the configured default Python. Rationale ========= Windows users with multiple copies of Python installed need a means of selecting which one to use. The Python launcher provides this facility by means of a ``py`` command that can be used to run either a configured "default" Python or a specific interpreter, by means of command line arguments. So typical usage would be:: # Run the Python interactive interpreter py # Execute an installed module py -m pip install pytest py -m pytest When using virtual environments, the ``py`` launcher is unaware that a virtualenv is active, and will continue to use the system Python. So different command invocations are needed to run the same commands in a virtualenv:: # Run the Python interactive interpreter python # Execute an installed module (these could use python -m, # which is longer to type but is a little mopre similar to the # launcher approach) pip install pytest py.test Having to use different commands is is error-prone, and in many cases the error is difficult to spot immediately. The PEP proposes making the ``py`` command usable with virtual environments, so that the first form of command can be used in all cases. Implementation ============== Both ``virtualenv`` and the core ``venv`` module set an environment variable ``VIRTUAL_ENV`` when activating a virtualenv. This PEP proposes that the launcher checks for the ``VIRTUAL_ENV`` environment variable whenever it would run the "default" Python interpreter for the system (i.e., when no specific version flags such as ``py -2.7`` are used) and if present, run the Python interpreter for the virtualenv rather than the default system Python. The "default" Python interpreter referred to above is (as per PEP 397) either the latest version of Python installed on the system, or a version configured via the ``py.ini`` configuration file. When the user specifies an explicit Python version on the command line, this will always be used (as at present). Impact on Script Launching ========================== As well as interactive use, the launcher is used as the Windows file association for Python scripts. In that case, a "shebang" (``#!``) line at the start of the script is used to identify the interpreter to run. A fully-qualified path can be used, or a version-specific Python (``python3`` or ``python2``, or even ``python3.5``), or the generic ``python``, which means to use the default interpreter. With the proposed change, scripts that start with ``#!python`` (or one of its equivalent forms) will be run using an active virtualenv. This is a change in behaviour, although it will only affect users running scripts from the command line with a virtualenv activated. Under Unix, the ``#!/usr/bin/env python`` shebang line will use the version of Python found on ``$PATH``, whereas ``#!/usr/bin/python`` will use the system Python. In order to match Unix behaviour as closely as possible, it is proposed that the two shebang forms:: #!/usr/bin/env python #!python use an active virtualenv, if present, whereas the forms:: #!/usr/bin/python #!/usr/local/bin/python use *only* the default system Python, and ignore the ``VIRTUAL_ENV`` environment variable. Exclusions ========== The PEP makes no attempt to promote the use of the launcher for running Python on Windows. Most existing documentation assumes the user of ``python`` as the command to run Python, and (for example) ``pip`` to run an installed Python command. This documentation is not expected to change, and users who choose to manage their ``PATH`` environment variable can continue to use this form. The focus of this PEP is purely on allowing users who prefer to use the launcher when dealing with their system Python installations, to be able to continue to do so when using virtual environments. References ========== .. [1] https://virtualenv.pypa.io/ Copyright ========= This document has been placed in the public domain. From ethan at stoneleaf.us Thu Feb 12 20:47:23 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Feb 2015 11:47:23 -0800 Subject: [Python-Dev] PEP 486: Make the Python Launcher aware of virtual environments In-Reply-To: References: Message-ID: <54DD034B.2030907@stoneleaf.us> Looks well thought-out. +1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Thu Feb 12 20:50:54 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Feb 2015 19:50:54 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: On 12 February 2015 at 16:42, Steve Dower wrote: > None of my installer changes so far have had a PEP, and only a few people have complained about that :) (it does have more documentation than I've ever written for an installer before though) :-) You shouldn't bet on my judgement of what needs a PEP, I usually get it wrong... > IIRC, there was a PEP for executing ZIP files directly (2.6-era?), which I believe are the purpose of those extensions. If "py.exe spam.pyz" already works, I don't see any need for a PEP to add the association in the installer. Yes, "py spam.pyz" works fine. +1 on having the associations. Actually, I've just remembered, it's already in PEP 441, which hasn't been approved but which goes further and includes a stdlib tool to create pyz files. I'm not sure if that changes things at all... Paul From v+python at g.nevcal.com Thu Feb 12 20:44:16 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 12 Feb 2015 11:44:16 -0800 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <54DCE56E.2080408@stoneleaf.us> References: , <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> Message-ID: <54DD0290.9020108@g.nevcal.com> On 2/12/2015 9:39 AM, Ethan Furman wrote: > On 02/12/2015 12:05 AM, Thomas Heller wrote: > >> Could not py.exe be extended so that it allows starting scripts in a >> somewhat similar way? 'py-script -2.7 myscript foo bar baz' ??? >> Which would execute the script myscript.exe, myscript.bat, myscript.py, >> myscript.cmd or whatever is in the Scripts directory on the Python 2.7 >> installation, without changing PATH presistently? > I'm pretty sure py.exe should only start .py files, not .bat, .cmd, or .anything-else. > IIRC, it already can be configured to start other things. I think I even used it for that early on in Vinay's development cycle, just to prove it, although the things I used it for then were not things I'm still doing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Feb 12 21:11:49 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Feb 2015 12:11:49 -0800 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <54DD0290.9020108@g.nevcal.com> References: , <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DD0290.9020108@g.nevcal.com> Message-ID: <54DD0905.5080207@stoneleaf.us> On 02/12/2015 11:44 AM, Glenn Linderman wrote: > On 2/12/2015 9:39 AM, Ethan Furman wrote: >> On 02/12/2015 12:05 AM, Thomas Heller wrote: >> >>> Could not py.exe be extended so that it allows starting scripts in a >>> somewhat similar way? 'py-script -2.7 myscript foo bar baz' ??? >>> Which would execute the script myscript.exe, myscript.bat, myscript.py, >>> myscript.cmd or whatever is in the Scripts directory on the Python 2.7 >>> installation, without changing PATH presistently? >> I'm pretty sure py.exe should only start .py files, not .bat, .cmd, or .anything-else. >> > > IIRC, it already can be configured to start other things. I think I even used it for that early on in Vinay's > development cycle, just to prove it, although the things I used it for then were not things I'm still doing. I believe you are correct; however, as the PEP for the launcher stated [1] "use as a general-purpose (aka non-python) launcher is explicitly not supported". -- ~Ethan~ [1] or something to that effect -- basically, it may work, but you're on your own if you use it that way. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Thu Feb 12 21:23:15 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Feb 2015 20:23:15 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <54DD0905.5080207@stoneleaf.us> References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DD0290.9020108@g.nevcal.com> <54DD0905.5080207@stoneleaf.us> Message-ID: On 12 February 2015 at 20:11, Ethan Furman wrote: > I believe you are correct; however, as the PEP for the launcher stated [1] "use as a general-purpose (aka non-python) > launcher is explicitly not supported". Yes, I once used it to start Perl scripts, just because it appealed to the perverse instinct in me. Part of me wishes there had been a problem, so that I could be the one who raised a legitimate bug on bugs.python.org saying "Python encounters an error when running a Perl script" :-) Paul From breamoreboy at yahoo.co.uk Thu Feb 12 21:53:37 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Thu, 12 Feb 2015 20:53:37 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DD0290.9020108@g.nevcal.com> <54DD0905.5080207@stoneleaf.us> Message-ID: On 12/02/2015 20:23, Paul Moore wrote: > On 12 February 2015 at 20:11, Ethan Furman wrote: >> I believe you are correct; however, as the PEP for the launcher stated [1] "use as a general-purpose (aka non-python) >> launcher is explicitly not supported". > > Yes, I once used it to start Perl scripts, just because it appealed to > the perverse instinct in me. Part of me wishes there had been a > problem, so that I could be the one who raised a legitimate bug on > bugs.python.org saying "Python encounters an error when running a Perl > script" :-) > > Paul > Get thee behind me, Satan. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From vinay_sajip at yahoo.co.uk Thu Feb 12 22:44:02 2015 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 12 Feb 2015 21:44:02 +0000 (UTC) Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: <1480317723.4066010.1423777442838.JavaMail.yahoo@mail.yahoo.com> > I've been making changes to py.exe in hg.p.o, so I hope the standalone one > is tracking. The msi for it as part of the official build can also > standalone, so maybe we should merge the two? > Hmm, sadly I don't think it is. Originally the standalone one was (I > believe) provided by Vinay for people using Pythons that didn't have > it bundled, and to add features (such as the new file extensions) on a > quicker timescale than Python releases. But since he passed it to the > pypa umbrella I don't think he's been keeping the two in sync. I've > copied him in case I'm completely wrong on this. Thanks for copying me in - I haven't been tracking changes to the launcher in hg.p.o purely because I have been really busy with consultancy work for the past few months, leading to reduced time available for open-source contributions. I intend to try and keep things synced up where feasible and when time permits, and your email has given me the heads-up to look and see what's changed in the hg.p.o version. > Personally if we now have a standalone launcher MSI, I'd like to > discontinue the external one completely, and officially publish the > standalone launcher MSI on python.org as a service for users of older > Pythons. The standalone repo/installer is just there for older Pythons where it's not bundled (and also to allow trying out features independently of the Python release cycle, as you mentioned). If python.org will offer an unbundled MSI just to install the launcher on Pythons < 3.3, then I suppose my installer could be retired. I don't know if the hg.p.o. installer would be equivalent to the standalone one (e.g. when the standalone launcher is uninstalled, the user gets a chance to associate file extensions with one of the installed Pythons found on the system). Off to read the whole thread on python-dev now ... Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Thu Feb 12 22:47:46 2015 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 12 Feb 2015 21:47:46 +0000 (UTC) Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: <297479927.4072832.1423777666429.JavaMail.yahoo@mail.yahoo.com> ----- Original Message ----- From: Steve Dower > I believe are the purpose of those extensions. If "py.exe spam.pyz" > already works, I don't see any need for a PEP to add the association in > the installer. That would be my view, too. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Thu Feb 12 22:55:42 2015 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 12 Feb 2015 21:55:42 +0000 (UTC) Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: <1741111003.4062313.1423778142064.JavaMail.yahoo@mail.yahoo.com> > Actually, I've just remembered, it's already in PEP 441, > which hasn't been approved but which goes further and > includes a stdlib tool to create pyz files. I'm not sure > if that changes things at all... Actually, I developed the pyzzer tool (which Paul may remember - he made some helpful suggestions to improve it) which does a similar job to PEP 441's pyzaa and provides improved Windows support (some of which might be supplanted by py.exe support for .pyz/.pyzw, but I have cases where I can't rely on py.exe being present). Regards, Vinay Sajip From p.f.moore at gmail.com Thu Feb 12 23:36:24 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Feb 2015 22:36:24 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <1480317723.4066010.1423777442838.JavaMail.yahoo@mail.yahoo.com> References: <1480317723.4066010.1423777442838.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 12 February 2015 at 21:44, Vinay Sajip wrote: > Off to read the whole thread on python-dev now I'd also appreciate your views on the spin-off thread and PEP 486 ("Make the Python Launcher aware of virtual environments"). Paul From vinay_sajip at yahoo.co.uk Fri Feb 13 01:15:00 2015 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 13 Feb 2015 00:15:00 +0000 (UTC) Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: <368981772.4110302.1423786500432.JavaMail.yahoo@mail.yahoo.com> ----- Original Message ----- From: Paul Moore > I'd also appreciate your views on the spin-off thread and PEP 486 > ("Make the Python Launcher aware of virtual environments"). On an initial reading it seems quite reasonable. I'll sleep on it and see if any other things spring to mind. By the way, although I turned over the standalone launcher to pypa, I forked that repo, and do development on my fork [1]. I remembered that I added a feature (with the help of Pawel Jasinski) to allow e.g. py -ipy to pick a command 'ipy' configured in the .ini file (this is used to launch IronPython, but it could be used for other things too). In addition, my version has some bugfixes (e.g. use of iswspace instead of isspace, which is wrong). Steve's changes seem to just be related to adding IP_VERSION - I'm not sure what that's about and I haven't looked into it yet; I just skimmed the differences, and didn't see anything major in launcher.c (I compared with the version in the default branch of cpython). Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/pylauncher From Steve.Dower at microsoft.com Fri Feb 13 01:20:44 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 13 Feb 2015 00:20:44 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <368981772.4110302.1423786500432.JavaMail.yahoo@mail.yahoo.com> References: <368981772.4110302.1423786500432.JavaMail.yahoo@mail.yahoo.com> Message-ID: Vinay Sajip wrote: > By the way, although I turned over the standalone launcher to pypa, I forked > that repo, and do development on my fork [1]. I remembered that I added a > feature (with the help of Pawel Jasinski) to allow e.g. py -ipy to pick a > command 'ipy' configured in the .ini file (this is used to launch IronPython, > but it could be used for other things too). In addition, my version has some > bugfixes (e.g. use of iswspace instead of isspace, which is wrong). Steve's > changes seem to just be related to adding IP_VERSION - I'm not sure what that's > about and I haven't looked into it yet; I just skimmed the differences, and > didn't see anything major in launcher.c (I compared with the version in the > default branch of cpython). I thought I did more than that, but maybe that was attempting to make things work :) The default branch is the most up to date. The big change is that the registry key for 32-bit Python now includes the "-32" suffix, so that you can install 32-bit and 64-bit versions per-user at the same time and we don't have a collision. The launcher needed some updates to handle that, since it was assuming that registry key names would never be more than three characters long. Probably everything else I did was elsewhere in Python or in the installer. (Btw, it sounds like your installer will work better as a standalone tool, though my MSI has the advantage that it won't collide with a Python 3.5+ installation. I'm not fussed about whose 'wins' though.) Cheers, Steve > Regards, > > Vinay Sajip > > [1] https://bitbucket.org/vinay.sajip/pylauncher From ethan at stoneleaf.us Fri Feb 13 01:36:51 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Feb 2015 16:36:51 -0800 Subject: [Python-Dev] subclassing builtin data structures Message-ID: <54DD4723.7030003@stoneleaf.us> I suspect the last big hurdle to making built-in data structures nicely subclassable is the insistence of such types to return new instances as the base class instead of the derived class. In case that wasn't clear ;) --> class MyInt(int): ... def __repr__(self): ... return 'MyInt(%d)' % self ... --> m = MyInt(42) --> m MyInt(42) --> m + 1 43 --> type(m+1) Besides the work it would take to rectify this, I imagine the biggest hurdle would be the performance hit in always looking up the type of self. Has anyone done any preliminary benchmarking? Are there other concerns? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From guido at python.org Fri Feb 13 01:55:56 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 12 Feb 2015 16:55:56 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DD4723.7030003@stoneleaf.us> References: <54DD4723.7030003@stoneleaf.us> Message-ID: On Thu, Feb 12, 2015 at 4:36 PM, Ethan Furman wrote: > I suspect the last big hurdle to making built-in data structures nicely > subclassable is the insistence of such types to > return new instances as the base class instead of the derived class. > > In case that wasn't clear ;) > > --> class MyInt(int): > ... def __repr__(self): > ... return 'MyInt(%d)' % self > ... > --> m = MyInt(42) > --> m > MyInt(42) > --> m + 1 > 43 > --> type(m+1) > > > Besides the work it would take to rectify this, I imagine the biggest > hurdle would be the performance hit in always > looking up the type of self. Has anyone done any preliminary > benchmarking? Are there other concerns? > Actually, the problem is that the base class (e.g. int) doesn't know how to construct an instance of the subclass -- there is no reason (in general) why the signature of a subclass constructor should match the base class constructor, and it often doesn't. So this is pretty much a no-go. It's not unique to Python -- it's a basic issue with OO. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Feb 13 02:00:37 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Feb 2015 17:00:37 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> Message-ID: <54DD4CB5.7030702@stoneleaf.us> On 02/12/2015 04:55 PM, Guido van Rossum wrote: > On Thu, Feb 12, 2015 at 4:36 PM, Ethan Furman > wrote: > > I suspect the last big hurdle to making built-in data structures nicely subclassable is the insistence of such types to > return new instances as the base class instead of the derived class. > > In case that wasn't clear ;) > > --> class MyInt(int): > ... def __repr__(self): > ... return 'MyInt(%d)' % self > ... > --> m = MyInt(42) > --> m > MyInt(42) > --> m + 1 > 43 > --> type(m+1) > > > Besides the work it would take to rectify this, I imagine the biggest hurdle would be the performance hit in always > looking up the type of self. Has anyone done any preliminary benchmarking? Are there other concerns? > > > Actually, the problem is that the base class (e.g. int) doesn't know how to construct an instance of the subclass -- > there is no reason (in general) why the signature of a subclass constructor should match the base class constructor, and > it often doesn't. > > So this is pretty much a no-go. It's not unique to Python -- it's a basic issue with OO. Thank you. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From python at mrabarnett.plus.com Fri Feb 13 02:46:21 2015 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 13 Feb 2015 01:46:21 +0000 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> Message-ID: <54DD576D.1040402@mrabarnett.plus.com> On 2015-02-13 00:55, Guido van Rossum wrote: > On Thu, Feb 12, 2015 at 4:36 PM, Ethan Furman > wrote: > > I suspect the last big hurdle to making built-in data structures > nicely subclassable is the insistence of such types to > return new instances as the base class instead of the derived class. > > In case that wasn't clear ;) > > --> class MyInt(int): > ... def __repr__(self): > ... return 'MyInt(%d)' % self > ... > --> m = MyInt(42) > --> m > MyInt(42) > --> m + 1 > 43 > --> type(m+1) > > > Besides the work it would take to rectify this, I imagine the > biggest hurdle would be the performance hit in always > looking up the type of self. Has anyone done any preliminary > benchmarking? Are there other concerns? > > > Actually, the problem is that the base class (e.g. int) doesn't know how > to construct an instance of the subclass -- there is no reason (in > general) why the signature of a subclass constructor should match the > base class constructor, and it often doesn't. > > So this is pretty much a no-go. It's not unique to Python -- it's a > basic issue with OO. > Really? >>> class BaseInt: ... def __init__(self, value): ... self._value = value ... def __add__(self, other): ... return type(self)(self._value + other) ... def __repr__(self): ... return '%s(%s)' % (type(self), self._value) ... >>> class MyInt(BaseInt): ... pass ... >>> >>> m = BaseInt(42) >>> m (42) >>> m + 1 (43) >>> type(m + 1) >>> >>> m = MyInt(42) >>> m (42) >>> m + 1 (43) >>> type(m + 1) >>> From ethan at stoneleaf.us Fri Feb 13 03:14:22 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Feb 2015 18:14:22 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DD576D.1040402@mrabarnett.plus.com> References: <54DD4723.7030003@stoneleaf.us> <54DD576D.1040402@mrabarnett.plus.com> Message-ID: <54DD5DFE.9090203@stoneleaf.us> On 02/12/2015 05:46 PM, MRAB wrote: > On 2015-02-13 00:55, Guido van Rossum wrote: >> On Thu, Feb 12, 2015 at 4:36 PM, Ethan Furman > > wrote: >> >> I suspect the last big hurdle to making built-in data structures >> nicely subclassable is the insistence of such types to >> return new instances as the base class instead of the derived class. >> >> In case that wasn't clear ;) >> >> --> class MyInt(int): >> ... def __repr__(self): >> ... return 'MyInt(%d)' % self >> ... >> --> m = MyInt(42) >> --> m >> MyInt(42) >> --> m + 1 >> 43 >> --> type(m+1) >> >> >> Besides the work it would take to rectify this, I imagine the >> biggest hurdle would be the performance hit in always >> looking up the type of self. Has anyone done any preliminary >> benchmarking? Are there other concerns? >> >> >> Actually, the problem is that the base class (e.g. int) doesn't know how >> to construct an instance of the subclass -- there is no reason (in >> general) why the signature of a subclass constructor should match the >> base class constructor, and it often doesn't. >> >> So this is pretty much a no-go. It's not unique to Python -- it's a >> basic issue with OO. >> > Really? What I was asking about, and Guido responded to, was not having to specifically override __add__, __mul__, __sub__, and all the others; if we do override them then there is no problem. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From alexander.belopolsky at gmail.com Fri Feb 13 03:39:16 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 12 Feb 2015 21:39:16 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> Message-ID: On Thu, Feb 12, 2015 at 7:55 PM, Guido van Rossum wrote: > the problem is that the base class (e.g. int) doesn't know how to > construct an instance of the subclass -- there is no reason (in general) > why the signature of a subclass constructor should match the base class > constructor, and it often doesn't. I hear this explanation every time we have a discussion about subclassing of datetime types and I don't really buy this. Consider this simple subclass: >>> from datetime import date >>> class Date(date): ... pass ... What do you think Date.today() should return? Since I did not override todat() in my Date class, it has to be datetime.date instance, right? However: >>> Date.today().__class__ Wait, Date "doesn't know how to construct an instance of the subclass .." Indeed, if I change the constructor signature, Date.today() won't work: >>> class Date(date): ... def __init__(self, extra): ... pass ... >>> Date.today() Traceback (most recent call last): File "", line 1, in TypeError: __init__() takes exactly 2 arguments (4 given) In my view, a constructor is no different from any other method. If the designers of the subclass decided to change the signature in an incompatible way, they should either override all methods that create new objects or live with tracebacks. On the other hand, if all I want in my Date class is a better __format__ method, I am forced to override all operators or have my objects silently degrade in situations like this: >>> d = Date.today() >>> d.__class__ >>> d += timedelta(1) >>> d.__class__ Having binary operations return subclass instances is not without precedent. For example, in numpy, >>> from numpy import ndarray >>> class Array(ndarray): ... pass ... >>> a = Array(1) >>> a[0] = 42 >>> a Array([ 42.]) >>> a + a Array([ 84.]) I believe numpy had this behavior since types became subclassable in Python, so this design is definitely not a "no-go." -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 13 03:40:50 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 13 Feb 2015 13:40:50 +1100 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DD576D.1040402@mrabarnett.plus.com> References: <54DD4723.7030003@stoneleaf.us> <54DD576D.1040402@mrabarnett.plus.com> Message-ID: On Fri, Feb 13, 2015 at 12:46 PM, MRAB wrote: >>>> class BaseInt: > ... def __init__(self, value): > ... self._value = value > ... def __add__(self, other): > ... return type(self)(self._value + other) On Fri, Feb 13, 2015 at 11:55 AM, Guido van Rossum wrote: > ... there is no reason (in general) why > the signature of a subclass constructor should match the base class > constructor, and it often doesn't. You're requiring that any subclass of BaseInt be instantiable with one argument, namely its value. That's requiring that the signature of the subclass constructor match the base class constructor. ChrisA From steve at pearwood.info Fri Feb 13 03:57:30 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 13 Feb 2015 13:57:30 +1100 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DD5DFE.9090203@stoneleaf.us> References: <54DD4723.7030003@stoneleaf.us> <54DD576D.1040402@mrabarnett.plus.com> <54DD5DFE.9090203@stoneleaf.us> Message-ID: <20150213025730.GK2498@ando.pearwood.info> On Thu, Feb 12, 2015 at 06:14:22PM -0800, Ethan Furman wrote: > On 02/12/2015 05:46 PM, MRAB wrote: > > On 2015-02-13 00:55, Guido van Rossum wrote: > >> Actually, the problem is that the base class (e.g. int) doesn't know how > >> to construct an instance of the subclass -- there is no reason (in > >> general) why the signature of a subclass constructor should match the > >> base class constructor, and it often doesn't. > >> > >> So this is pretty much a no-go. It's not unique to Python -- it's a > >> basic issue with OO. > >> > > Really? > > What I was asking about, and Guido responded to, was not having to specifically override __add__, __mul__, __sub__, and > all the others; if we do override them then there is no problem. I think you have misunderstood MRAB's comment. My interpretation is that MRAB is suggesting that methods in the base classes should use type(self) rather than hard-coding their own type. E.g. if int were written in pure Python, it might look something like this: class int(object): def __new__(cls, arg): ... def __add__(self, other): return int(self, other) (figuratively, rather than literally). But if it looked like this: def __add__(self, other): return type(self)(self, other) then sub-classing would "just work" without the sub-class having to override each and every method. -- Steve From wizzat at gmail.com Fri Feb 13 03:59:32 2015 From: wizzat at gmail.com (Mark Roberts) Date: Thu, 12 Feb 2015 18:59:32 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD576D.1040402@mrabarnett.plus.com> Message-ID: > On Feb 12, 2015, at 18:40, Chris Angelico wrote: > > On Fri, Feb 13, 2015 at 12:46 PM, MRAB wrote: >>>>> class BaseInt: >> ... def __init__(self, value): >> ... self._value = value >> ... def __add__(self, other): >> ... return type(self)(self._value + other) > >> On Fri, Feb 13, 2015 at 11:55 AM, Guido van Rossum wrote: >> ... there is no reason (in general) why >> the signature of a subclass constructor should match the base class >> constructor, and it often doesn't. > > You're requiring that any subclass of BaseInt be instantiable with one > argument, namely its value. That's requiring that the signature of the > subclass constructor match the base class constructor. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/wizzat%40gmail.com No, it seems like he's asking that the type return a new object of the same type instead of one of the superclass. In effect, making the Date class call type(self)(*args) instead of datetime.date(*args). He seems completely willing to accept the consequences of changing the constructor (namely that he will have to override all the methods that call the constructor). It seems like good object oriented design to me. -Mark From ethan at stoneleaf.us Fri Feb 13 04:00:41 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Feb 2015 19:00:41 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <20150213025730.GK2498@ando.pearwood.info> References: <54DD4723.7030003@stoneleaf.us> <54DD576D.1040402@mrabarnett.plus.com> <54DD5DFE.9090203@stoneleaf.us> <20150213025730.GK2498@ando.pearwood.info> Message-ID: <54DD68D9.4090503@stoneleaf.us> On 02/12/2015 06:57 PM, Steven D'Aprano wrote: > On Thu, Feb 12, 2015 at 06:14:22PM -0800, Ethan Furman wrote: >> On 02/12/2015 05:46 PM, MRAB wrote: >>> On 2015-02-13 00:55, Guido van Rossum wrote: > >>>> Actually, the problem is that the base class (e.g. int) doesn't know how >>>> to construct an instance of the subclass -- there is no reason (in >>>> general) why the signature of a subclass constructor should match the >>>> base class constructor, and it often doesn't. >>>> >>>> So this is pretty much a no-go. It's not unique to Python -- it's a >>>> basic issue with OO. >>>> >>> Really? >> >> What I was asking about, and Guido responded to, was not having to specifically override __add__, __mul__, __sub__, and >> all the others; if we do override them then there is no problem. > > I think you have misunderstood MRAB's comment. My interpretation is > that MRAB is suggesting that methods in the base classes should use > type(self) rather than hard-coding their own type. That makes more sense, thanks. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Fri Feb 13 04:41:22 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Feb 2015 19:41:22 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> Message-ID: <54DD7262.4040000@stoneleaf.us> On 02/12/2015 06:39 PM, Alexander Belopolsky wrote: > In my view, a constructor is no different from any other method. If the designers of the subclass decided to change the > signature in an incompatible way, they should either override all methods that create new objects or live with tracebacks. > On the other hand, if all I want in my Date class is a better __format__ method, I am forced to override all operators > or have my objects silently degrade [...] So there are basically two choices: 1) always use the type of the most-base class when creating new instances pros: - easy - speedy code - no possible tracebacks on new object instantiation cons: - a subclass that needs/wants to maintain itself must override all methods that create new instances, even if the only change is to the type of object returned 2) always use the type of self when creating new instances pros: - subclasses automatically maintain type - much less code in the simple cases [1] cons: - if constructor signatures change, must override all methods which create new objects Unless there are powerful reasons against number 2 (such as performance, or the effort to affect the change), it sure seems like the nicer way to go. So back to my original question: what other concerns are there, and has anybody done any benchmarks? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From guido at python.org Fri Feb 13 05:01:19 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 12 Feb 2015 20:01:19 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DD7262.4040000@stoneleaf.us> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> Message-ID: On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman wrote: > On 02/12/2015 06:39 PM, Alexander Belopolsky wrote: > > > In my view, a constructor is no different from any other method. If the > designers of the subclass decided to change the > > signature in an incompatible way, they should either override all > methods that create new objects or live with tracebacks. > > > On the other hand, if all I want in my Date class is a better __format__ > method, I am forced to override all operators > > or have my objects silently degrade [...] > > So there are basically two choices: > > 1) always use the type of the most-base class when creating new instances > > pros: > - easy > - speedy code > - no possible tracebacks on new object instantiation > > cons: > - a subclass that needs/wants to maintain itself must override all > methods that create new instances, even if the only change is to > the type of object returned > > 2) always use the type of self when creating new instances > > pros: > - subclasses automatically maintain type > - much less code in the simple cases [1] > > cons: > - if constructor signatures change, must override all methods which > create new objects > > Unless there are powerful reasons against number 2 (such as performance, > or the effort to affect the change), it sure > seems like the nicer way to go. > > So back to my original question: what other concerns are there, and has > anybody done any benchmarks? > Con for #2 is a showstopper. Forget about it. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Feb 13 05:58:33 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Feb 2015 20:58:33 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> Message-ID: <54DD8479.8050600@stoneleaf.us> On 02/12/2015 08:01 PM, Guido van Rossum wrote: > On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman wrote: >> >> 2) always use the type of self when creating new instances >> >> cons: >> - if constructor signatures change, must override all methods which >> create new objects > > Con for #2 is a showstopper. Forget about it. Happy to, but can you explain why requiring the programmer to override the necessary methods, or get tracebacks, is a showstopper? Is there a previous email thread I can read that discusses it? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From theller at ctypes.org Fri Feb 13 07:59:44 2015 From: theller at ctypes.org (Thomas Heller) Date: Fri, 13 Feb 2015 07:59:44 +0100 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <54DCE56E.2080408@stoneleaf.us> References: , <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> Message-ID: Am 12.02.2015 um 18:39 schrieb Ethan Furman: > On 02/12/2015 12:05 AM, Thomas Heller wrote: > >> Could not py.exe be extended so that it allows starting scripts in a >> somewhat similar way? 'py-script -2.7 myscript foo bar baz' ??? >> Which would execute the script myscript.exe, myscript.bat, myscript.py, >> myscript.cmd or whatever is in the Scripts directory on the Python 2.7 >> installation, without changing PATH presistently? > > I'm pretty sure py.exe should only start .py files, not .bat, .cmd, or .anything-else. To make it clear: My suggestion is (or was, maybe it isn't a good idea) to have some way to start 'something' that is in the Scripts directory of a Python installation (which is usually a python script or an exe-wrapper for it), without typing in the full path. And without changing the PATH envrionment variable. Thomas From ncoghlan at gmail.com Fri Feb 13 08:02:37 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Feb 2015 17:02:37 +1000 Subject: [Python-Dev] PEP 486: Make the Python Launcher aware of virtual environments In-Reply-To: <54DD034B.2030907@stoneleaf.us> References: <54DD034B.2030907@stoneleaf.us> Message-ID: On 13 February 2015 at 05:47, Ethan Furman wrote: > Looks well thought-out. > > +1 I don't see any holes in the proposed behaviour either, so +1 from me. Procedurally, since it's a Windows specific change, and assuming Guido doesn't want to pronounce on this one himself, perhaps Steve Dower would be an appropriate BDFL-Delegate? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Feb 13 08:25:26 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Feb 2015 17:25:26 +1000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: On 13 February 2015 at 05:50, Paul Moore wrote: > On 12 February 2015 at 16:42, Steve Dower wrote: >> IIRC, there was a PEP for executing ZIP files directly (2.6-era?), which I believe are the purpose of those extensions. If "py.exe spam.pyz" already works, I don't see any need for a PEP to add the association in the installer. > > Yes, "py spam.pyz" works fine. +1 on having the associations. > Actually, I've just remembered, it's already in PEP 441, which hasn't > been approved but which goes further and includes a stdlib tool to > create pyz files. I'm not sure if that changes things at all... Zip archive & directory execution support was added in 2.6 by way of a tracker issue rather than a full PEP, but we didn't realise we left it out of the What's New docs until a few years later. That meant a lot of folks missed the addition of the feature. PEP 441 was aimed at giving the feature more visibility, in addition to making appropriately designed archives easier to create (IIRC, my main request for that PEP was to change the proposed module name to the more prosaic, but also more self-explanatory, "pyzapp", although it looks like I may have suggested that to Daniel offline rather than through the mailing list). Regardless, the PEP's not controversial as far as I am aware, so it would be nice if someone had the time to get it cleaned up and formally proposed for 3.5 (assuming Daniel was OK with that). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Feb 13 09:59:42 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 08:59:42 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <368981772.4110302.1423786500432.JavaMail.yahoo@mail.yahoo.com> References: <368981772.4110302.1423786500432.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 13 February 2015 at 00:15, Vinay Sajip wrote: > I remembered that I added a feature (with the help of Pawel Jasinski) to allow e.g. py -ipy to pick > a command 'ipy' configured in the .ini file (this is used to launch IronPython, but it could be used > for other things too). At the time the launcher was first released I *really* wanted that feature, but couldn't figure out how to cleanly add it. I'd love to see that go into the main repo :-) Paul From p.f.moore at gmail.com Fri Feb 13 10:13:09 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 09:13:09 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> Message-ID: On 13 February 2015 at 06:59, Thomas Heller wrote: > To make it clear: My suggestion is (or was, maybe it isn't a good idea) > to have some way to start 'something' that is in the Scripts > directory of a Python installation (which is usually a python script > or an exe-wrapper for it), without typing in the full path. And without > changing the PATH envrionment variable. People using the old-style "scripts" keyword to setup() can put pretty much anything into the Scripts directory - I've seen .py files, Python files with no suffix, bat files, even Unix shell files (not much use on Windows...) in there. I don't think it's reasonable for the launcher to try to run all of those things. Maybe restricting it to running ".py" files or ".exe" files would be reasonable. That covers entry points (which should be the norm for newer projects) and Python scripts (which are the most likely things to be portable). Paul From p.f.moore at gmail.com Fri Feb 13 10:18:02 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 09:18:02 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> Message-ID: On 13 February 2015 at 07:25, Nick Coghlan wrote: > PEP 441 was aimed at giving the feature more visibility, in addition > to making appropriately designed archives easier to create (IIRC, my > main request for that PEP was to change the proposed module name to > the more prosaic, but also more self-explanatory, "pyzapp", although > it looks like I may have suggested that to Daniel offline rather than > through the mailing list). > > Regardless, the PEP's not controversial as far as I am aware, so it > would be nice if someone had the time to get it cleaned up and > formally proposed for 3.5 (assuming Daniel was OK with that). I'd be willing to give that a go. The PEPs URL is giving me a "Bad Gateway" error at the moment, but I think the only coding needed in there is writing the "pyzaa/pyzapp" module. (If Steve would be willing to add the file associations - I'm not sure I want to dive into the new installer technology along with all the *other* things I'm volunteering for right now :-)) Paul From j.wielicki at sotecware.net Fri Feb 13 10:44:48 2015 From: j.wielicki at sotecware.net (Jonas Wielicki) Date: Fri, 13 Feb 2015 10:44:48 +0100 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> Message-ID: <54DDC790.2090306@sotecware.net> If I may humbly chime in this, with a hint... On 13.02.2015 05:01, Guido van Rossum wrote: > On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman wrote: >> [snip] >> 2) always use the type of self when creating new instances >> >> pros: >> - subclasses automatically maintain type >> - much less code in the simple cases [1] >> >> cons: >> - if constructor signatures change, must override all methods which >> create new objects >> >> Unless there are powerful reasons against number 2 (such as performance, >> or the effort to affect the change), it sure >> seems like the nicer way to go. >> >> So back to my original question: what other concerns are there, and has >> anybody done any benchmarks? >> > > Con for #2 is a showstopper. Forget about it. I would like to mention that there is another language out there which knows about virtual constructors (virtual like in virtual methods, with signature match requirements and such), which is FreePascal (and Delphi, and I think original Object Pascal too). It is actually a feature I liked about these languages, compared to C++03 and others, that constructors could be virtual and that classes were first-class citizens. Of course, Python cannot check the signature at compile time. But I think as long as it is documented, there should be no reason not to allow and support it. It really is analogous to other methods which need to have a matching signature. just my two cents, jwi -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From victor.stinner at gmail.com Fri Feb 13 10:46:55 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 13 Feb 2015 10:46:55 +0100 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? Message-ID: Hi, TL;DR: on POSIX, is it useful to know the inode number (st_ino) without the device number (st_dev)? While reading feedback on the Python 3.5 alpha 1 release, I saw a comment saying that the current design of os.scandir() (PEP 471) doesn't fit a very specific usecase where the inode number is needed: "Ah, turns out we needed even more optimizations than that is able to give us; in particular, the underlying system readdir call gives us the inode number, which we need to compare against a cache of hard links, in order to avoid having to stat the underlying files if we've already done so on another hard link. It looks like the DirEntry API used here only includes the path and name, not the inode number, without invoking another stat call, and we needed to optimize out that extra stat call." https://www.reddit.com/r/Python/comments/2synry/so_8_peps_are_currently_being_proposed_for_python/cnvnz1w Since the C function readdir() provides the inode number (d_ino field of the dirent structure), I propose add a new DirEntry.inode() method. *** Now the real question: is it useful to know the inode number (st_ino) without the device number (st_dev)? *** On POSIX, you can still get the st_dev from DirEntry.stat(), but it always require a system call. So you loose the whole purpose of DirEntry (no extra syscall). I wrote a script script check_stdev.py, attached to this email, to check if all entries of a directory have the same st_dev value than the directory itself: - same for /usr/bin, /usr/lib, /tmp, /proc, ... - different for /dev What about "union" file systems like UnionFS or thinks like "mount -o bind"? Can someone test? Does anyone have some information? So the answer looks to be: it's useful for all directories except of /dev. Example: --- /dev/hugepages st_dev is different: 35 vs 5 /dev/mqueue st_dev is different: 13 vs 5 /dev/pts st_dev is different: 11 vs 5 /dev/shm st_dev is different: 17 vs 5 --- On POSIX, DirEntry.inode() just exposes the d_ino value from readdir(). On Windows, FirstFindFileW/FindFindFileW returns almost a full stat_result structure, except of st_ino, st_dev and st_nlink fields which are set to 0. So DirEntry.inode() has to call os.lstat() to read the inode number. The inode number will be cached by DirEntry.inode() in the DirEntry object, but the os.lstat() result is dropped. On Windows, I don't want to cache the full os.lstat() result from DirEntry.inode() into DirEntry to replace the previous incomplete stat_result from FirstFindFileW/FindFindFileW, because DirEntry.stat() would return a different result (st_ino, st_dev, st_nlink fields set or not) depending if the inode() methode was called or not. Note: scandir-6.patch of http://bugs.python.org/issue22524 contains an implementation of os.scandir() with DirEntry.inode(), if you want to play. Victor -------------- next part -------------- A non-text attachment was scrubbed... Name: check_stdev.py Type: text/x-python Size: 300 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Fri Feb 13 10:47:15 2015 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 13 Feb 2015 09:47:15 +0000 (UTC) Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: <2042003663.4217375.1423820835953.JavaMail.yahoo@mail.yahoo.com> ----- Original Message ----- From: Paul Moore > I'd also appreciate your views on the spin-off thread and PEP 486 > ("Make the Python Launcher aware of virtual environments"). I just thought of something ... as far as I know, we've enabled searching for a command on the path - since when a venv is active in the sense of VIRTUAL_ENV being defined, PATH is also set up so that the venv's Scripts folder is on it - so we would just need to find "python.exe" on the PATH and invoke that - no need to even use the value of VIRTUAL_ENV, making implementation simpler. What do you think? Regards, Vinay From p.f.moore at gmail.com Fri Feb 13 10:57:53 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 09:57:53 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <2042003663.4217375.1423820835953.JavaMail.yahoo@mail.yahoo.com> References: <2042003663.4217375.1423820835953.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 13 February 2015 at 09:47, Vinay Sajip wrote: > I just thought of something ... as far as I know, we've enabled searching for a command on the path - since when a venv is active in the sense of VIRTUAL_ENV being defined, PATH is also set up so that the venv's Scripts folder is on it - so we would just need to find "python.exe" on the PATH and invoke that - no need to even use the value of VIRTUAL_ENV, making implementation simpler. What do you think? Two comments: 1. I'm not sure (but I've not dug into the implementation yet) that there's much practical difference between locating python.exe via a PATH search, or locating it via concatenating VIRTUAL_ENV and the relative path. 2. It would be a nice, although extremely obscure, feature to be able to allow people who want to keep Python off their path to set VIRTUAL_ENV but *not* set PATH, so that py.exe does the right thing but there's *still* no python.exe on PATH. Limit the dependency on a single environment variable rather than two, in other words. Paul From victor.stinner at gmail.com Fri Feb 13 11:07:03 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 13 Feb 2015 11:07:03 +0100 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) Message-ID: Hi, TL,DR: are you ok to add 800 lines of C code for os.scandir(), 4x faster than os.listdir() when the file type is checked? I accepted the PEP 471 (os.scandir) a few months ago, but it is not implement yet in Python 3.5, because I didn't make a choice on the implementation. Ben Hoyt wrote different implementations: - full C: os.scandir() and DirEntry are written in C (no change on os.py) - C+Python: os._scandir() (wrapper for opendir/readdir and FindFirstFileW/FindNextFileW) in C, DirEntry in Python - ctypes: os.scandir() and DirEntry fully implemented in Python I'm not interested by the ctypes implementation. It's useful for a third party project hosted at PyPI, but for CPython I prefer to wrap C functions using C code. In short, the C implementation is faster than the C+Python implementation. The issue #22524 (*) is full of benchmark numbers. IMO the most interesting benchmark is to compare os.listdir() + os.stat() versus os.scandir() + Direntry.is_dir(). Let me try to summarize results of this benchmark: * C implementation: scandir is at least 3.5x faster than listdir, up to 44.6x faster on Windows * C+Python implementation: scandir is not really faster than listdir, between 1.3x and 1.4x faster (*) http://bugs.python.org/issue22524 Ben Hoyt reminded me that os.scandir() (PEP 471) doesn't add any new feature: pathlib already provides a nice API on top of os and os.path modules. (You may even notice that DirEntry a much fewer methods ;-)) The main (only?) purpose of the PEP is performance. If os.scandir() is "only" 1.4x faster, I don't think that it is interesting to use os.scandir() in an application. I guess that all applications/libraries will want to keep compatibility with Python 3.4 and older and so will anyway have to duplicate the code to use os.listdir() + os.stat(). So is it worth to duplicate code for such small speedup? Now I see 3 choices: - take the full C implementation, because it's much faster (at least 3.4x faster!) - reject the whole PEP 471 (not nice), because it adds too much code for a minor speedup (not true on Windows: up to 44x faster!) - take the C+Python implementation, because maintenance matters more than performances (only 1.3x faster, sorry) => IMO the best option is to take the C implementation. What do you think? I'm concerned by the length of the C code: the full C implementations adds ~800 lines of C code to posixmodule.c. This file is already the longest C file in CPython. I don't want to make it longer, but I'm not motived to start to split it. Last time I proposed to split a file (unicodeobject.c), some developers complained that it makes search harder. I don't understand this, there are so many tools to navigate in C code. But it was enough for me to give up on this idea. A alternative is to add a new _scandir.c module to host the new C code, and share some code with posixmodule.c: remove "static" keyword from required C functions (functions to convert Windows attributes to a os.stat_result object). That's a reasonable choice. What do you think? FYI I ran the benchmark on different hardware (SSD, HDD, tmpfs), file systems (ext4, tmpfs, NFS/ext4), operating systems (Linux, Windows). Victor From mistersheik at gmail.com Fri Feb 13 11:08:14 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 13 Feb 2015 05:08:14 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DDC790.2090306@sotecware.net> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DDC790.2090306@sotecware.net> Message-ID: With Python's cooperative inheritance, I think you want to do everything through one constructor sending keyword arguments up the chain. The keyword arguments are popped off as needed. With this setup I don't think you need "overloaded constructors". Best, Neil On Fri, Feb 13, 2015 at 4:44 AM, Jonas Wielicki wrote: > If I may humbly chime in this, with a hint... > > On 13.02.2015 05:01, Guido van Rossum wrote: > > On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman > wrote: > >> [snip] > >> 2) always use the type of self when creating new instances > >> > >> pros: > >> - subclasses automatically maintain type > >> - much less code in the simple cases [1] > >> > >> cons: > >> - if constructor signatures change, must override all methods which > >> create new objects > >> > >> Unless there are powerful reasons against number 2 (such as performance, > >> or the effort to affect the change), it sure > >> seems like the nicer way to go. > >> > >> So back to my original question: what other concerns are there, and has > >> anybody done any benchmarks? > >> > > > > Con for #2 is a showstopper. Forget about it. > > I would like to mention that there is another language out there which > knows about virtual constructors (virtual like in virtual methods, with > signature match requirements and such), which is FreePascal (and Delphi, > and I think original Object Pascal too). > > It is actually a feature I liked about these languages, compared to > C++03 and others, that constructors could be virtual and that classes > were first-class citizens. > > Of course, Python cannot check the signature at compile time. But I > think as long as it is documented, there should be no reason not to > allow and support it. It really is analogous to other methods which need > to have a matching signature. > > just my two cents, > jwi > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Feb 13 11:19:00 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 10:19:00 +0000 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: On 13 February 2015 at 10:07, Victor Stinner wrote: > => IMO the best option is to take the C implementation. What do you think? FWIW (as I'm not a core dev) I agree. The Windows speedup is huge, and well worth adding the code. I'm assuming that the majority of the C code is cross-platform, so we're not adding a big chunk of code needing *both* Windows and C skills to maintain (any dev with C skills could handle it). Paul From v+python at g.nevcal.com Fri Feb 13 11:15:51 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 13 Feb 2015 02:15:51 -0800 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> Message-ID: <54DDCED7.9080606@g.nevcal.com> On 2/13/2015 1:13 AM, Paul Moore wrote: > On 13 February 2015 at 06:59, Thomas Heller wrote: >> To make it clear: My suggestion is (or was, maybe it isn't a good idea) >> to have some way to start 'something' that is in the Scripts >> directory of a Python installation (which is usually a python script >> or an exe-wrapper for it), without typing in the full path. And without >> changing the PATH envrionment variable. > People using the old-style "scripts" keyword to setup() can put pretty > much anything into the Scripts directory - I've seen .py files, Python > files with no suffix, bat files, even Unix shell files (not much use > on Windows...) in there. I don't think it's reasonable for the > launcher to try to run all of those things. > > Maybe restricting it to running ".py" files or ".exe" files would be > reasonable. That covers entry points (which should be the norm for > newer projects) and Python scripts (which are the most likely things > to be portable). The WINDOWS py launcher hardly needs to be portable. Batch/CMD files also seem useful on WINDOWS. And Powershell? If a launcher is developed for or enhanced for other systems, shell scripts may be useful, and Batch/CMD not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Feb 13 11:33:11 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 13 Feb 2015 11:33:11 +0100 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: 2015-02-13 11:19 GMT+01:00 Paul Moore : > On 13 February 2015 at 10:07, Victor Stinner wrote: >> => IMO the best option is to take the C implementation. What do you think? > > FWIW (as I'm not a core dev) I agree. The Windows speedup is huge, and > well worth adding the code. I'm assuming that the majority of the C > code is cross-platform, so we're not adding a big chunk of code > needing *both* Windows and C skills to maintain (any dev with C skills > could handle it). > > Paul The patch can be read here: http://bugs.python.org/file36963/scandir-2.patch Or using Rietveld: http://bugs.python.org/review/22524/#ps13104 The C code is quite simple. It takes 800 lines because C code is more "verbose" than Python code. Manipulate strings, manage memory, take care of the reference counter, etc. just takes more lines. Victor From storchaka at gmail.com Fri Feb 13 11:52:36 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 13 Feb 2015 12:52:36 +0200 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: On 13.02.15 12:07, Victor Stinner wrote: > TL,DR: are you ok to add 800 lines of C code for os.scandir(), 4x > faster than os.listdir() when the file type is checked? You can try to make Python implementation faster if 1) Don't set attributes to None in constructor. 2) Implement scandir as: def scandir(path): return map(partial(DirEntry, path), _scandir(path)). 3) Or pass DirEntry to _scandir: def scandir(path): yield from _scandir(path, DirEntry) From p.f.moore at gmail.com Fri Feb 13 12:19:54 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 11:19:54 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <54DDCED7.9080606@g.nevcal.com> References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DDCED7.9080606@g.nevcal.com> Message-ID: On 13 February 2015 at 10:15, Glenn Linderman wrote: > Maybe restricting it to running ".py" files or ".exe" files would be > reasonable. That covers entry points (which should be the norm for > newer projects) and Python scripts (which are the most likely things > to be portable). > > The WINDOWS py launcher hardly needs to be portable. Batch/CMD files also > seem useful on WINDOWS. And Powershell? > > If a launcher is developed for or enhanced for other systems, shell scripts > may be useful, and Batch/CMD not. By "portable" in this sense I meant "not written by a Unix developer who hadn't thought about Windows much". I was comparing Python files with a .py extension and extensionless Python files, specifically (my wording didn't make that clear, sorry). Yes, files with extensions of .bat or .cmd or .ps1 are probably written for Windows only. But I think it's getting out of scope of the launcher to decide to use cmd.exe to launch .bat and .cmd files, and powershell to launch .ps1 files. Regardless, my real point was that there can be all sorts of stuff in the Scripts directory, and I'm not convinced that this is a direction the launcher should be going in. Someone *could* make a case for launching certain types of file that might be in there, but I'm not going to be that person. One thought - it's not hard using pkg_resources to enumerate all available console_script entry points, pick one and run it. So you could pretty simply write a script (that itself could be run with the launcher) to run anything in Scripts that's built via entry points (which is the recommended way these days). # launch.py import sys import pkg_resources # Get the script name script = sys.argv[1] # Remove our name from sys.argv del sys.argv[0] # There's probably a better API you can use for ep in pkg_resources.working_set.iter_entry_points('console_scripts'): if ep.name == script: fn = ep.load() sys.exit(fn()) # Couldn't find the entry point print("No entry point called {} available".format(script), file=sys.stderr) sys.exit(1) Now you can do "py launch.py pip --help" and it will run pip for you. If a project isn't exposing its scripts as entry points, ask them to :-) Paul Paul From storchaka at gmail.com Fri Feb 13 12:27:04 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 13 Feb 2015 13:27:04 +0200 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: On 13.02.15 12:07, Victor Stinner wrote: > * C implementation: scandir is at least 3.5x faster than listdir, up > to 44.6x faster on Windows Results on Windows was obtained in the becnhmark that doesn't drop disk caches and runs listdir before scandir. From victor.stinner at gmail.com Fri Feb 13 12:34:01 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 13 Feb 2015 12:34:01 +0100 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: 2015-02-13 12:27 GMT+01:00 Serhiy Storchaka : > On 13.02.15 12:07, Victor Stinner wrote: >> >> * C implementation: scandir is at least 3.5x faster than listdir, up >> to 44.6x faster on Windows > > > Results on Windows was obtained in the becnhmark that doesn't drop disk > caches and runs listdir before scandir. The benchmark code is here: http://bugs.python.org/file38120/bench_scandir2.py Eeach test is repeated 5 times, I compared the duration of the fastest call. A test calls the function 5 times in a loop. Anyway, it's not the first call to listdir() which fills the disk cache, but the call to count_entries() (which is implemented with os.scandir). So is there any issue in the benchmark script or not? On Linux, you can use "bench_nocache" (and "bench_nostat_nocache") which flushs the cache: it writes "3" into /proc/sys/vm/drop_caches. scandir is always faster when the disk cache is flushed. Victor From ncoghlan at gmail.com Fri Feb 13 12:37:52 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Feb 2015 21:37:52 +1000 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: On 13 February 2015 at 20:33, Victor Stinner wrote: > 2015-02-13 11:19 GMT+01:00 Paul Moore : >> On 13 February 2015 at 10:07, Victor Stinner wrote: >>> => IMO the best option is to take the C implementation. What do you think? >> >> FWIW (as I'm not a core dev) I agree. The Windows speedup is huge, and >> well worth adding the code. I'm assuming that the majority of the C >> code is cross-platform, so we're not adding a big chunk of code >> needing *both* Windows and C skills to maintain (any dev with C skills >> could handle it). >> >> Paul > > The patch can be read here: > http://bugs.python.org/file36963/scandir-2.patch > > Or using Rietveld: > http://bugs.python.org/review/22524/#ps13104 > > The C code is quite simple. It takes 800 lines because C code is more > "verbose" than Python code. Manipulate strings, manage memory, take > care of the reference counter, etc. just takes more lines. This isn't code I'd expect us to have to change very often, so the maintenance risks associated with the pure C implementation seem low. Having it in a separate file rather than making the main implementation file for os even larger does seem like an attractive structural option though. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Fri Feb 13 12:48:10 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 13 Feb 2015 12:48:10 +0100 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: 2015-02-13 11:52 GMT+01:00 Serhiy Storchaka : > You can try to make Python implementation faster if > > 1) Don't set attributes to None in constructor. The class uses __slots__. Setting attributes in the class body is forbidden when __slots__ is used. > 3) Or pass DirEntry to _scandir: > > def scandir(path): > yield from _scandir(path, DirEntry) I implemented that and there is no major change (1.3x faster => 1.5x, it's still far from 3.5x faster seen with the C implementation). I analyzed numbers (on my desktop PC, HDD, ext4): - readdir: 380 ns - os.stat: 1500 ns - DirEntry(C): 100 ns - DirEntry (Py): 530 ns (5.3x slower) - is_dir(C): 75 ns - is_dir (Py): 260 ns (3.5x slower) listdir+stat benchmarks takes (readdir + stat) nanoseconds scandir+is_dir takes (readdir + DirEntry + is_dir) nanoseconds => scandir+is_dir is faster than list+stat if (DirEntry+is_dir) is faster than (stat). Callig os.stat takes 1500 ns, while readdir() only provides informations required by the benchmark. So if DirEntry + DirEntry.is_dir is faster than 1500 ns, we won :-) The Python implementation takes 790 ns, but the C implementation takes only 175 ns! (4.5x faster) I don't think that any Python performance trick can reduce the Python overhead to make the C+Python implementation interesting compared to os.listdir+os.stat. We are talking about nanoseconds, Python cannot beat C at this resolution. Victor From solipsis at pitrou.net Fri Feb 13 13:20:27 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Feb 2015 13:20:27 +0100 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) References: Message-ID: <20150213132027.2c27c29d@fsol> On Fri, 13 Feb 2015 13:27:04 +0200 Serhiy Storchaka wrote: > On 13.02.15 12:07, Victor Stinner wrote: > > * C implementation: scandir is at least 3.5x faster than listdir, up > > to 44.6x faster on Windows > > Results on Windows was obtained in the becnhmark that doesn't drop disk > caches and runs listdir before scandir. Well, that's the point. The objective here is to speed up Python, not the hard drive. Regards Antoine. From solipsis at pitrou.net Fri Feb 13 13:25:38 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Feb 2015 13:25:38 +0100 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) References: Message-ID: <20150213132538.31e1fc02@fsol> On Fri, 13 Feb 2015 11:07:03 +0100 Victor Stinner wrote: > > * C implementation: scandir is at least 3.5x faster than listdir, up > to 44.6x faster on Windows > * C+Python implementation: scandir is not really faster than listdir, > between 1.3x and 1.4x faster So amusingly, the bottleneck is not so much the cost of system calls, but the cost of Python wrappers around system calls. Regards Antoine. From benhoyt at gmail.com Fri Feb 13 14:35:00 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Fri, 13 Feb 2015 08:35:00 -0500 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: <20150213132538.31e1fc02@fsol> References: <20150213132538.31e1fc02@fsol> Message-ID: > > * C implementation: scandir is at least 3.5x faster than listdir, up > > to 44.6x faster on Windows > > * C+Python implementation: scandir is not really faster than listdir, > > between 1.3x and 1.4x faster > > So amusingly, the bottleneck is not so much the cost of system calls, > but the cost of Python wrappers around system calls. Yes, that's basically right. Or put another way, the cost of the extra system calls is dwarfed by the cost of wrapping things in Python. Victor's given a great summary of the issues at the top of this thread, and I'm definitely for the all-C version -- otherwise we gain a bunch of speed by not calling stat(), but then lose most of it again with the Python wrapping. As Victor noted, the rationale for PEP 471 has always been about performance, and if we don't have much of that (especially on Linux), it's not nearly as worthwhile. Re maintenance of the C code: yes, the pure C version is about twice as many lines as the half Python version (~800 vs ~400), but I think Nick makes a good point here: "This isn't code I'd expect us to have to change very often, so the maintenance risks associated with the pure C implementation seem low." We have to vet this code thoroughly basically once, now. :-) If we go ahead with the all C approach, I'd be in favour of refactoring a little and putting the new scandir code into a separate C file. There are two ways to do this: a) sticking with a single Python module and just referencing the non-static functions in scandir.c from posixmodule.c, or b) sharing some functions but making _scandir.c its own importable module. Option (a) is somewhat simpler as there's not module setup stuff twice, but I don't know if there's a precedent for that way of doing things. -Ben From benhoyt at gmail.com Fri Feb 13 14:40:14 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Fri, 13 Feb 2015 08:40:14 -0500 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: References: Message-ID: > TL;DR: on POSIX, is it useful to know the inode number (st_ino) > without the device number (st_dev)? I can't answer this question (not being a Linux dev and not knowing much about this), but I'm +1 for adding DirEntry.inode(). On Windows, we're exposing all the information FindFirst/FindNext give us, but on Linux we expose everything useful from readdir except d_ino, which is easy to add, and according to that reddit comment, may make scandir useful in more real scenarios. -Ben From solipsis at pitrou.net Fri Feb 13 14:44:08 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Feb 2015 14:44:08 +0100 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: <20150213132538.31e1fc02@fsol> Message-ID: <20150213144408.34c9d4aa@fsol> On Fri, 13 Feb 2015 08:35:00 -0500 Ben Hoyt wrote: > > If we go ahead with the all C approach, I'd be in favour of > refactoring a little and putting the new scandir code into a separate > C file. There are two ways to do this: a) sticking with a single > Python module and just referencing the non-static functions in > scandir.c from posixmodule.c, or b) sharing some functions but making > _scandir.c its own importable module. Option (a) is somewhat simpler > as there's not module setup stuff twice, but I don't know if there's a > precedent for that way of doing things. The _io module already does things that way (the (a) option, I mean). Regards Antoine. From Steve.Dower at microsoft.com Fri Feb 13 15:36:28 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 13 Feb 2015 14:36:28 +0000 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: <20150213144408.34c9d4aa@fsol> References: <20150213132538.31e1fc02@fsol> , <20150213144408.34c9d4aa@fsol> Message-ID: I think posixmodule is a great candidate for splitting up by platform rather than function, as the whole file is packed with ifdef. It's really only lacking a volunteer to do it, but we could start here (ie. make posixmodule_nt.c for the Windows impl, etc.) and progressively move function implementations out over time? All the module setup and probably most of the Python layer can stay where it is. More likely we're going to get bogged down discussing it again though, so if that happens my vote is to just make posixmodule.c 800 lines longer. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Antoine Pitrou Sent: ?2/?13/?2015 5:44 To: python-dev at python.org Subject: Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) On Fri, 13 Feb 2015 08:35:00 -0500 Ben Hoyt wrote: > > If we go ahead with the all C approach, I'd be in favour of > refactoring a little and putting the new scandir code into a separate > C file. There are two ways to do this: a) sticking with a single > Python module and just referencing the non-static functions in > scandir.c from posixmodule.c, or b) sharing some functions but making > _scandir.c its own importable module. Option (a) is somewhat simpler > as there's not module setup stuff twice, but I don't know if there's a > precedent for that way of doing things. The _io module already does things that way (the (a) option, I mean). Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Fri Feb 13 15:27:31 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 13 Feb 2015 14:27:31 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DDCED7.9080606@g.nevcal.com>, Message-ID: Launching non-Python scripts from py.exe is not going to work as well as you may hope. There will be numerous subtle bugs due to the cmd->py.exe->cmd wrapping (Powershell users would have seen this with some console apps or batch files). I think having a global command to add a particular Python to PATH for the current session will work best here, and yes, I'm working on it. If py.exe starts defaulting to whatever is on PATH then I don't see the point of it. Knowing that you'll get the latest 2.x version by default is quite handy (I'd be quite okay changing that to 3.x, though it is specified in the original PEP). For me, the point of py.exe is to be able to ignore PATH completely. Adding more commands to the launcher args (eg. -ipy) will make the parsing far more complex, especially when deciding which args to consume and which to forward. I think you can make a copy of the launcher called ipy.exe and add ipy.ini with the path to IronPython in it as default. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Paul Moore Sent: ?2/?13/?2015 3:20 To: Glenn Linderman Cc: Python Dev Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows On 13 February 2015 at 10:15, Glenn Linderman wrote: > Maybe restricting it to running ".py" files or ".exe" files would be > reasonable. That covers entry points (which should be the norm for > newer projects) and Python scripts (which are the most likely things > to be portable). > > The WINDOWS py launcher hardly needs to be portable. Batch/CMD files also > seem useful on WINDOWS. And Powershell? > > If a launcher is developed for or enhanced for other systems, shell scripts > may be useful, and Batch/CMD not. By "portable" in this sense I meant "not written by a Unix developer who hadn't thought about Windows much". I was comparing Python files with a .py extension and extensionless Python files, specifically (my wording didn't make that clear, sorry). Yes, files with extensions of .bat or .cmd or .ps1 are probably written for Windows only. But I think it's getting out of scope of the launcher to decide to use cmd.exe to launch .bat and .cmd files, and powershell to launch .ps1 files. Regardless, my real point was that there can be all sorts of stuff in the Scripts directory, and I'm not convinced that this is a direction the launcher should be going in. Someone *could* make a case for launching certain types of file that might be in there, but I'm not going to be that person. One thought - it's not hard using pkg_resources to enumerate all available console_script entry points, pick one and run it. So you could pretty simply write a script (that itself could be run with the launcher) to run anything in Scripts that's built via entry points (which is the recommended way these days). # launch.py import sys import pkg_resources # Get the script name script = sys.argv[1] # Remove our name from sys.argv del sys.argv[0] # There's probably a better API you can use for ep in pkg_resources.working_set.iter_entry_points('console_scripts'): if ep.name == script: fn = ep.load() sys.exit(fn()) # Couldn't find the entry point print("No entry point called {} available".format(script), file=sys.stderr) sys.exit(1) Now you can do "py launch.py pip --help" and it will run pip for you. If a project isn't exposing its scripts as entry points, ask them to :-) Paul Paul _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Feb 13 15:49:31 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 13 Feb 2015 15:49:31 +0100 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: <20150213132538.31e1fc02@fsol> <20150213144408.34c9d4aa@fsol> Message-ID: 2015-02-13 15:36 GMT+01:00 Steve Dower : > I think posixmodule is a great candidate for splitting up by platform rather > than function, as the whole file is packed with ifdef. It's really only > lacking a volunteer to do it, but we could start here (ie. make > posixmodule_nt.c for the Windows impl, etc.) and progressively move function > implementations out over time? All the module setup and probably most of the > Python layer can stay where it is. > > More likely we're going to get bogged down discussing it again though, so if > that happens my vote is to just make posixmodule.c 800 lines longer. Since there are many ways to split this huge file, I agree that it's just fine to add these 800 lines and *then* think how the huge file can be splitted. It's a different topic. Victor From jonas at wielicki.name Fri Feb 13 09:41:57 2015 From: jonas at wielicki.name (Jonas Wielicki) Date: Fri, 13 Feb 2015 09:41:57 +0100 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> Message-ID: <54DDB8D5.8040905@wielicki.name> If I may humbly chime in this, with a hint... On 13.02.2015 05:01, Guido van Rossum wrote: > On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman wrote: >> [snip] >> 2) always use the type of self when creating new instances >> >> pros: >> - subclasses automatically maintain type >> - much less code in the simple cases [1] >> >> cons: >> - if constructor signatures change, must override all methods which >> create new objects >> >> Unless there are powerful reasons against number 2 (such as performance, >> or the effort to affect the change), it sure >> seems like the nicer way to go. >> >> So back to my original question: what other concerns are there, and has >> anybody done any benchmarks? >> > > Con for #2 is a showstopper. Forget about it. I would like to mention that there is another language out there which knows about virtual constructors (virtual like in virtual methods, with signature match requirements and such), which is FreePascal (and Delphi, and I think original Object Pascal too). It is actually a feature I liked about these languages, compared to C++03 and others, that constructors could be virtual and that classes were first-class citizens. Of course, Python cannot check the signature at compile time. But I think as long as it is documented, there should be no reason not to allow and support it. It really is analogous to other methods which need to have a matching signature. just my two cents, jwi -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Fri Feb 13 16:45:50 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 15:45:50 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DDCED7.9080606@g.nevcal.com> Message-ID: On 13 February 2015 at 14:27, Steve Dower wrote: > If py.exe starts defaulting to whatever is on PATH then I don't see the > point of it. Knowing that you'll get the latest 2.x version by default is > quite handy (I'd be quite okay changing that to 3.x, though it is specified > in the original PEP). For me, the point of py.exe is to be able to ignore > PATH completely. It does seem a bit bizarre to me that a launcher shipped with Python 3 defaults to using Python 2 if it's available. I'd like to see that change, but as it's specified in the original PEP I guess it might meet with some pushback :-( > Adding more commands to the launcher args (eg. -ipy) will make the parsing > far more complex, especially when deciding which args to consume and which > to forward. I think you can make a copy of the launcher called ipy.exe and > add ipy.ini with the path to IronPython in it as default. Well, Vinay has this functionality already available in his fork, so it's not impossible. But whether or not it's added I do think that all distributed versions of py.exe should have the same functionality, so either the CPython version should add it or Vinay should remove it (or rename his version and describe it as an *enhanced* version of the py.exe launcher, rather than as a *standalone* version...) I'm +0 on the functionality personally - I quite like the idea, but I've no real need for it and I've lived without it long enough now. Paul From benhoyt at gmail.com Fri Feb 13 16:54:55 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Fri, 13 Feb 2015 10:54:55 -0500 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: <20150213132538.31e1fc02@fsol> <20150213144408.34c9d4aa@fsol> Message-ID: > Since there are many ways to split this huge file, I agree that it's > just fine to add these 800 lines and *then* think how the huge file > can be splitted. It's a different topic. That's a good idea. Consider adding the new feature (scandir) and the larger issue of refactoring posixmodule as separate issues, separate commits, etc. -Ben From ethan at stoneleaf.us Fri Feb 13 17:25:24 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Feb 2015 08:25:24 -0800 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: <54DE2574.8010700@stoneleaf.us> On 02/13/2015 02:07 AM, Victor Stinner wrote: > Hi, > > TL,DR: are you ok to add 800 lines of C code for os.scandir(), 4x > faster than os.listdir() when the file type is checked? +1 for the all-C version. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Fri Feb 13 17:31:01 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Feb 2015 08:31:01 -0800 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <2042003663.4217375.1423820835953.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54DE26C5.6070305@stoneleaf.us> On 02/13/2015 01:57 AM, Paul Moore wrote: > 2. It would be a nice, although extremely obscure, feature to be able > to allow people who want to keep Python off their path to set > VIRTUAL_ENV but *not* set PATH, so that py.exe does the right thing > but there's *still* no python.exe on PATH. Limit the dependency on a > single environment variable rather than two, in other words. +1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From alexander.belopolsky at gmail.com Fri Feb 13 17:55:48 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 13 Feb 2015 11:55:48 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> Message-ID: On Thu, Feb 12, 2015 at 11:01 PM, Guido van Rossum wrote: > >> 2) always use the type of self when creating new instances >> .. >> cons: >> - if constructor signatures change, must override all methods which >> create new objects >> > > Con for #2 is a showstopper. Forget about it. > Sorry if I am missing something obvious, but I still don't understand why the same logic does not apply to class methods that create new instances: >>> from datetime import * >>> date.today() datetime.date(2015, 2, 13) >>> datetime.today() datetime.datetime(2015, 2, 13, 11, 37, 23, 678680) >>> class Date(date): ... pass ... >>> Date.today() Date(2015, 2, 13) (I actually find datetime.today() returning a datetime rather than a date a questionable design decision, but probably the datetime type should not have been a subclass of the date to begin with.) Are there any date subclasses in the wild that don't accept year, month, day in the constructor? If you create such a class, wouldn't you want to override __add__ and friends anyways? We already know that you will have to override today(). -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Feb 13 18:08:13 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 13 Feb 2015 18:08:13 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20150213170813.E43D2560BE@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-02-06 - 2015-02-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4790 (+25) closed 30430 (+32) total 35220 (+57) Open issues with patches: 2254 Issues opened (47) ================== #21793: httplib client/server status refactor http://bugs.python.org/issue21793 reopened by serhiy.storchaka #23402: dynload_shlib does not close ldl handles when the interpreter http://bugs.python.org/issue23402 opened by Adam #23403: Use pickle protocol 4 by default? http://bugs.python.org/issue23403 opened by serhiy.storchaka #23404: 'make touch' does not work with git clones of the source repos http://bugs.python.org/issue23404 opened by vlee #23405: Tools/freeze "make" gets missing file error with unix shared b http://bugs.python.org/issue23405 opened by ned.deily #23406: interning and list comprehension leads to unexpected behavior http://bugs.python.org/issue23406 opened by Abraham.Smith #23407: os.walk always follows Windows junctions http://bugs.python.org/issue23407 opened by craigh #23408: Pickle tests use incorrect test data http://bugs.python.org/issue23408 opened by serhiy.storchaka #23410: Document more BaseHTTPRequestHandler attributes http://bugs.python.org/issue23410 opened by vadmium #23411: Update urllib.parse.__all__ http://bugs.python.org/issue23411 opened by vadmium #23414: seek(count, whence) accepts bogus whence on windows, python2.7 http://bugs.python.org/issue23414 opened by mattip #23415: add-to-pydotorg does not support .exe installers for Windows http://bugs.python.org/issue23415 opened by larry #23416: Make urllib.parse.SplitResult etc arguments optional http://bugs.python.org/issue23416 opened by vadmium #23418: Keep http.server.__all__ up to date http://bugs.python.org/issue23418 opened by vadmium #23419: Faster default __reduce__ for classes without __init__ http://bugs.python.org/issue23419 opened by serhiy.storchaka #23420: python -m cProfile -s fails with non informative message http://bugs.python.org/issue23420 opened by rkuska #23422: Clarify docs for importlib.import_module() http://bugs.python.org/issue23422 opened by brett.cannon #23423: XPath Support in ElementTree doc omission http://bugs.python.org/issue23423 opened by mbakeranalecta #23424: Unicode character ends interactive session http://bugs.python.org/issue23424 opened by AGrzes #23425: Windows getlocale unix-like with french, german, portuguese, s http://bugs.python.org/issue23425 opened by fomcl at yahoo.com #23426: run_setup is broken in distutils http://bugs.python.org/issue23426 opened by belopolsky #23427: Python should expose command when invoked with -c http://bugs.python.org/issue23427 opened by jgehrcke #23428: Use the monotonic clock for thread conditions on POSIX platfor http://bugs.python.org/issue23428 opened by haypo #23430: socketserver.BaseServer.handle_error() should not catch exitin http://bugs.python.org/issue23430 opened by vadmium #23431: Idle Application Not Responding http://bugs.python.org/issue23431 opened by ww0115 #23432: Duplicate content in SystemExit documentation http://bugs.python.org/issue23432 opened by berker.peksag #23434: RFC6266 support (Content-Disposition for HTTP) http://bugs.python.org/issue23434 opened by Myroslav.Opyr #23436: xml.dom.minidom.Element.ownerDocument is hiden http://bugs.python.org/issue23436 opened by krocard #23437: Make user scripts directory versioned on Windows http://bugs.python.org/issue23437 opened by pmoore #23439: Fixed http.client.__all__ and added a test http://bugs.python.org/issue23439 opened by vadmium #23440: Extend http.server.SimpleHTTPRequestHandler testing http://bugs.python.org/issue23440 opened by vadmium #23441: rlcompleter: tab on empty prefix => insert spaces http://bugs.python.org/issue23441 opened by arigo #23442: http.client.REQUEST_HEADER_FIELDS_TOO_LARGE renamed in 3.5 http://bugs.python.org/issue23442 opened by vadmium #23443: XMLRPCLIB Exception uses str not class or instance http://bugs.python.org/issue23443 opened by kmarsh #23444: HCI Bluetooth socket bind error on an arm crosscompiled enviro http://bugs.python.org/issue23444 opened by Thomas.Chiroux #23446: Use PyMem_New instead of PyMem_Malloc http://bugs.python.org/issue23446 opened by serhiy.storchaka #23448: urllib2 needs to remove scope from IPv6 address when creating http://bugs.python.org/issue23448 opened by ngierman #23449: Fatal errors rebuilding 3.5 from Visual Studio Windows 8.1 64 http://bugs.python.org/issue23449 opened by BreamoreBoy #23450: Possible loss of data warnings building 3.5 Visual Studio Wind http://bugs.python.org/issue23450 opened by BreamoreBoy #23451: Deprecation warnings building 3.5 Visual Studio Windows 8.1 64 http://bugs.python.org/issue23451 opened by BreamoreBoy #23452: Build errors using VS Express 2013 in win32 mode http://bugs.python.org/issue23452 opened by BreamoreBoy #23453: Opening a stream with tarfile.open() triggers a TypeError: can http://bugs.python.org/issue23453 opened by chaica_ #23455: file iterator "deemed broken"; can resume after StopIteration http://bugs.python.org/issue23455 opened by dalke #23456: asyncio: add missing @coroutine decorators http://bugs.python.org/issue23456 opened by haypo #23457: make test failures http://bugs.python.org/issue23457 opened by dcrs6000 at gmail.com #23458: [2.7] random: make the file descriptor non-inheritable (on POS http://bugs.python.org/issue23458 opened by haypo #23459: Linux: expose the new execveat() syscall http://bugs.python.org/issue23459 opened by haypo Most recent 15 issues with no replies (15) ========================================== #23459: Linux: expose the new execveat() syscall http://bugs.python.org/issue23459 #23457: make test failures http://bugs.python.org/issue23457 #23456: asyncio: add missing @coroutine decorators http://bugs.python.org/issue23456 #23455: file iterator "deemed broken"; can resume after StopIteration http://bugs.python.org/issue23455 #23452: Build errors using VS Express 2013 in win32 mode http://bugs.python.org/issue23452 #23444: HCI Bluetooth socket bind error on an arm crosscompiled enviro http://bugs.python.org/issue23444 #23443: XMLRPCLIB Exception uses str not class or instance http://bugs.python.org/issue23443 #23440: Extend http.server.SimpleHTTPRequestHandler testing http://bugs.python.org/issue23440 #23436: xml.dom.minidom.Element.ownerDocument is hiden http://bugs.python.org/issue23436 #23430: socketserver.BaseServer.handle_error() should not catch exitin http://bugs.python.org/issue23430 #23425: Windows getlocale unix-like with french, german, portuguese, s http://bugs.python.org/issue23425 #23423: XPath Support in ElementTree doc omission http://bugs.python.org/issue23423 #23422: Clarify docs for importlib.import_module() http://bugs.python.org/issue23422 #23420: python -m cProfile -s fails with non informative message http://bugs.python.org/issue23420 #23419: Faster default __reduce__ for classes without __init__ http://bugs.python.org/issue23419 Most recent 15 issues waiting for review (15) ============================================= #23458: [2.7] random: make the file descriptor non-inheritable (on POS http://bugs.python.org/issue23458 #23456: asyncio: add missing @coroutine decorators http://bugs.python.org/issue23456 #23451: Deprecation warnings building 3.5 Visual Studio Windows 8.1 64 http://bugs.python.org/issue23451 #23450: Possible loss of data warnings building 3.5 Visual Studio Wind http://bugs.python.org/issue23450 #23446: Use PyMem_New instead of PyMem_Malloc http://bugs.python.org/issue23446 #23444: HCI Bluetooth socket bind error on an arm crosscompiled enviro http://bugs.python.org/issue23444 #23442: http.client.REQUEST_HEADER_FIELDS_TOO_LARGE renamed in 3.5 http://bugs.python.org/issue23442 #23441: rlcompleter: tab on empty prefix => insert spaces http://bugs.python.org/issue23441 #23440: Extend http.server.SimpleHTTPRequestHandler testing http://bugs.python.org/issue23440 #23439: Fixed http.client.__all__ and added a test http://bugs.python.org/issue23439 #23437: Make user scripts directory versioned on Windows http://bugs.python.org/issue23437 #23434: RFC6266 support (Content-Disposition for HTTP) http://bugs.python.org/issue23434 #23432: Duplicate content in SystemExit documentation http://bugs.python.org/issue23432 #23430: socketserver.BaseServer.handle_error() should not catch exitin http://bugs.python.org/issue23430 #23428: Use the monotonic clock for thread conditions on POSIX platfor http://bugs.python.org/issue23428 Top 10 most discussed issues (10) ================================= #22524: PEP 471 implementation: os.scandir() directory scanning functi http://bugs.python.org/issue22524 13 msgs #23450: Possible loss of data warnings building 3.5 Visual Studio Wind http://bugs.python.org/issue23450 11 msgs #17911: traceback: add a new thin class storing a traceback without st http://bugs.python.org/issue17911 8 msgs #23328: urllib2 fails for proxy credentials that contain a '/' charact http://bugs.python.org/issue23328 8 msgs #23404: 'make touch' does not work with git clones of the source repos http://bugs.python.org/issue23404 8 msgs #19143: Finding the Windows version getting messier (detect windows 8. http://bugs.python.org/issue19143 7 msgs #21793: httplib client/server status refactor http://bugs.python.org/issue21793 7 msgs #23442: http.client.REQUEST_HEADER_FIELDS_TOO_LARGE renamed in 3.5 http://bugs.python.org/issue23442 7 msgs #23314: Disabling CRT asserts in debug build http://bugs.python.org/issue23314 6 msgs #23342: run() - unified high-level interface for subprocess http://bugs.python.org/issue23342 6 msgs Issues closed (26) ================== #7200: multiprocessing deadlock on Mac OS X when queue collected befo http://bugs.python.org/issue7200 closed by davin #9122: Problems with multiprocessing, Python embedding and Windows http://bugs.python.org/issue9122 closed by davin #9207: multiprocessing occasionally spits out exception during shutdo http://bugs.python.org/issue9207 closed by davin #11240: Running unit tests in a command line tool leads to infinite lo http://bugs.python.org/issue11240 closed by davin #12954: Multiprocessing logging under Windows http://bugs.python.org/issue12954 closed by davin #17023: Subprocess does not find executable on Windows if it is PATH w http://bugs.python.org/issue17023 closed by tim.golden #20416: Marshal: performance regression with versions 3 and 4 http://bugs.python.org/issue20416 closed by serhiy.storchaka #21568: Win32 pip doesn't use relative path to found site-packages. http://bugs.python.org/issue21568 closed by steve.dower #21840: Fix os.path in unicodeless build http://bugs.python.org/issue21840 closed by serhiy.storchaka #22430: Build failure if configure flags --prefix or --exec-prefix is http://bugs.python.org/issue22430 closed by ned.deily #22864: Add filter to multiprocessing.Pool http://bugs.python.org/issue22864 closed by davin #23344: Faster marshalling http://bugs.python.org/issue23344 closed by serhiy.storchaka #23357: pyvenv help shows incorrect usage http://bugs.python.org/issue23357 closed by python-dev #23361: integer overflow in winapi_createprocess http://bugs.python.org/issue23361 closed by python-dev #23383: Clean up bytes formatting http://bugs.python.org/issue23383 closed by serhiy.storchaka #23412: importlib sometimes fails to import a recently created module http://bugs.python.org/issue23412 closed by brett.cannon #23413: Incorrect division result http://bugs.python.org/issue23413 closed by benjamin.peterson #23417: Windows 8.1 and Windows Server 2012 R2 are not displayed prope http://bugs.python.org/issue23417 closed by steve.dower #23421: tarfile module does not correctly choose compression algorithm http://bugs.python.org/issue23421 closed by serhiy.storchaka #23429: -5**4 returns -625 instead of 625 http://bugs.python.org/issue23429 closed by serhiy.storchaka #23433: undefined behaviour in faulthandler.c, exposed by GCC 5 http://bugs.python.org/issue23433 closed by haypo #23435: installation with full path as prefix incomplete http://bugs.python.org/issue23435 closed by ned.deily #23438: HTMLParser doesn't know how to deal with 'ampersand' http://bugs.python.org/issue23438 closed by amaury.forgeotdarc #23445: Use -Og for debug builds http://bugs.python.org/issue23445 closed by pitrou #23447: Relative imports with __all__ attribute http://bugs.python.org/issue23447 closed by brett.cannon #23454: plistlib and xml.parsers.expat python3 problems with strings a http://bugs.python.org/issue23454 closed by serhiy.storchaka From contact at ionelmc.ro Fri Feb 13 16:58:36 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Fri, 13 Feb 2015 17:58:36 +0200 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> Message-ID: Can we at least make it use the constructor (if there's a custom one)? Seems like a reasonable compromise to me (let whoever implements a custom __new__ deal with argument variance). Eg, make it use a __new__ like this: >>> class FancyInt(int): ... def __new__(self, value): ... return int.__new__(FancyInt, value) ... ... def __repr__(self): ... return "FancyInt(%s)" % super().__repr__() ... >>> x = FancyInt(1) >>> >>> x FancyInt(1) >>> x += 1 >>> x # it should be FancyInt(2) 2 Thanks, -- Ionel Cristian M?rie?, blog.ionelmc.ro On Fri, Feb 13, 2015 at 6:01 AM, Guido van Rossum wrote: > On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman wrote: > >> On 02/12/2015 06:39 PM, Alexander Belopolsky wrote: >> >> > In my view, a constructor is no different from any other method. If >> the designers of the subclass decided to change the >> > signature in an incompatible way, they should either override all >> methods that create new objects or live with tracebacks. >> >> > On the other hand, if all I want in my Date class is a better >> __format__ method, I am forced to override all operators >> > or have my objects silently degrade [...] >> >> So there are basically two choices: >> >> 1) always use the type of the most-base class when creating new instances >> >> pros: >> - easy >> - speedy code >> - no possible tracebacks on new object instantiation >> >> cons: >> - a subclass that needs/wants to maintain itself must override all >> methods that create new instances, even if the only change is to >> the type of object returned >> >> 2) always use the type of self when creating new instances >> >> pros: >> - subclasses automatically maintain type >> - much less code in the simple cases [1] >> >> cons: >> - if constructor signatures change, must override all methods which >> create new objects >> >> Unless there are powerful reasons against number 2 (such as performance, >> or the effort to affect the change), it sure >> seems like the nicer way to go. >> >> So back to my original question: what other concerns are there, and has >> anybody done any benchmarks? >> > > Con for #2 is a showstopper. Forget about it. > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/contact%40ionelmc.ro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Feb 13 18:31:44 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 13 Feb 2015 09:31:44 -0800 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: I vote for the C implementation. On Fri, Feb 13, 2015 at 2:07 AM, Victor Stinner wrote: > Hi, > > TL,DR: are you ok to add 800 lines of C code for os.scandir(), 4x > faster than os.listdir() when the file type is checked? > > I accepted the PEP 471 (os.scandir) a few months ago, but it is not > implement yet in Python 3.5, because I didn't make a choice on the > implementation. > > Ben Hoyt wrote different implementations: > - full C: os.scandir() and DirEntry are written in C (no change on os.py) > - C+Python: os._scandir() (wrapper for opendir/readdir and > FindFirstFileW/FindNextFileW) in C, DirEntry in Python > - ctypes: os.scandir() and DirEntry fully implemented in Python > > I'm not interested by the ctypes implementation. It's useful for a > third party project hosted at PyPI, but for CPython I prefer to wrap C > functions using C code. > > > In short, the C implementation is faster than the C+Python implementation. > > The issue #22524 (*) is full of benchmark numbers. IMO the most > interesting benchmark is to compare os.listdir() + os.stat() versus > os.scandir() + Direntry.is_dir(). Let me try to summarize results of > this benchmark: > > * C implementation: scandir is at least 3.5x faster than listdir, up > to 44.6x faster on Windows > * C+Python implementation: scandir is not really faster than listdir, > between 1.3x and 1.4x faster > > (*) http://bugs.python.org/issue22524 > > > Ben Hoyt reminded me that os.scandir() (PEP 471) doesn't add any new > feature: pathlib already provides a nice API on top of os and os.path > modules. (You may even notice that DirEntry a much fewer methods ;-)) > The main (only?) purpose of the PEP is performance. > > If os.scandir() is "only" 1.4x faster, I don't think that it is > interesting to use os.scandir() in an application. I guess that all > applications/libraries will want to keep compatibility with Python 3.4 > and older and so will anyway have to duplicate the code to use > os.listdir() + os.stat(). So is it worth to duplicate code for such > small speedup? > > Now I see 3 choices: > > - take the full C implementation, because it's much faster (at least > 3.4x faster!) > - reject the whole PEP 471 (not nice), because it adds too much code > for a minor speedup (not true on Windows: up to 44x faster!) > - take the C+Python implementation, because maintenance matters more > than performances (only 1.3x faster, sorry) > > => IMO the best option is to take the C implementation. What do you think? > > > I'm concerned by the length of the C code: the full C implementations > adds ~800 lines of C code to posixmodule.c. This file is already the > longest C file in CPython. I don't want to make it longer, but I'm not > motived to start to split it. Last time I proposed to split a file > (unicodeobject.c), some developers complained that it makes search > harder. I don't understand this, there are so many tools to navigate > in C code. But it was enough for me to give up on this idea. > > A alternative is to add a new _scandir.c module to host the new C > code, and share some code with posixmodule.c: remove "static" keyword > from required C functions (functions to convert Windows attributes to > a os.stat_result object). That's a reasonable choice. What do you > think? > > > FYI I ran the benchmark on different hardware (SSD, HDD, tmpfs), file > systems (ext4, tmpfs, NFS/ext4), operating systems (Linux, Windows). > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Feb 13 18:35:15 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 13 Feb 2015 09:35:15 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DD8479.8050600@stoneleaf.us> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> Message-ID: On Thu, Feb 12, 2015 at 8:58 PM, Ethan Furman wrote: > On 02/12/2015 08:01 PM, Guido van Rossum wrote: > > On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman wrote: > >> > >> 2) always use the type of self when creating new instances > >> > >> cons: > >> - if constructor signatures change, must override all methods > which > >> create new objects > > > > Con for #2 is a showstopper. Forget about it. > > Happy to, but can you explain why requiring the programmer to override the > necessary methods, or get tracebacks, is a > showstopper? Is there a previous email thread I can read that discusses > it? > IIUC you're proposing that the base class should *try* to construct an instance of the subclass by calling the type with an argument, and fail if it doesn't work. But that makes the whole thing brittle in the light of changes to the subclass constructor. Also, what should the argument be? The only answer I can think of is an instance of the base class. Finally, this would require more special-casing in every built-in class (at least every built-in class that sometimes returns instances of itself). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Feb 13 18:41:44 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 13 Feb 2015 09:41:44 -0800 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: On 13 Feb 2015 02:09, "Victor Stinner" wrote: > > A alternative is to add a new _scandir.c module to host the new C > code, and share some code with posixmodule.c: remove "static" keyword > from required C functions (functions to convert Windows attributes to > a os.stat_result object). Hopefully not too annoying question from an outsider: has cpython's build system added the necessary bits to do this on a safe, portable, non-symbol-namespace polluting way? E.g. using -fvisibility=hidden on Linux? (I'm partially wondering because until very recently numpy was built by concatenating all the different c files together and compiling that, because that was the only portable way to let different files share access to symbols without also exporting those symbols publicly from the resulting module shared objects. And numpy supports a lot fewer platforms than cpython proper...) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Feb 13 19:02:41 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 13 Feb 2015 13:02:41 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> Message-ID: On Fri, Feb 13, 2015 at 12:35 PM, Guido van Rossum wrote: > IIUC you're proposing that the base class should *try* to construct an > instance of the subclass by calling the type with an argument, and fail if > it doesn't work. But that makes the whole thing brittle in the light of > changes to the subclass constructor. Also, what should the argument be? The > only answer I can think of is an instance of the base class. No. The arguments should be whatever arguments are appropriate for the baseclass's __init__ or __new__. In the case of datetime.date that would be year, month, day. Note that the original pure python prototype of the datetime module had date.__add__ and friends call self.__class__(year, month, day). Unfortunately, it looks like the original sandbox did not survive the the hg conversion, so I cannot provide a link to the relevant history. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Feb 13 19:11:56 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 13 Feb 2015 10:11:56 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> Message-ID: On Fri, Feb 13, 2015 at 10:02 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Fri, Feb 13, 2015 at 12:35 PM, Guido van Rossum > wrote: > >> IIUC you're proposing that the base class should *try* to construct an >> instance of the subclass by calling the type with an argument, and fail if >> it doesn't work. But that makes the whole thing brittle in the light of >> changes to the subclass constructor. Also, what should the argument be? The >> only answer I can think of is an instance of the base class. > > > No. The arguments should be whatever arguments are appropriate for the > baseclass's __init__ or __new__. In the case of datetime.date that would > be year, month, day. > Agreed. (I was thinking of the case that Ethan brought up, which used int as an example.) > Note that the original pure python prototype of the datetime module had > date.__add__ and friends call self.__class__(year, month, day). > Unfortunately, it looks like the original sandbox did not survive the the > hg conversion, so I cannot provide a link to the relevant history. > FWIW you're wrong when you claim that "a constructor is no different from any other method". Someone else should probably explain this (it's an old argument that's been thoroughly settled). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Feb 13 19:19:08 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 13 Feb 2015 13:19:08 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> Message-ID: On Fri, Feb 13, 2015 at 1:11 PM, Guido van Rossum wrote: > > >> Note that the original pure python prototype of the datetime module had >> date.__add__ and friends call self.__class__(year, month, day). >> Unfortunately, it looks like the original sandbox did not survive the the >> hg conversion, so I cannot provide a link to the relevant history. >> > > FWIW you're wrong when you claim that "a constructor is no different from > any other method". Someone else should probably explain this (it's an old > argument that's been thoroughly settled). > Well, the best answer I've got in the past [1] was "ask on python-dev since Guido called the operator overriding expectation." :-) [1] http://bugs.python.org/issue2267#msg108060 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Feb 13 19:22:18 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 13 Feb 2015 13:22:18 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> Message-ID: On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: >> >> FWIW you're wrong when you claim that "a constructor is no different from any other method". Someone else should probably explain this (it's an old argument that's been thoroughly settled). > > > Well, the best answer I've got in the past [1] was "ask on python-dev since Guido called the operator overriding expectation." :-) And let me repost this bit of history [1]: Here is the annotated pre-r82065 code: 39876 gvanrossum def __add__(self, other): 39876 gvanrossum if isinstance(other, timedelta): 39928 gvanrossum return self.__class__(self.__days + other.__days, 39876 gvanrossum self.__seconds + other.__seconds, 39876 gvanrossum self.__microseconds + other.__microseconds) 40207 tim_one return NotImplemented 39876 gvanrossum [1] http://bugs.python.org/issue2267#msg125979 -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Feb 13 19:25:15 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 13 Feb 2015 10:25:15 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> Message-ID: Are you willing to wait 10 days for an answer? I'm out of round tuits for a while. On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > >> > >> FWIW you're wrong when you claim that "a constructor is no different > from any other method". Someone else should probably explain this (it's an > old argument that's been thoroughly settled). > > > > > > Well, the best answer I've got in the past [1] was "ask on python-dev > since Guido called the operator overriding expectation." :-) > > > And let me repost this bit of history [1]: > > Here is the annotated pre-r82065 code: > > 39876 gvanrossum def __add__(self, other): > 39876 gvanrossum if isinstance(other, timedelta): > 39928 gvanrossum return self.__class__(self.__days + > other.__days, > 39876 gvanrossum self.__seconds + > other.__seconds, > 39876 gvanrossum self.__microseconds + > other.__microseconds) > 40207 tim_one return NotImplemented > 39876 gvanrossum > > > [1] http://bugs.python.org/issue2267#msg125979 > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Feb 13 20:28:15 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 13 Feb 2015 14:28:15 -0500 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: On Fri, Feb 13, 2015 at 5:07 AM, Victor Stinner wrote: > Now I see 3 choices: > > - take the full C implementation, because it's much faster (at least > 3.4x faster!) > - reject the whole PEP 471 (not nice), because it adds too much code > for a minor speedup (not true on Windows: up to 44x faster!) > - take the C+Python implementation, because maintenance matters more > than performances (only 1.3x faster, sorry) > > => IMO the best option is to take the C implementation. What do you think? > +1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Feb 13 22:30:05 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 21:30:05 +0000 Subject: [Python-Dev] PEP 486: Make the Python Launcher aware of virtual environments In-Reply-To: References: Message-ID: On 12 February 2015 at 19:44, Paul Moore wrote: > Impact on Script Launching > ========================== Now that I'm looking into the details of the code for the launcher, I've noticed that a shebang line of "#!/usr/bin/env python" will first of all search PATH for a python executable, before falling back to the default Python. This is *not* specified in PEP 397, although it makes sense (as it parallels the behaviour in Unix). Given that this behaviour exists, doing an explicit check for VIRTUAL_ENV during shebang processing is unnecessary. The same effect can be gained just by using "#!/usr/bin/env python" (it won't cover the case of someone setting VIRTUAL_ENV but *not* adding it to PATH, but that's no great loss). I propose simply dropping this section of the PEP. I'll replace it with a note explaining why it was dropped, and explaining the behaviour of /usr/bin/env. Is that sufficient? (By the way, on a procedural note, how do I update a PEP? Do I just send an updated version to peps at python.org, or is there a better way?) Paul From ischwabacher at wisc.edu Fri Feb 13 21:37:27 2015 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Fri, 13 Feb 2015 14:37:27 -0600 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <76b0c98db1162.54de6062@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> Message-ID: <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> On 15-02-13, Guido van Rossum wrote: > Are you willing to wait 10 days for an answer? I'm out of round tuits for a while. IIUC, the argument is that the Liskov Substitution Principle is a statement about how objects of a subtype behave relative to objects of a supertype, and it doesn't apply to constructors because they aren't behaviors of existing objects. So other overriding methods *should* be able to handle the same inputs that the respective overridden methods do, but constructors don't need to. Even though __init__ is written as an instance method, it seems like it's "morally" a part of the class method __new__ that's only split off for convenience. If this message is unclear, it's because I don't really understand this myself and I'm trying to articulate my best understanding of what's been said on this thread and those it links to. ijs > On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky wrote: > > > > > On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky wrote: > > >> > > >> FWIW you're wrong when you claim that "a constructor is no different from any other method". Someone else should probably explain this (it's an old argument that's been thoroughly settled). > > > > > > > > > Well, the best answer I've got in the past [1] was "ask on python-dev since Guido called the operator overriding expectation." :-) > > > > > > And let me repost this bit of history [1]: > > > > Here is the annotated pre-r82065 code: > > > > 39876 gvanrossum def __add__(self, other): > > 39876 gvanrossum if isinstance(other, timedelta): > > 39928 gvanrossum return self.__class__(self.__days + other.__days, > > 39876 gvanrossum self.__seconds + other.__seconds, > > 39876 gvanrossum self.__microseconds + other.__microseconds) > > 40207 tim_one return NotImplemented > > 39876 gvanrossum > > > > > > > > [1]?http://bugs.python.org/issue2267#msg125979 > > > > > > > > > -- > --Guido van Rossum (python.org/~guido(http://python.org/~guido)) From mistersheik at gmail.com Fri Feb 13 22:44:12 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 13 Feb 2015 16:44:12 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> Message-ID: Interesting: http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle On Fri, Feb 13, 2015 at 3:37 PM, Isaac Schwabacher wrote: > On 15-02-13, Guido van Rossum wrote: > > Are you willing to wait 10 days for an answer? I'm out of round > tuits for a while. > > IIUC, the argument is that the Liskov Substitution Principle is a > statement about how objects of a subtype behave relative to objects of a > supertype, and it doesn't apply to constructors because they aren't > behaviors of existing objects. So other overriding methods *should* be able > to handle the same inputs that the respective overridden methods do, but > constructors don't need to. Even though __init__ is written as an instance > method, it seems like it's "morally" a part of the class method __new__ > that's only split off for convenience. > > If this message is unclear, it's because I don't really understand this > myself and I'm trying to articulate my best understanding of what's been > said on this thread and those it links to. > > ijs > > > On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky < > alexander.belopolsky at gmail.com(javascript:main.compose()> wrote: > > > > > > > > On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky < > alexander.belopolsky at gmail.com(javascript:main.compose()> wrote: > > > >> > > > >> FWIW you're wrong when you claim that "a constructor is no > different from any other method". Someone else should probably explain this > (it's an old argument that's been thoroughly settled). > > > > > > > > > > > > Well, the best answer I've got in the past [1] was "ask on > python-dev since Guido called the operator overriding expectation." :-) > > > > > > > > > And let me repost this bit of history [1]: > > > > > > Here is the annotated pre-r82065 code: > > > > > > 39876 gvanrossum def __add__(self, other): > > > 39876 gvanrossum if isinstance(other, timedelta): > > > 39928 gvanrossum return self.__class__(self.__days + other.__days, > > > 39876 gvanrossum self.__seconds + other.__seconds, > > > 39876 gvanrossum self.__microseconds + other.__microseconds) > > > 40207 tim_one return NotImplemented > > > 39876 gvanrossum > > > > > > > > > > > > [1] http://bugs.python.org/issue2267#msg125979 > > > > > > > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido(http://python.org/~guido)) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ischwabacher at wisc.edu Fri Feb 13 22:50:47 2015 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Fri, 13 Feb 2015 15:50:47 -0600 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <7460853cb295d.54de71ad@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <75c0da14b61e6.54de7132@wiscmail.wisc.edu> <75e09c5cb5ce8.54de716e@wiscmail.wisc.edu> <7460853cb295d.54de71ad@wiscmail.wisc.edu> Message-ID: <746083cdb703c.54de1d57@wiscmail.wisc.edu> On 15-02-13, Neil Girdhar wrote: > Interesting:?http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle You'd better believe I read that thread not 20 minutes ago. :) > On Fri, Feb 13, 2015 at 3:37 PM, Isaac Schwabacher ischwabacher at wisc.edu> wrote: > > > On 15-02-13, Guido van Rossum wrote: > > > Are you willing to wait 10 days for an answer? I'm out of round tuits for a while. > > > > IIUC, the argument is that the Liskov Substitution Principle is a statement about how objects of a subtype behave relative to objects of a supertype, and it doesn't apply to constructors because they aren't behaviors of existing objects. So other overriding methods *should* be able to handle the same inputs that the respective overridden methods do, but constructors don't need to. Even though __init__ is written as an instance method, it seems like it's "morally" a part of the class method __new__ that's only split off for convenience. > > > > If this message is unclear, it's because I don't really understand this myself and I'm trying to articulate my best understanding of what's been said on this thread and those it links to. > > > > ijs > > > > > On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky (java_script:main.compose()> wrote: > > > > > > > > > > > On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky (java_script:main.compose()> wrote: > > > > >> > > > > >> FWIW you're wrong when you claim that "a constructor is no different from any other method". Someone else should probably explain this (it's an old argument that's been thoroughly settled). > > > > > > > > > > > > > > > Well, the best answer I've got in the past [1] was "ask on python-dev since Guido called the operator overriding expectation." :-) > > > > > > > > > > > > And let me repost this bit of history [1]: > > > > > > > > Here is the annotated pre-r82065 code: > > > > > > > > 39876 gvanrossum def __add__(self, other): > > > > 39876 gvanrossum if isinstance(other, timedelta): > > > > 39928 gvanrossum return self.__class__(self.__days + other.__days, > > > > 39876 gvanrossum self.__seconds + other.__seconds, > > > > 39876 gvanrossum self.__microseconds + other.__microseconds) > > > > 40207 tim_one return NotImplemented > > > > 39876 gvanrossum > > > > > > > > > > > > > > > > [1]?> --Guido van Rossum (python.org/~guido(http://python.org/~guido)(http://python.org/~guido(http://python.org/~guido))) > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > From guido at python.org Fri Feb 13 22:55:03 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 13 Feb 2015 13:55:03 -0800 Subject: [Python-Dev] (no subject) In-Reply-To: <54DBB872.2080409@canterbury.ac.nz> References: <2D213CE9-B396-46AB-8520-6ED7DCC38B04@stufft.io> <54DBB872.2080409@canterbury.ac.nz> Message-ID: I am still in favor of this PEP but have run out of time to review it and the feedback. I'm going on vacation for a week or so, maybe I'll find time, if not I'll start reviewing this around Feb 23. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Feb 13 23:09:04 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 22:09:04 +0000 Subject: [Python-Dev] Building on Windows - importlib.h changed Message-ID: I'm working on a patch for the Python launcher. I built Python (current tip, on MS Windows, with VS 2015), and I've just noticed that hg status shows: >>hg status -mard M Doc\using\windows.rst M PC\launcher.c M Python\importlib.h I didn't change importlib.h, and I don't see that the changes I did make would affect importlib. I presume I'm OK *not* including the importlib.h change in my patch? Paul PS By the way, the Windows build process is beautifully clean these days. Many thanks to Steve Dower and whoever else has worked on streamlining it! From storchaka at gmail.com Fri Feb 13 23:31:14 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 14 Feb 2015 00:31:14 +0200 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DD7262.4040000@stoneleaf.us> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> Message-ID: On 13.02.15 05:41, Ethan Furman wrote: > So there are basically two choices: > > 1) always use the type of the most-base class when creating new instances > > pros: > - easy > - speedy code > - no possible tracebacks on new object instantiation > > cons: > - a subclass that needs/wants to maintain itself must override all > methods that create new instances, even if the only change is to > the type of object returned > > 2) always use the type of self when creating new instances > > pros: > - subclasses automatically maintain type > - much less code in the simple cases [1] > > cons: > - if constructor signatures change, must override all methods which > create new objects And switching to (2) would break existing code which uses subclasses with constructors with different signature (e.g. defaultdict). The third choice is to use different specially designed constructor. class A(int): >>> class A(int): ... def __add__(self, other): ... return self.__make_me__(int(self) + int(other)) ... def __repr__(self): ... return 'A(%d)' % self ... >>> A.__make_me__ = A >>> A(2) + 3 A(5) >>> class B(A): ... def __repr__(self): ... return 'B(%d)' % self ... >>> B.__make_me__ = B >>> B(2) + 3 B(5) We can add special attribute used to creating results of operations to all basic classes. By default it would be equal to the base class constructor. From zachary.ware+pydev at gmail.com Fri Feb 13 23:49:47 2015 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 13 Feb 2015 16:49:47 -0600 Subject: [Python-Dev] Building on Windows - importlib.h changed In-Reply-To: References: Message-ID: On Fri, Feb 13, 2015 at 4:09 PM, Paul Moore wrote: > I'm working on a patch for the Python launcher. I built Python > (current tip, on MS Windows, with VS 2015), and I've just noticed that > hg status shows: > >>>hg status -mard > M Doc\using\windows.rst > M PC\launcher.c > M Python\importlib.h > > I didn't change importlib.h, and I don't see that the changes I did > make would affect importlib. I presume I'm OK *not* including the > importlib.h change in my patch? Yes, importlib.h changes should never be included in a patch (it would make the patch extremely huge for no benefit). It's strange that you're getting modifications, though; I can't reproduce on Linux (and don't have quick access to Windows right now). Make sure you're definitely at tip, and if it's still changing please open an issue. -- Zach From alexander.belopolsky at gmail.com Fri Feb 13 23:55:37 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 13 Feb 2015 17:55:37 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> Message-ID: On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar wrote: > Interesting: > http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle > Let me humbly conjecture that the people who wrote the top answers have background in less capable languages than Python. Not every language allows you to call self.__class__(). In the languages that don't you can get away with incompatible constructor signatures. However, let me try to focus the discussion on a specific issue before we go deep into OOP theory. With python's standard datetime.date we have: >>> from datetime import * >>> class Date(date): ... pass ... >>> Date.today() Date(2015, 2, 13) >>> Date.fromordinal(1) Date(1, 1, 1) Both .today() and .fromordinal(1) will break in a subclass that redefines __new__ as follows: >>> class Date2(date): ... def __new__(cls, ymd): ... return date.__new__(cls, *ymd) ... >>> Date2.today() Traceback (most recent call last): File "", line 1, in TypeError: __new__() takes 2 positional arguments but 4 were given >>> Date2.fromordinal(1) Traceback (most recent call last): File "", line 1, in TypeError: __new__() takes 2 positional arguments but 4 were given Why is this acceptable, but we have to sacrifice the convenience of having Date + timedelta return Date to make it work with Date2: >>> Date2((1,1,1)) + timedelta(1) datetime.date(1, 1, 2) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Feb 13 23:56:43 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 13 Feb 2015 15:56:43 -0700 Subject: [Python-Dev] Building on Windows - importlib.h changed In-Reply-To: References: Message-ID: On Fri, Feb 13, 2015 at 3:49 PM, Zachary Ware wrote: > On Fri, Feb 13, 2015 at 4:09 PM, Paul Moore wrote: >> I'm working on a patch for the Python launcher. I built Python >> (current tip, on MS Windows, with VS 2015), and I've just noticed that >> hg status shows: >> >>>>hg status -mard >> M Doc\using\windows.rst >> M PC\launcher.c >> M Python\importlib.h >> >> I didn't change importlib.h, and I don't see that the changes I did >> make would affect importlib. I presume I'm OK *not* including the >> importlib.h change in my patch? > > Yes, importlib.h changes should never be included in a patch (it would Unless they should. :) E.g. you modified importlib/_bootstrap.py, the marshal format, bytecodes, etc. -eric From mistersheik at gmail.com Sat Feb 14 00:03:35 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 13 Feb 2015 18:03:35 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> Message-ID: I personally don't think this is a big enough issue to warrant any changes, but I think Serhiy's solution would be the ideal best with one additional parameter: the caller's type. Something like def __make_me__(self, cls, *args, **kwargs) and the idea is that any time you want to construct a type, instead of self.__class__(assumed arguments?) where you are not sure that the derived class' constructor knows the right argument types, you do def SomeCls: def some_method(self, ...): return self.__make_me__(SomeCls, assumed arguments?) Now the derived class knows who is asking for a copy. In the case of defaultdict, for example, he can implement __make_me__ as follows: def __make_me__(self, cls, *args, **kwargs): if cls is dict: return default_dict(self.default_factory, *args, **kwargs) return default_dict(*args, **kwargs) essentially the caller is identifying himself so that the receiver knows how to interpret the arguments. Best, Neil On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar > wrote: > >> Interesting: >> http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle >> > > Let me humbly conjecture that the people who wrote the top answers have > background in less capable languages than Python. > > Not every language allows you to call self.__class__(). In the languages > that don't you can get away with incompatible constructor signatures. > > However, let me try to focus the discussion on a specific issue before we > go deep into OOP theory. > > With python's standard datetime.date we have: > > >>> from datetime import * > >>> class Date(date): > ... pass > ... > >>> Date.today() > Date(2015, 2, 13) > >>> Date.fromordinal(1) > Date(1, 1, 1) > > Both .today() and .fromordinal(1) will break in a subclass that redefines > __new__ as follows: > > >>> class Date2(date): > ... def __new__(cls, ymd): > ... return date.__new__(cls, *ymd) > ... > >>> Date2.today() > Traceback (most recent call last): > File "", line 1, in > TypeError: __new__() takes 2 positional arguments but 4 were given > >>> Date2.fromordinal(1) > Traceback (most recent call last): > File "", line 1, in > TypeError: __new__() takes 2 positional arguments but 4 were given > > Why is this acceptable, but we have to sacrifice the convenience of having > Date + timedelta > return Date to make it work with Date2: > > >>> Date2((1,1,1)) + timedelta(1) > datetime.date(1, 1, 2) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Sat Feb 14 00:07:38 2015 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 13 Feb 2015 17:07:38 -0600 Subject: [Python-Dev] Building on Windows - importlib.h changed In-Reply-To: References: Message-ID: On Fri, Feb 13, 2015 at 4:56 PM, Eric Snow wrote: > On Fri, Feb 13, 2015 at 3:49 PM, Zachary Ware > wrote: >> Yes, importlib.h changes should never be included in a patch (it would > > Unless they should. :) E.g. you modified importlib/_bootstrap.py, the > marshal format, bytecodes, etc. If I'm not mistaken, the devguide suggests leaving generated files out of patches, though I can't find where to link to it. If not, it probably should :). Generated files ought to be generated by the build process the same way on any system, so leaving them in the patch amounts to unnecessary noise, especially in the case of importlib.h which is not human-readable (and huge). In the case of a generated file being affected by a patch, a friendly reminder that files will need to be regenerated should be included when posting the patch, though. -- Zach From p.f.moore at gmail.com Sat Feb 14 00:12:24 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 13 Feb 2015 23:12:24 +0000 Subject: [Python-Dev] Building on Windows - importlib.h changed In-Reply-To: References: Message-ID: On 13 February 2015 at 22:49, Zachary Ware wrote: > It's strange that > you're getting modifications, though; I can't reproduce on Linux (and > don't have quick access to Windows right now). Make sure you're > definitely at tip, and if it's still changing please open an issue. Definitely at tip. I just redid the build and confirmed it's reproducible. There is a message generated during the build that says importlib.h has changed. Reported at http://bugs.python.org/issue23461 Paul. From mistersheik at gmail.com Sat Feb 14 01:05:17 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 13 Feb 2015 19:05:17 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <75c08852b7610.54de3ba2@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <76a0a5e6b41d2.54de84bf@wiscmail.wisc.edu> <7460bc10b4e87.54de84fc@wiscmail.wisc.edu> <7740a7e8b66fe.54de85ec@wiscmail.wisc.edu> <76a09ea0b7783.54de86dd@wiscmail.wisc.edu> <76a0f0f5b15a4.54de8719@wiscmail.wisc.edu> <7610e537b528b.54de8755@wiscmail.wisc.edu> <7460f3ddb7500.54de87ce@wiscmail.wisc.edu> <7460ee8eb47e6.54de88fb@wiscmail.wisc.edu> <7590ea33b4e12.54de8938@wiscmail.wisc.edu> <7590ce56b3f59.54de8974@wiscmail.wisc.edu> <75e0b16fb2e22.54de89b0@wiscmail.wisc.edu> <75a0e6fdb13fb.54de89ed@wiscmail.wisc.edu> <76a0b5beb787c.54de8a68@wiscmail.wisc.edu> <7590a705b0317.54de8aa7@wiscmail.wisc.edu> <75c08965b4498.54de8ae3@wiscmail.wisc.edu> <75e080b6b454a.54de8b22@wiscmail.wisc.edu> <7740cad7b4553.54de8b5e@wiscmail.wisc.edu> <75a0d194b27a6.54de8b9b@wiscmail.wisc.edu> <75e0d877b71fb.54de8c13@wiscmail.wisc.edu> <7690c481b75f3.54de8c52@wiscmail.wisc.edu> <7460b72bb0e8d.54de8c8e@wiscmail.wisc.edu> <75e0efafb72e2.54de8ccb@wiscmail.wisc.edu> <75a0847db1b11.54de8d07@wiscmail.wisc.edu> <76a0d81ab1672.54de8d44@wiscmail.wisc.edu> <75909961b7d81.54de8d80@wiscmail.wisc.edu> <75908c67b7a2a.54de8dbc@wiscmail.wisc.edu> <7590d2abb120c.54de8df9@wiscmail.wisc.edu> <7590b578b2c45.54de8e35@wiscmail.wisc.edu> <7690d252b6239.54de8e72@wiscmail.wisc.edu> <769098cbb63a2.54de8eae@wiscmail.wisc.edu> <762093bbb5f17.54de8eea@wiscmail.wisc.edu> <7610a1c2b5bea.54de8f27@wiscmail.wisc.edu> <75c0a487b271d.54de8f63@wiscmail.wisc.edu> <75a0da55b1347.54de8fa0@wiscmail.wisc.edu> <75c0d012b000e.54de8fdc@wiscmail.wisc.edu> <75c08852b7610.54de3ba2@wiscmail.wisc.edu> Message-ID: Unlike a regular method, you would never need to call super since you should know everyone that could be calling you. Typically, when you call super, you have something like this: A < B, C B < D so you end up with mro: A, B, C, D And then when A calls super and B calls super it gets C which it doesn't know about. But in the case of make_me, it's someone like C who is calling make_me. If it gets a method in B, then that's a straight-up bug. make_me needs to be reimplemented in A as well, and A would never delegate up since other classes in the mro chain (like B) might not know about C. Best, Neil On Fri, Feb 13, 2015 at 7:00 PM, Isaac Schwabacher wrote: > On 15-02-13, Neil Girdhar wrote: > > I personally don't think this is a big enough issue to warrant any > changes, but I think Serhiy's solution would be the ideal best with one > additional parameter: the caller's type. Something like > > > > def __make_me__(self, cls, *args, **kwargs) > > > > > > and the idea is that any time you want to construct a type, instead of > > > > > > self.__class__(assumed arguments?) > > > > > > where you are not sure that the derived class' constructor knows the > right argument types, you do > > > > > > def SomeCls: > > def some_method(self, ...): > > return self.__make_me__(SomeCls, assumed arguments?) > > > > > > Now the derived class knows who is asking for a copy. In the case of > defaultdict, for example, he can implement __make_me__ as follows: > > > > > > def __make_me__(self, cls, *args, **kwargs): > > if cls is dict: return default_dict(self.default_factory, *args, > **kwargs) > > return default_dict(*args, **kwargs) > > > > > > essentially the caller is identifying himself so that the receiver knows > how to interpret the arguments. > > > > > > Best, > > > > > > Neil > > Such a method necessarily involves explicit switching on classes... ew. > Also, to make this work, a class needs to have a relationship with its > superclass's superclasses. So in order for DefaultDict's subclasses not to > need to know about dict, it would need to look like this: > > class DefaultDict(dict): > .... at classmethod # instance method doesn't make sense here > ....def __make_me__(cls, base, *args, **kwargs): # make something like > base(*args, **kwargs) > ........# when we get here, nothing in cls.__mro__ above DefaultDict knows > how to construct an equivalent to base(*args, **kwargs) using its own > constructor > ........if base is DefaultDict: > ............return DefaultDict(*args, **kwargs) # if DefaultDict is the > best we can do, do it > ........elif base is dict: > ............return cls.__make_me__(DefaultDict, None, *args, **kwargs) # > subclasses that know about DefaultDict but not dict will intercept this > ........else: > ............super(DefaultDict, cls).__make_me__(base, *args, **kwargs) # > we don't know how to make an equivalent to base.__new__(*args, **kwargs), > so keep looking > > I don't even think this is guaranteed to construct an object of class cls > corresponding to a base(*args, **kwargs) even if it were possible, since > multiple inheritance can screw things up. You might need to have an > explicit list of "these are the superclasses whose constructors I can > imitate", and have the interpreter find an optimal path for you. > > > On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky < > alexander.belopolsky at gmail.com(javascript:main.compose()> wrote: > > > > > > > > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar > wrote: > > > > > > > Interesting: > http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle > > > > > > > > > > > > > Let me humbly conjecture that the people who wrote the top answers > have background in less capable languages than Python. > > > > > > > > > Not every language allows you to call self.__class__(). In the > languages that don't you can get away with incompatible constructor > signatures. > > > > > > > > > However, let me try to focus the discussion on a specific issue before > we go deep into OOP theory. > > > > > > > > > With python's standard datetime.date we have: > > > > > > > > > >>> from datetime import * > > > >>> class Date(date): > > > ... pass > > > ... > > > >>> Date.today() > > > Date(2015, 2, 13) > > > >>> Date.fromordinal(1) > > > Date(1, 1, 1) > > > > > > > > > Both .today() and .fromordinal(1) will break in a subclass that > redefines __new__ as follows: > > > > > > > > > >>> class Date2(date): > > > ... def __new__(cls, ymd): > > > ... return date.__new__(cls, *ymd) > > > ... > > > >>> Date2.today() > > > Traceback (most recent call last): > > > File "", line 1, in > > > TypeError: __new__() takes 2 positional arguments but 4 were given > > > >>> Date2.fromordinal(1) > > > Traceback (most recent call last): > > > File "", line 1, in > > > TypeError: __new__() takes 2 positional arguments but 4 were given > > > > > > > > > > > > > > > Why is this acceptable, but we have to sacrifice the convenience of > having Date + timedelta > > > return Date to make it work with Date2: > > > > > > > > > >>> Date2((1,1,1)) + timedelta(1) > > > datetime.date(1, 1, 2) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 14 01:07:03 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Feb 2015 10:07:03 +1000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DDCED7.9080606@g.nevcal.com> Message-ID: On 14 Feb 2015 01:47, "Paul Moore" wrote: > > On 13 February 2015 at 14:27, Steve Dower wrote: > > If py.exe starts defaulting to whatever is on PATH then I don't see the > > point of it. Knowing that you'll get the latest 2.x version by default is > > quite handy (I'd be quite okay changing that to 3.x, though it is specified > > in the original PEP). For me, the point of py.exe is to be able to ignore > > PATH completely. > > It does seem a bit bizarre to me that a launcher shipped with Python 3 > defaults to using Python 2 if it's available. I'd like to see that > change, but as it's specified in the original PEP I guess it might > meet with some pushback :-( If there's no Python 2 on the system it doesn't matter, and if they're both there, we opted to remain consistent with the Linux distro world. OTOH, it may be time to reconsider our recommendation to distros as well, suggesting that for Python 3.5+, we will consider it appropriate to have the "python" symlink refer to "python3". Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Feb 14 01:19:15 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 14 Feb 2015 13:19:15 +1300 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> Message-ID: <54DE9483.3040402@canterbury.ac.nz> Isaac Schwabacher wrote: > IIUC, the argument is that the Liskov Substitution Principle is a statement > about how objects of a subtype behave relative to objects of a supertype, and > it doesn't apply to constructors because they aren't behaviors of existing > objects. Another way to say that is that constructors are class methods, not instance methods. -- Greg From ischwabacher at wisc.edu Sat Feb 14 01:55:11 2015 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Fri, 13 Feb 2015 18:55:11 -0600 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <75c0db14b60ef.54de9ceb@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <76a0a5e6b41d2.54de84bf@wiscmail.wisc.edu> <7460bc10b4e87.54de84fc@wiscmail.wisc.edu> <7740a7e8b66fe.54de85ec@wiscmail.wisc.edu> <76a09ea0b7783.54de86dd@wiscmail.wisc.edu> <76a0f0f5b15a4.54de8719@wiscmail.wisc.edu> <7610e537b528b.54de8755@wiscmail.wisc.edu> <7460f3ddb7500.54de87ce@wiscmail.wisc.edu> <7460ee8eb47e6.54de88fb@wiscmail.wisc.edu> <7590ea33b4e12.54de8938@wiscmail.wisc.edu> <7590ce56b3f59.54de8974@wiscmail.wisc.edu> <75e0b16fb2e22.54de89b0@wiscmail.wisc.edu> <75a0e6fdb13fb.54de89ed@wiscmail.wisc.edu> <76a0b5beb787c.54de8a68@wiscmail.wisc.edu> <7590a705b0317.54de8aa7@wiscmail.wisc.edu> <75c08965b4498.54de8ae3@wiscmail.wisc.edu> <75e080b6b454a.54de8b22@wiscmail.wisc.edu> <7740cad7b4553.54de8b5e@wiscmail.wisc.edu> <75a0d194b27a6.54de8b9b@wiscmail.wisc.edu> <75e0d877b71fb.54de8c13@wiscmail.wisc.edu> <7690c481b75f3.54de8c52@wiscmail.wisc.edu> <7460b72bb0e8d.54de8c8e@wiscmail.wisc.edu> <75e0efafb72e2.54de8ccb@wiscmail.wisc.edu> <75a0847db1b11.54de8d07@wiscmail.wisc.edu> <76a0d81ab1672.54de8d44@wiscmail.wisc.edu> <75909961b7d81.54de8d80@wiscmail.wisc.edu> <75908c67b7a2a.54de8dbc@wiscmail.wisc.edu> <7590d2abb120c.54de8df9@wiscmail.wisc.edu> <7590b578b2c45.54de8e35@wiscmail.wisc.edu> <7690d252b6239.54de8e72@wiscmail.wisc.edu> <769098cbb63a2.54de8eae@wiscmail.wisc.edu> <762093bbb5f17.54de8eea@wiscmail.wisc.edu> <7610a1c2b5bea.54de8f27@wiscmail.wisc.edu> <75c0a487b271d.54de8f63@wiscmail.wisc.edu> <75a0da55b1347.54de8fa0@wiscmail.wisc.edu> <75c0d012b000e.54de8fdc@wiscmail.wisc.edu> <75c08852b7610.54de3ba2@wiscmail.wisc.edu> <75c0b241b693e.54de99dc@wiscmail.wisc.edu> <76a0df4ab5cf3.54de9b81@wiscmail.wisc.edu> <75c0edf2b6640.54de9bbd@wiscmail.wisc.edu> <7770d5a5b283d.54de9bfa@wiscmail.wisc.edu> <7620bdd7b005f.54de9c36@wiscmail.wisc.edu> <7790b166b55af.54de9c72@wiscmail.wisc.edu> <760084bab38cf.54de9caf@wiscmail.wisc.edu> <75c0db14b60ef.54de9ceb@wiscmail.wisc.edu> Message-ID: <75c0f102b2026.54de488f@wiscmail.wisc.edu> On 15-02-13, Neil Girdhar wrote: > Unlike a regular method, you would never need to call super since you should know everyone that could be calling you. Typically, when you call super, you have something like this: > > A < B, C > > > B < D > > > so you end up with > > > mro: A, B, C, D > > > And then when A calls super and B calls super it gets C which it doesn't know about. But C calls super and gets D. The scenario I'm concerned with is that A knows how to mimic B's constructor and B knows how to mimic D's, but A doesn't know about D. So D asks A if it knows how to mimic D's constructor, and it says no. Via super, B gets a shot, and it does know, so it translates the arguments to D's constructor into arguments to B's constructor, and again asks A if it knows how to handle them. Then A says yes, translates the args, and constructs an A. If C ever gets consulted, it responds "I don't know a thing" and calls super. > But in the case of make_me, it's someone like C who is calling make_me. If it gets a method in B, then that's a straight-up bug. make_me needs to be reimplemented in A as well, and A would never delegate up since other classes in the mro chain (like B) might not know about C. This scheme (as I've written it) depends strongly on all the classes in the MRO having __make_me__ methods with this very precisely defined structure: test base against yourself, then any superclasses you care to mimic, then call super. Any antisocial superclass ruins everyone's party. > Best, > Neil > > > On Fri, Feb 13, 2015 at 7:00 PM, Isaac Schwabacher ischwabacher at wisc.edu> wrote: > > > On 15-02-13, Neil Girdhar wrote: > > > I personally don't think this is a big enough issue to warrant any changes, but I think Serhiy's solution would be the ideal best with one additional parameter: the caller's type. Something like > > > > > > def __make_me__(self, cls, *args, **kwargs) > > > > > > > > > and the idea is that any time you want to construct a type, instead of > > > > > > > > > self.__class__(assumed arguments?) > > > > > > > > > where you are not sure that the derived class' constructor knows the right argument types, you do > > > > > > > > > def SomeCls: > > > def some_method(self, ...): > > > return self.__make_me__(SomeCls, assumed arguments?) > > > > > > > > > Now the derived class knows who is asking for a copy. In the case of defaultdict, for example, he can implement __make_me__ as follows: > > > > > > > > > def __make_me__(self, cls, *args, **kwargs): > > > if cls is dict: return default_dict(self.default_factory, *args, **kwargs) > > > return default_dict(*args, **kwargs) > > > > > > > > > essentially the caller is identifying himself so that the receiver knows how to interpret the arguments. > > > > > > > > > Best, > > > > > > > > > Neil > > > > Such a method necessarily involves explicit switching on classes... ew. > > Also, to make this work, a class needs to have a relationship with its superclass's superclasses. So in order for DefaultDict's subclasses not to need to know about dict, it would need to look like this: > > > > class DefaultDict(dict): > > .... at classmethod # instance method doesn't make sense here > > ....def __make_me__(cls, base, *args, **kwargs): # make something like base(*args, **kwargs) > > ........# when we get here, nothing in cls.__mro__ above DefaultDict knows how to construct an equivalent to base(*args, **kwargs) using its own constructor > > ........if base is DefaultDict: > > ............return DefaultDict(*args, **kwargs) # if DefaultDict is the best we can do, do it > > ........elif base is dict: > > ............return cls.__make_me__(DefaultDict, None, *args, **kwargs) # subclasses that know about DefaultDict but not dict will intercept this > > ........else: > > ............super(DefaultDict, cls).__make_me__(base, *args, **kwargs) # we don't know how to make an equivalent to base.__new__(*args, **kwargs), so keep looking > > > > I don't even think this is guaranteed to construct an object of class cls corresponding to a base(*args, **kwargs) even if it were possible, since multiple inheritance can screw things up. You might need to have an explicit list of "these are the superclasses whose constructors I can imitate", and have the interpreter find an optimal path for you. > > > > > On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky (java_script:main.compose()> wrote: > > > > > > > > > > > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar (java_script:main.compose()> wrote: > > > > > > > > > Interesting:?> > Not every language allows you to call self.__class__(). In the languages that don't you can get away with incompatible constructor signatures. > > > > > > > > > > > > However, let me try to focus the discussion on a specific issue before we go deep into OOP theory. > > > > > > > > > > > > With python's standard datetime.date we have: > > > > > > > > > > > > >>> from datetime import * > > > > >>> class Date(date): > > > > ... pass > > > > ... > > > > >>> Date.today() > > > > Date(2015, 2, 13) > > > > >>> Date.fromordinal(1) > > > > Date(1, 1, 1) > > > > > > > > > > > > Both .today() and .fromordinal(1) will break in a subclass that redefines __new__ as follows: > > > > > > > > > > > > >>> class Date2(date): > > > > ... def __new__(cls, ymd): > > > > ... return date.__new__(cls, *ymd) > > > > ... > > > > >>> Date2.today() > > > > Traceback (most recent call last): > > > > File "", line 1, in > > > > TypeError: __new__() takes 2 positional arguments but 4 were given > > > > >>> Date2.fromordinal(1) > > > > Traceback (most recent call last): > > > > File "", line 1, in > > > > TypeError: __new__() takes 2 positional arguments but 4 were given > > > > > > > > > > > > > > > > > > > > Why is this acceptable, but we have to sacrifice the convenience of having Date + timedelta > > > > return Date to make it work with Date2: > > > > > > > > > > > > >>> Date2((1,1,1)) + timedelta(1) > > > > datetime.date(1, 1, 2) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From mistersheik at gmail.com Sat Feb 14 01:57:13 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 13 Feb 2015 19:57:13 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <75c0f102b2026.54de488f@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <76a0a5e6b41d2.54de84bf@wiscmail.wisc.edu> <7460bc10b4e87.54de84fc@wiscmail.wisc.edu> <7740a7e8b66fe.54de85ec@wiscmail.wisc.edu> <76a09ea0b7783.54de86dd@wiscmail.wisc.edu> <76a0f0f5b15a4.54de8719@wiscmail.wisc.edu> <7610e537b528b.54de8755@wiscmail.wisc.edu> <7460f3ddb7500.54de87ce@wiscmail.wisc.edu> <7460ee8eb47e6.54de88fb@wiscmail.wisc.edu> <7590ea33b4e12.54de8938@wiscmail.wisc.edu> <7590ce56b3f59.54de8974@wiscmail.wisc.edu> <75e0b16fb2e22.54de89b0@wiscmail.wisc.edu> <75a0e6fdb13fb.54de89ed@wiscmail.wisc.edu> <76a0b5beb787c.54de8a68@wiscmail.wisc.edu> <7590a705b0317.54de8aa7@wiscmail.wisc.edu> <75c08965b4498.54de8ae3@wiscmail.wisc.edu> <75e080b6b454a.54de8b22@wiscmail.wisc.edu> <7740cad7b4553.54de8b5e@wiscmail.wisc.edu> <75a0d194b27a6.54de8b9b@wiscmail.wisc.edu> <75e0d877b71fb.54de8c13@wiscmail.wisc.edu> <7690c481b75f3.54de8c52@wiscmail.wisc.edu> <7460b72bb0e8d.54de8c8e@wiscmail.wisc.edu> <75e0efafb72e2.54de8ccb@wiscmail.wisc.edu> <75a0847db1b11.54de8d07@wiscmail.wisc.edu> <76a0d81ab1672.54de8d44@wiscmail.wisc.edu> <75909961b7d81.54de8d80@wiscmail.wisc.edu> <75908c67b7a2a.54de8dbc@wiscmail.wisc.edu> <7590d2abb120c.54de8df9@wiscmail.wisc.edu> <7590b578b2c45.54de8e35@wiscmail.wisc.edu> <7690d252b6239.54de8e72@wiscmail.wisc.edu> <769098cbb63a2.54de8eae@wiscmail.wisc.edu> <762093bbb5f17.54de8eea@wiscmail.wisc.edu> <7610a1c2b5bea.54de8f27@wiscmail.wisc.edu> <75c0a487b271d.54de8f63@wiscmail.wisc.edu> <75a0da55b1347.54de8fa0@wiscmail.wisc.edu> <75c0d012b000e.54de8fdc@wiscmail.wisc.edu> <75c08852b7610.54de3ba2@wiscmail.wisc.edu> <75c0b241b693e.54de99dc@wiscmail.wisc.edu> <76a0df4ab5cf3.54de9b81@wiscmail.wisc.edu> <75c0edf2b6640.54de9bbd@wiscmail.wisc.edu> <7770d5a5b283d.54de9bfa@wiscmail.wisc.edu> <7620bdd7b005f.54de9c36@wiscmail.wisc.edu> <7790b166b55af.54de9c72@wiscmail.wisc.edu> <760084bab38cf.54de9caf@wiscmail.wisc.edu> <75c0db14b60ef.54de9ceb@wiscmail.wisc.edu> <75c0f102b2026.54de488f@wiscmail.wisc.edu> Message-ID: You're right. On Fri, Feb 13, 2015 at 7:55 PM, Isaac Schwabacher wrote: > On 15-02-13, Neil Girdhar wrote: > > Unlike a regular method, you would never need to call super since you > should know everyone that could be calling you. Typically, when you call > super, you have something like this: > > > > A < B, C > > > > > > B < D > > > > > > so you end up with > > > > > > mro: A, B, C, D > > > > > > And then when A calls super and B calls super it gets C which it > doesn't know about. > > But C calls super and gets D. The scenario I'm concerned with is that A > knows how to mimic B's constructor and B knows how to mimic D's, but A > doesn't know about D. So D asks A if it knows how to mimic D's constructor, > and it says no. Via super, B gets a shot, and it does know, so it > translates the arguments to D's constructor into arguments to B's > constructor, and again asks A if it knows how to handle them. Then A says > yes, translates the args, and constructs an A. If C ever gets consulted, it > responds "I don't know a thing" and calls super. > > > But in the case of make_me, it's someone like C who is calling > make_me. If it gets a method in B, then that's a straight-up bug. > make_me needs to be reimplemented in A as well, and A would never delegate > up since other classes in the mro chain (like B) might not know about C. > > This scheme (as I've written it) depends strongly on all the classes in > the MRO having __make_me__ methods with this very precisely defined > structure: test base against yourself, then any superclasses you care to > mimic, then call super. Any antisocial superclass ruins everyone's party. > > > Best, > > Neil > > > > > > On Fri, Feb 13, 2015 at 7:00 PM, Isaac Schwabacher < > alexander.belopolsky at gmail.com > ischwabacher at wisc.edu> wrote: > > > > > On 15-02-13, Neil Girdhar wrote: > > > > I personally don't think this is a big enough issue to warrant > any changes, but I think Serhiy's solution would be the ideal best with > one additional parameter: the caller's type. Something like > > > > > > > > def __make_me__(self, cls, *args, **kwargs) > > > > > > > > > > > > and the idea is that any time you want to construct a type, instead > of > > > > > > > > > > > > self.__class__(assumed arguments?) > > > > > > > > > > > > where you are not sure that the derived class' constructor knows > the right argument types, you do > > > > > > > > > > > > def SomeCls: > > > > def some_method(self, ...): > > > > return self.__make_me__(SomeCls, assumed arguments?) > > > > > > > > > > > > Now the derived class knows who is asking for a copy. In the case of > defaultdict, for example, he can implement __make_me__ as follows: > > > > > > > > > > > > def __make_me__(self, cls, *args, **kwargs): > > > > if cls is dict: return default_dict(self.default_factory, *args, > **kwargs) > > > > return default_dict(*args, **kwargs) > > > > > > > > > > > > essentially the caller is identifying himself so that the receiver > knows how to interpret the arguments. > > > > > > > > > > > > Best, > > > > > > > > > > > > Neil > > > > > > Such a method necessarily involves explicit switching on classes... ew. > > > Also, to make this work, a class needs to have a relationship with its > superclass's superclasses. So in order for DefaultDict's subclasses > not to need to know about dict, it would need to look like this: > > > > > > class DefaultDict(dict): > > > .... at classmethod # instance method doesn't make sense here > > > ....def __make_me__(cls, base, *args, **kwargs): # make something like > base(*args, **kwargs) > > > ........# when we get here, nothing in cls.__mro__ above DefaultDict > knows how to construct an equivalent to base(*args, **kwargs) using its own > constructor > > > ........if base is DefaultDict: > > > ............return DefaultDict(*args, **kwargs) # if DefaultDict is > the best we can do, do it > > > ........elif base is dict: > > > ............return cls.__make_me__(DefaultDict, None, *args, **kwargs) > # subclasses that know about DefaultDict but not dict will intercept this > > > ........else: > > > ............super(DefaultDict, cls).__make_me__(base, *args, **kwargs) > # we don't know how to make an equivalent to base.__new__(*args, > **kwargs), so keep looking > > > > > > I don't even think this is guaranteed to construct an object of > class cls corresponding to a base(*args, **kwargs) even if it were > possible, since multiple inheritance can screw things up. You might need to > have an explicit list of "these are the superclasses whose constructors I > can imitate", and have the interpreter find an optimal path for you. > > > > > > > On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky < > http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle(javascript:main.compose('new', > 't=alexander.belopolsky at gmail.com>(java_script:main.compose()> wrote: > > > > > > > > > > > > > > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar < > mistersheik at gmail.com (java_script:main.compose()> > wrote: > > > > > > > > > > > Interesting: > > Not every language allows you to call > self.__class__(). In the languages that don't you can get away with > incompatible constructor signatures. > > > > > > > > > > > > > > > However, let me try to focus the discussion on a specific issue > before we go deep into OOP theory. > > > > > > > > > > > > > > > With python's standard datetime.date we have: > > > > > > > > > > > > > > > >>> from datetime import * > > > > > >>> class Date(date): > > > > > ... pass > > > > > ... > > > > > >>> Date.today() > > > > > Date(2015, 2, 13) > > > > > >>> Date.fromordinal(1) > > > > > Date(1, 1, 1) > > > > > > > > > > > > > > > Both .today() and .fromordinal(1) will break in a subclass that > redefines __new__ as follows: > > > > > > > > > > > > > > > >>> class Date2(date): > > > > > ... def __new__(cls, ymd): > > > > > ... return date.__new__(cls, *ymd) > > > > > ... > > > > > >>> Date2.today() > > > > > Traceback (most recent call last): > > > > > File "", line 1, in > > > > > TypeError: __new__() takes 2 positional arguments but 4 were given > > > > > >>> Date2.fromordinal(1) > > > > > Traceback (most recent call last): > > > > > File "", line 1, in > > > > > TypeError: __new__() takes 2 positional arguments but 4 were given > > > > > > > > > > > > > > > > > > > > > > > > > Why is this acceptable, but we have to sacrifice the convenience > of having Date + timedelta > > > > > return Date to make it work with Date2: > > > > > > > > > > > > > > > >>> Date2((1,1,1)) + timedelta(1) > > > > > datetime.date(1, 1, 2) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Feb 14 02:12:42 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Feb 2015 17:12:42 -0800 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> Message-ID: <54DEA10A.6010403@stoneleaf.us> On 02/13/2015 02:31 PM, Serhiy Storchaka wrote: > On 13.02.15 05:41, Ethan Furman wrote: >> So there are basically two choices: >> >> 1) always use the type of the most-base class when creating new instances >> >> pros: >> - easy >> - speedy code >> - no possible tracebacks on new object instantiation >> >> cons: >> - a subclass that needs/wants to maintain itself must override all >> methods that create new instances, even if the only change is to >> the type of object returned >> >> 2) always use the type of self when creating new instances >> >> pros: >> - subclasses automatically maintain type >> - much less code in the simple cases [1] >> >> cons: >> - if constructor signatures change, must override all methods which >> create new objects > > And switching to (2) would break existing code which uses subclasses with constructors with different signature (e.g. > defaultdict). I don't think defaultdict is a good example -- I don't see any methods on it that return a new dict, default or otherwise. So if this change happened, defaultdict would have to have its own __add__ and not rely on dict's __add__. > The third choice is to use different specially designed constructor. > > class A(int): > > --> class A(int): > ... def __add__(self, other): > ... return self.__make_me__(int(self) + int(other)) > > ... def __repr__(self): > ... return 'A(%d)' % self How would this help in the case of defaultdict? __make_me__ is a class method, but it needs instance info to properly create a new dict with the same default factory. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From mistersheik at gmail.com Sat Feb 14 02:16:08 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 13 Feb 2015 20:16:08 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DEA10A.6010403@stoneleaf.us> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DEA10A.6010403@stoneleaf.us> Message-ID: I think it works as Isaac explained if __make_me__ is an instance method that also accepts the calling class type. On Fri, Feb 13, 2015 at 8:12 PM, Ethan Furman wrote: > On 02/13/2015 02:31 PM, Serhiy Storchaka wrote: > > On 13.02.15 05:41, Ethan Furman wrote: > >> So there are basically two choices: > >> > >> 1) always use the type of the most-base class when creating new > instances > >> > >> pros: > >> - easy > >> - speedy code > >> - no possible tracebacks on new object instantiation > >> > >> cons: > >> - a subclass that needs/wants to maintain itself must override all > >> methods that create new instances, even if the only change is to > >> the type of object returned > >> > >> 2) always use the type of self when creating new instances > >> > >> pros: > >> - subclasses automatically maintain type > >> - much less code in the simple cases [1] > >> > >> cons: > >> - if constructor signatures change, must override all methods > which > >> create new objects > > > > And switching to (2) would break existing code which uses subclasses > with constructors with different signature (e.g. > > defaultdict). > > I don't think defaultdict is a good example -- I don't see any methods on > it that return a new dict, default or > otherwise. So if this change happened, defaultdict would have to have its > own __add__ and not rely on dict's __add__. > > > > The third choice is to use different specially designed constructor. > > > > class A(int): > > > > --> class A(int): > > ... def __add__(self, other): > > ... return self.__make_me__(int(self) + int(other)) > > > > ... def __repr__(self): > > ... return 'A(%d)' % self > > How would this help in the case of defaultdict? __make_me__ is a class > method, but it needs instance info to properly > create a new dict with the same default factory. > > -- > ~Ethan~ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ischwabacher at wisc.edu Sat Feb 14 01:00:02 2015 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Fri, 13 Feb 2015 18:00:02 -0600 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <75c0d012b000e.54de8fdc@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <76a0a5e6b41d2.54de84bf@wiscmail.wisc.edu> <7460bc10b4e87.54de84fc@wiscmail.wisc.edu> <7740a7e8b66fe.54de85ec@wiscmail.wisc.edu> <76a09ea0b7783.54de86dd@wiscmail.wisc.edu> <76a0f0f5b15a4.54de8719@wiscmail.wisc.edu> <7610e537b528b.54de8755@wiscmail.wisc.edu> <7460f3ddb7500.54de87ce@wiscmail.wisc.edu> <7460ee8eb47e6.54de88fb@wiscmail.wisc.edu> <7590ea33b4e12.54de8938@wiscmail.wisc.edu> <7590ce56b3f59.54de8974@wiscmail.wisc.edu> <75e0b16fb2e22.54de89b0@wiscmail.wisc.edu> <75a0e6fdb13fb.54de89ed@wiscmail.wisc.edu> <76a0b5beb787c.54de8a68@wiscmail.wisc.edu> <7590a705b0317.54de8aa7@wiscmail.wisc.edu> <75c08965b4498.54de8ae3@wiscmail.wisc.edu> <75e080b6b454a.54de8b22@wiscmail.wisc.edu> <7740cad7b4553.54de8b5e@wiscmail.wisc.edu> <75a0d194b27a6.54de8b9b@wiscmail.wisc.edu> <75e0d877b71fb.54de8c13@wiscmail.wisc.edu> <7690c481b75f3.54de8c52@wiscmail.wisc.edu> <7460b72bb0e8d.54de8c8e@wiscmail.wisc.edu> <75e0efafb72e2.54de8ccb@wiscmail.wisc.edu> <75a0847db1b11.54de8d07@wiscmail.wisc.edu> <76a0d81ab1672.54de8d44@wiscmail.wisc.edu> <75909961b7d81.54de8d80@wiscmail.wisc.edu> <75908c67b7a2a.54de8dbc@wiscmail.wisc.edu> <7590d2abb120c.54de8df9@wiscmail.wisc.edu> <7590b578b2c45.54de8e35@wiscmail.wisc.edu> <7690d252b6239.54de8e72@wiscmail.wisc.edu> <769098cbb63a2.54de8eae@wiscmail.wisc.edu> <762093bbb5f17.54de8eea@wiscmail.wisc.edu> <7610a1c2b5bea.54de8f27@wiscmail.wisc.edu> <75c0a487b271d.54de8f63@wiscmail.wisc.edu> <75a0da55b1347.54de8fa0@wiscmail.wisc.edu> <75c0d012b000e.54de8fdc@wiscmail.wisc.edu> Message-ID: <75c08852b7610.54de3ba2@wiscmail.wisc.edu> On 15-02-13, Neil Girdhar wrote: > I personally don't think this is a big enough issue to warrant any changes, but I think Serhiy's solution would be the ideal best with one additional parameter: the caller's type. Something like > > def __make_me__(self, cls, *args, **kwargs) > > > and the idea is that any time you want to construct a type, instead of > > > self.__class__(assumed arguments?) > > > where you are not sure that the derived class' constructor knows the right argument types, you do > > > def SomeCls: > def some_method(self, ...): > return self.__make_me__(SomeCls, assumed arguments?) > > > Now the derived class knows who is asking for a copy. In the case of defaultdict, for example, he can implement __make_me__ as follows: > > > def __make_me__(self, cls, *args, **kwargs): > if cls is dict: return default_dict(self.default_factory, *args, **kwargs) > return default_dict(*args, **kwargs) > > > essentially the caller is identifying himself so that the receiver knows how to interpret the arguments. > > > Best, > > > Neil Such a method necessarily involves explicit switching on classes... ew. Also, to make this work, a class needs to have a relationship with its superclass's superclasses. So in order for DefaultDict's subclasses not to need to know about dict, it would need to look like this: class DefaultDict(dict): .... at classmethod # instance method doesn't make sense here ....def __make_me__(cls, base, *args, **kwargs): # make something like base(*args, **kwargs) ........# when we get here, nothing in cls.__mro__ above DefaultDict knows how to construct an equivalent to base(*args, **kwargs) using its own constructor ........if base is DefaultDict: ............return DefaultDict(*args, **kwargs) # if DefaultDict is the best we can do, do it ........elif base is dict: ............return cls.__make_me__(DefaultDict, None, *args, **kwargs) # subclasses that know about DefaultDict but not dict will intercept this ........else: ............super(DefaultDict, cls).__make_me__(base, *args, **kwargs) # we don't know how to make an equivalent to base.__new__(*args, **kwargs), so keep looking I don't even think this is guaranteed to construct an object of class cls corresponding to a base(*args, **kwargs) even if it were possible, since multiple inheritance can screw things up. You might need to have an explicit list of "these are the superclasses whose constructors I can imitate", and have the interpreter find an optimal path for you. > On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky wrote: > > > > > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar wrote: > > > > > Interesting:?http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle > > > > > > > > > Let me humbly conjecture that the people who wrote the top answers have background in less capable languages than Python. > > > > > > Not every language allows you to call self.__class__(). In the languages that don't you can get away with incompatible constructor signatures. > > > > > > However, let me try to focus the discussion on a specific issue before we go deep into OOP theory. > > > > > > With python's standard datetime.date we have: > > > > > > >>> from datetime import * > > >>> class Date(date): > > ... pass > > ... > > >>> Date.today() > > Date(2015, 2, 13) > > >>> Date.fromordinal(1) > > Date(1, 1, 1) > > > > > > Both .today() and .fromordinal(1) will break in a subclass that redefines __new__ as follows: > > > > > > >>> class Date2(date): > > ... def __new__(cls, ymd): > > ... return date.__new__(cls, *ymd) > > ... > > >>> Date2.today() > > Traceback (most recent call last): > > File "", line 1, in > > TypeError: __new__() takes 2 positional arguments but 4 were given > > >>> Date2.fromordinal(1) > > Traceback (most recent call last): > > File "", line 1, in > > TypeError: __new__() takes 2 positional arguments but 4 were given > > > > > > > > > > Why is this acceptable, but we have to sacrifice the convenience of having Date + timedelta > > return Date to make it work with Date2: > > > > > > >>> Date2((1,1,1)) + timedelta(1) > > datetime.date(1, 1, 2) > > > > > > > > > > > > > > > > > > > > From ischwabacher at wisc.edu Sat Feb 14 02:37:24 2015 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Fri, 13 Feb 2015 19:37:24 -0600 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <7710cf6974706.54dea6b8@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DEA10A.6010403@stoneleaf.us> <772096a276229.54dea512@wiscmail.wisc.edu> <7710a1e67544e.54dea54e@wiscmail.wisc.edu> <76f0ce1170c6d.54dea58a@wiscmail.wisc.edu> <7560ebaf71d98.54dea5c7@wiscmail.wisc.edu> <73b0a86271da7.54dea603@wiscmail.wisc.edu> <75e08954727e6.54dea640@wiscmail.wisc.edu> <756081f271a75.54dea67c@wiscmail.wisc.edu> <7710cf6974706.54dea6b8@wiscmail.wisc.edu> Message-ID: <7710b6097060f.54de5274@wiscmail.wisc.edu> Hmm... super *is* a problem, because if we super, we can end up asking superclasses that we then won't know how to mimic. So what we really want is to move the superclasses we can mimic to the front of the MRO. If none of those can indirectly mimic the base class, then we try the other classes in the MRO to see if we can get a partial upgrade. On 15-02-13, Neil Girdhar wrote: > I think it works as Isaac explained if __make_me__ is an instance method that also accepts the calling class type. > > On Fri, Feb 13, 2015 at 8:12 PM, Ethan Furman ethan at stoneleaf.us> wrote: > > > On 02/13/2015 02:31 PM, Serhiy Storchaka wrote: > > > On 13.02.15 05:41, Ethan Furman wrote: > > >> So there are basically two choices: > > >> > > >> 1) always use the type of the most-base class when creating new instances > > >> > > >> pros: > > >> - easy > > >> - speedy code > > >> - no possible tracebacks on new object instantiation > > >> > > >> cons: > > >> - a subclass that needs/wants to maintain itself must override all > > >> methods that create new instances, even if the only change is to > > >> the type of object returned > > >> > > >> 2) always use the type of self when creating new instances > > >> > > >> pros: > > >> - subclasses automatically maintain type > > >> - much less code in the simple cases [1] > > >> > > >> cons: > > >> - if constructor signatures change, must override all methods which > > >> create new objects > > > > > > And switching to (2) would break existing code which uses subclasses with constructors with different signature (e.g. > > > defaultdict). > > > > I don't think defaultdict is a good example -- I don't see any methods on it that return a new dict, default or > > otherwise. So if this change happened, defaultdict would have to have its own __add__ and not rely on dict's __add__. > > > > > > > The third choice is to use different specially designed constructor. > > > > > > class A(int): > > > > > > --> class A(int): > > > ... def __add__(self, other): > > > ... return self.__make_me__(int(self) + int(other)) > > > > > > ... def __repr__(self): > > > ... return 'A(%d)' % self > > > > How would this help in the case of defaultdict? __make_me__ is a class method, but it needs instance info to properly > > create a new dict with the same default factory. > > > > -- > > ~Ethan~ > > > > > > _______________________________________________ > > Python-Dev mailing list > > https://mail.python.org/mailman/listinfo/python-dev(javascript:main.compose('new', 't=Python-Dev at python.org> > > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: make_me.py Type: text/x-python-script Size: 1820 bytes Desc: not available URL: From stephen at xemacs.org Sat Feb 14 03:35:11 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 14 Feb 2015 11:35:11 +0900 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: References: Message-ID: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> Victor Stinner writes: > *** Now the real question: is it useful to know the inode number > (st_ino) without the device number (st_dev)? *** > > On POSIX, you can still get the st_dev from DirEntry.stat(), but it > always require a system call. So you loose the whole purpose of > DirEntry (no extra syscall). True, but I suppose in many cases the user will know that all file system objects handled are on the same device, or will be willing to risk an occasional anomoly. IMO: Document the limitation (if no extra syscall) or inefficiency (with the syscall), and let the user choose. The remaining issue is whether to provide a convenience function for the device number, with appropriately loud warnings about how inefficient it is, or to deter the user with the need to call .stat() and extract the device number. From cs at zip.com.au Sat Feb 14 04:15:44 2015 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 14 Feb 2015 14:15:44 +1100 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20150214031544.GA6319@cskk.homeip.net> On 14Feb2015 11:35, Stephen J. Turnbull wrote: >Victor Stinner writes: > > *** Now the real question: is it useful to know the inode number > > (st_ino) without the device number (st_dev)? *** > > > > On POSIX, you can still get the st_dev from DirEntry.stat(), but it > > always require a system call. So you loose the whole purpose of > > DirEntry (no extra syscall). > >True, but I suppose in many cases the user will know that all file >system objects handled are on the same device, or will be willing to >risk an occasional anomoly. In POSIX, all filsystem objects named by a directory are on the same device unless one is a mount point. (And in that case, d_ino from stat won't match d_ino from scandir; I expect.) >IMO: Document the limitation (if no extra syscall) or inefficiency >(with the syscall), and let the user choose. +1 on .inode(): d_ino has been available in the directory data on POSIX since at least V7 UNIX (1970s), almost certainly earlier. Agree the limitation should be mentioned. >The remaining issue is whether to provide a convenience function for >the device number, with appropriately loud warnings about how >inefficient it is, or to deter the user with the need to call .stat() >and extract the device number. -1 on that. People will use it! Given the doco above, it should be obvious under what circumstances one might choose to call stat, and making that stat overt means it is less likely to be called unwisely. Since scandir is all about efficiency, providing a very costly convenience function seems to go against the grain. Regarding usefulness: Victor, you've got the typical use case in another post (i.e. useful as in "advantageous"), and your own tests show that st_dev of the dir matches st_dev of a dir's entries in all normal/regular filesystems (i.e. useful as in "meaningful/consistent"). Special filesystems like /dev may be weird, but people relying on this should be aware of the constraint anyway. Since a directory at the low level is essentially a mapping of names to inodes within the directory's filesystem, this is to be expected. Cheers, Cameron Simpson Uh, this is only temporary...unless it works. - Red Green From storchaka at gmail.com Sat Feb 14 07:01:40 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 14 Feb 2015 08:01:40 +0200 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <54DEA10A.6010403@stoneleaf.us> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DEA10A.6010403@stoneleaf.us> Message-ID: On 14.02.15 03:12, Ethan Furman wrote: >> The third choice is to use different specially designed constructor. >> >> class A(int): >> >> --> class A(int): >> ... def __add__(self, other): >> ... return self.__make_me__(int(self) + int(other)) >> >> ... def __repr__(self): >> ... return 'A(%d)' % self > > How would this help in the case of defaultdict? __make_me__ is a class method, but it needs instance info to properly > create a new dict with the same default factory. In case of defaultdict (when dict would have to have __add__ and like) either __make_me__ == dict (then defaultdict's methods will return dicts) or it will be instance method. def __make_me__(self, other): return defaultdict(self.default_factory, other) From storchaka at gmail.com Sat Feb 14 07:22:24 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 14 Feb 2015 08:22:24 +0200 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> Message-ID: On 14.02.15 01:03, Neil Girdhar wrote: > Now the derived class knows who is asking for a copy. In the case of > defaultdict, for example, he can implement __make_me__ as follows: > > def __make_me__(self, cls, *args, **kwargs): > if cls is dict: return default_dict(self.default_factory, *args, > **kwargs) > return default_dict(*args, **kwargs) > > essentially the caller is identifying himself so that the receiver knows > how to interpret the arguments. No, my idea was that __make_me__ has the same signature in all subclasses. It takes exactly one argument and creates an instance of concrete class, so it never fails. If you want to create an instance of different class in the derived class, you should explicitly override __make_me__. From ncoghlan at gmail.com Sat Feb 14 10:04:57 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Feb 2015 19:04:57 +1000 Subject: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python) In-Reply-To: References: Message-ID: On 14 Feb 2015 03:43, "Nathaniel Smith" wrote: > > On 13 Feb 2015 02:09, "Victor Stinner" wrote: > > > > A alternative is to add a new _scandir.c module to host the new C > > code, and share some code with posixmodule.c: remove "static" keyword > > from required C functions (functions to convert Windows attributes to > > a os.stat_result object). > > Hopefully not too annoying question from an outsider: has cpython's build system added the necessary bits to do this on a safe, portable, non-symbol-namespace polluting way? E.g. using -fvisibility=hidden on Linux? We just add a "_Py" prefix on the things that we're making available to the linker solely for own use and don't provide any backwards compatibility guarantees for other people that decide to use them directly despite the leading underscore and lack of documentation. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 14 10:08:42 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Feb 2015 19:08:42 +1000 Subject: [Python-Dev] PEP 486: Make the Python Launcher aware of virtual environments In-Reply-To: References: Message-ID: On 14 Feb 2015 07:31, "Paul Moore" wrote: > > (By the way, on a procedural note, how do I update a PEP? Do I just > send an updated version to peps at python.org, or is there a better way?) If you're happy to handle the PEP editor responsibilities described in PEP 1 yourself, you can also ask for commit access to the PEPs repo to update it directly. Cheers, Nick. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 14 10:13:42 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Feb 2015 19:13:42 +1000 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: <20150214031544.GA6319@cskk.homeip.net> References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> <20150214031544.GA6319@cskk.homeip.net> Message-ID: On 14 Feb 2015 13:17, "Cameron Simpson" wrote: > > -1 on that. People will use it! Given the doco above, it should be obvious under what circumstances one might choose to call stat, and making that stat overt means it is less likely to be called unwisely. > > Since scandir is all about efficiency, providing a very costly convenience function seems to go against the grain. > > Regarding usefulness: Victor, you've got the typical use case in another post (i.e. useful as in "advantageous"), and your own tests show that st_dev of the dir matches st_dev of a dir's entries in all normal/regular filesystems (i.e. useful as in "meaningful/consistent"). Special filesystems like /dev may be weird, but people relying on this should be aware of the constraint anyway. Since a directory at the low level is essentially a mapping of names to inodes within the directory's filesystem, this is to be expected. +1 from me for Cameron's perspective & rationale - it's useful for detecting hardlinks, it will usually work, and the cases where it isn't sufficient on its own are filesystem handling edge cases in more ways than one. Cheers, Nick. > > Cheers, > Cameron Simpson > > Uh, this is only temporary...unless it works. - Red Green > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 14 10:37:30 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Feb 2015 19:37:30 +1000 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> Message-ID: On 14 Feb 2015 07:39, "Isaac Schwabacher" wrote: > > On 15-02-13, Guido van Rossum wrote: > > Are you willing to wait 10 days for an answer? I'm out of round tuits for a while. > > IIUC, the argument is that the Liskov Substitution Principle is a statement about how objects of a subtype behave relative to objects of a supertype, and it doesn't apply to constructors because they aren't behaviors of existing objects. So other overriding methods *should* be able to handle the same inputs that the respective overridden methods do, but constructors don't need to. Even though __init__ is written as an instance method, it seems like it's "morally" a part of the class method __new__ that's only split off for convenience. A potentially helpful example is to consider a type system where Square is a subclass of Rectangle. To specify a Rectangle takes a height and a width, but a Square only needs the length of one side, and letting the height and width be specified independently would be outright wrong. Many possible operations on a Square also *should* formally return a Rectangle, as doing something like doubling the height gives you a result that isn't a square any more. (That's also why shapes must be immutable for this particular type hierarchy to make any sense) (You can construct similar examples for Circle & Ellipse, and Python's numeric hierarchy is a real-world example of some of the complexity that can arise) There's simply no sensible default behaviour other than the status quo in the face of scenarios like that - whether or not there's a better answer than "return an instance of the parent class" for any given inherited method depends entirely on the invariants of the subclass and how they differ from those of the parent class. It's certainly possible to write methods that return a new instance of the current type (that's common in alternative constructors, for example), but it usually involves placing additional expectations on the developers of subclasses. That's most commonly seen within the confines of a single project, or when defining a development framework, rather than in libraries that choose to expose some public types and also supports their use as base types. Cheers, Nick. > > If this message is unclear, it's because I don't really understand this myself and I'm trying to articulate my best understanding of what's been said on this thread and those it links to. > > ijs > > > On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com(javascript:main.compose()> wrote: > > > > > > > > On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com(javascript:main.compose()> wrote: > > > >> > > > >> FWIW you're wrong when you claim that "a constructor is no different from any other method". Someone else should probably explain this (it's an old argument that's been thoroughly settled). > > > > > > > > > > > > Well, the best answer I've got in the past [1] was "ask on python-dev since Guido called the operator overriding expectation." :-) > > > > > > > > > And let me repost this bit of history [1]: > > > > > > Here is the annotated pre-r82065 code: > > > > > > 39876 gvanrossum def __add__(self, other): > > > 39876 gvanrossum if isinstance(other, timedelta): > > > 39928 gvanrossum return self.__class__(self.__days + other.__days, > > > 39876 gvanrossum self.__seconds + other.__seconds, > > > 39876 gvanrossum self.__microseconds + other.__microseconds) > > > 40207 tim_one return NotImplemented > > > 39876 gvanrossum > > > > > > > > > > > > [1] http://bugs.python.org/issue2267#msg125979 > > > > > > > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido(http://python.org/~guido)) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 14 10:44:19 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Feb 2015 19:44:19 +1000 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <54DD4723.7030003@stoneleaf.us> <54DD7262.4040000@stoneleaf.us> <54DD8479.8050600@stoneleaf.us> <76a0e9fdb742a.54de5b6e@wiscmail.wisc.edu> <76909504b6745.54de5be8@wiscmail.wisc.edu> <75e0add3b0c12.54de5c25@wiscmail.wisc.edu> <75e0b259b62e6.54de5c61@wiscmail.wisc.edu> <76b0ff0eb41b0.54de5e06@wiscmail.wisc.edu> <7600aceeb2d2f.54de5e43@wiscmail.wisc.edu> <76a0ed19b3746.54de5e7f@wiscmail.wisc.edu> <7610bc0ab00bb.54de5ebb@wiscmail.wisc.edu> <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> Message-ID: On 14 Feb 2015 08:57, "Alexander Belopolsky" wrote: > > > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar wrote: >> >> Interesting: http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle > > > Let me humbly conjecture that the people who wrote the top answers have background in less capable languages than Python. > > Not every language allows you to call self.__class__(). In the languages that don't you can get away with incompatible constructor signatures. > > However, let me try to focus the discussion on a specific issue before we go deep into OOP theory. > > With python's standard datetime.date we have: > > >>> from datetime import * > >>> class Date(date): > ... pass > ... > >>> Date.today() > Date(2015, 2, 13) > >>> Date.fromordinal(1) > Date(1, 1, 1) > > Both .today() and .fromordinal(1) will break in a subclass that redefines __new__ as follows: > > >>> class Date2(date): > ... def __new__(cls, ymd): > ... return date.__new__(cls, *ymd) > ... > >>> Date2.today() > Traceback (most recent call last): > File "", line 1, in > TypeError: __new__() takes 2 positional arguments but 4 were given > >>> Date2.fromordinal(1) > Traceback (most recent call last): > File "", line 1, in > TypeError: __new__() takes 2 positional arguments but 4 were given > > Why is this acceptable, but we have to sacrifice the convenience of having Date + timedelta > return Date to make it work with Date2: > > >>> Date2((1,1,1)) + timedelta(1) > datetime.date(1, 1, 2) Coupling alternative constructors to the default constructor signature is pretty normal - it just means that if you override the signature of the default constructor, you may need to update the alternative ones accordingly. Cheers, Nick. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Feb 14 11:57:15 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 14 Feb 2015 11:57:15 +0100 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Le samedi 14 f?vrier 2015, Stephen J. Turnbull a ?crit : > > IMO: Document the limitation (if no extra syscall) or inefficiency > (with the syscall), and let the user choose. Hum, by the way, I don't know if we should dd the method on Windows. As I said, I don't want to cache The result of the os.lstat(). Basically, there is no benfit for other methods to call inode(). A method may be a trap for Windows users. I propose something else: a DirEntry.inode read-only property which would be None on Windows. So you see dirrectly that the property is for POSIX, and that calling os.stat() is required on Windows. os.stat() not DirEntry.stat(), DirEntry.stat() doesn't fill st_ino, st_dev and st_nlink are not filled on Windows. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Feb 14 12:17:11 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 14 Feb 2015 12:17:11 +0100 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: 2015-02-14 11:57 GMT+01:00 Victor Stinner : > I propose something else: a DirEntry.inode read-only property (...) Full DirEntry API: - name (str) attribute - path (str) read-only property, created at the first call - inode (int or None) attribute <=== my proposition - is_dir(*, follow_symlinks=True) - is_file(*, follow_symlinks=True) - is_symlink(*, follow_symlinks=True) - stat(*, follow_symlinks=True) is_dir(), is_file(), is_symlink() and stat() are method because they may all require a syscall (os.stat or os.lstat). They all cache their result. In some cases, the result is already known when DirEntry is created. In most cases, a single call to os.stat() is required to fill the result of all methods. Victor From steve at pearwood.info Sat Feb 14 13:23:32 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 Feb 2015 23:23:32 +1100 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> Message-ID: <20150214122332.GV2498@ando.pearwood.info> On Fri, Feb 13, 2015 at 06:03:35PM -0500, Neil Girdhar wrote: > I personally don't think this is a big enough issue to warrant any changes, > but I think Serhiy's solution would be the ideal best with one additional > parameter: the caller's type. Something like > > def __make_me__(self, cls, *args, **kwargs) > > and the idea is that any time you want to construct a type, instead of > > self.__class__(assumed arguments?) > > where you are not sure that the derived class' constructor knows the right > argument types, you do > > def SomeCls: > def some_method(self, ...): > return self.__make_me__(SomeCls, assumed arguments?) > > Now the derived class knows who is asking for a copy. What if you wish to return an instance from a classmethod? You don't have a `self` available. class SomeCls: def __init__(self, x, y, z): ... @classmethod def from_spam(cls, spam): x, y, z = process(spam) return cls.__make_me__(self, cls, x, y, z) # oops, no self Even if you are calling from an instance method, and self is available, you cannot assume that the information needed for the subclass constructor is still available. Perhaps that information is used in the constructor and then discarded. The problem we wish to solve is that when subclassing, methods of some base class blindly return instances of itself, instead of self's type: py> class MyInt(int): ... pass ... py> n = MyInt(23) py> assert isinstance(n, MyInt) py> assert isinstance(n+1, MyInt) Traceback (most recent call last): File "", line 1, in ? AssertionError The means that subclasses often have to override all the parent's methods, just to ensure the type is correct: class MyInt(int): def __add__(self, other): o = super().__add__(other) if o is not NotImplemented: o = type(self)(o) return o Something like that, repeated for all the int methods, should work: py> n = MyInt(23) py> type(n+1) This is tedious and error prone, but at least once it is done, subclasses of MyInt will Just Work: py> class MyOtherInt(MyInt): ... pass ... py> a = MyOtherInt(42) py> type(a + 1000) (At least, *in general* they will work. See below.) So, why not have int's methods use type(self) instead of hard coding int? The answer is that *some* subclasses might override the constructor, which would cause the __add__ method to fail: # this will fail if the constructor has a different signature o = type(self)(o) Okay, but changing the constructor signature is quite unusual. Mostly, people subclass to add new methods or attributes, or to override a specific method. The dict/defaultdict situation is relatively uncommon. Instead of requiring *every* subclass to override all the methods, couldn't we require the base classes (like int) to assume that the signature is unchanged and call type(self), and leave it up to the subclass to override all the methods *only* if the signature has changed? (Which they probably would have to do anyway.) As the MyInt example above shows, or datetime in the standard library, this actually works fine in practice: py> from datetime import datetime py> class MySpecialDateTime(datetime): ... pass ... py> t = MySpecialDateTime.today() py> type(t) Why can't int, str, list, tuple etc. be more like datetime? -- Steve From p.f.moore at gmail.com Sat Feb 14 15:44:14 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 14 Feb 2015 14:44:14 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support Message-ID: I'm looking at putting together a patch for CPython to implement PEP 441. In doing so, there are a few issues that I'd like to raise with the PEP. These are all to do with the supporting app "pyzaa" (IIRC, Nick proposed renaming this to "pyzapp", which I like, but it's not a big deal either way). 1. The PEP says that the application will warn if you try to add C extensions to the archive. In practice, this will be quite hard to do - extensions vary across platforms, and it's not only C extensions that won't work from an archive (consider a DLL with a pure-python ctypes wrapper). We could check the filenames for "known" extensions (maybe using importlib.machinery.EXTENSION_SUFFIXES, but that varies by platform) but I think that's going to be fragile. IMO, the costs aren't worth the benefits, and I'd like to remove this proposal and simply document that applications packed up with pyzaa need to be tested to ensure they work from a zipfile. 2. The option to compile Python source seems to me to be of limited use. It ties the archive to a particular Python version (more precisely, only one version gets the benefits of precompiled bytecode). It's also tricky to implement, particularly the option to compile an existing archive, because there standard library py_compile and compileall modules work on the filesystem, and for compiling files in an archive it would be better to build the bytecode in memory (no need to use tempfiles). I'd prefer to say that if users want to include bytecode, they can precompile on disk (using compileall) for as many versions as they want. FWIW, Daniel's standalone pyzaa program doesn't currently include either of the above features. As well as removing things, I'd like to add a programmable API for the utility. At the moment I only have a rough idea of how that would work, but I'm thinking of a PyZApp class, which can be created from a filename or an open stream (like a ZipFile object) and which has methods to set the shebang line, setting the main entry point, and adding files or directories. Does that need to go into the PEP or is it OK to review the API as part of the patch? I don't mind either way, but my normal coding process tends to be to fiddle with the API as I code it up and use it in the main app, so I'd normally leave documenting the final API till it's written... If the above seems like a reasonable plan, I'll work up a patch with the intention that it gets approved and implemented in time for 3.5. Paul From Steve.Dower at microsoft.com Sat Feb 14 15:53:26 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 14 Feb 2015 14:53:26 +0000 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <20150214122332.GV2498@ando.pearwood.info> References: <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> , <20150214122332.GV2498@ando.pearwood.info> Message-ID: "Instead of requiring *every* subclass to override all the methods, couldn't we require the base classes (like int) to assume that the signature is unchanged and call type(self), and leave it up to the subclass to override all the methods *only* if the signature has changed?" I assumed everyone was just saying this point over and over, so I haven't been following the thread closely. This is precisely how inheritance works: subclasses are constrained by the base class. If you want to play, you *must* play by its rules. (Else, use composition.) It's fine for base classes to assume a compatible constructor, and if builtins can do it without hurting performance for the non-subclassed case, I don't see why not. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Steven D'Aprano Sent: ?2/?14/?2015 4:24 To: python-dev at python.org Subject: Re: [Python-Dev] subclassing builtin data structures On Fri, Feb 13, 2015 at 06:03:35PM -0500, Neil Girdhar wrote: > I personally don't think this is a big enough issue to warrant any changes, > but I think Serhiy's solution would be the ideal best with one additional > parameter: the caller's type. Something like > > def __make_me__(self, cls, *args, **kwargs) > > and the idea is that any time you want to construct a type, instead of > > self.__class__(assumed arguments?) > > where you are not sure that the derived class' constructor knows the right > argument types, you do > > def SomeCls: > def some_method(self, ...): > return self.__make_me__(SomeCls, assumed arguments?) > > Now the derived class knows who is asking for a copy. What if you wish to return an instance from a classmethod? You don't have a `self` available. class SomeCls: def __init__(self, x, y, z): ... @classmethod def from_spam(cls, spam): x, y, z = process(spam) return cls.__make_me__(self, cls, x, y, z) # oops, no self Even if you are calling from an instance method, and self is available, you cannot assume that the information needed for the subclass constructor is still available. Perhaps that information is used in the constructor and then discarded. The problem we wish to solve is that when subclassing, methods of some base class blindly return instances of itself, instead of self's type: py> class MyInt(int): ... pass ... py> n = MyInt(23) py> assert isinstance(n, MyInt) py> assert isinstance(n+1, MyInt) Traceback (most recent call last): File "", line 1, in ? AssertionError The means that subclasses often have to override all the parent's methods, just to ensure the type is correct: class MyInt(int): def __add__(self, other): o = super().__add__(other) if o is not NotImplemented: o = type(self)(o) return o Something like that, repeated for all the int methods, should work: py> n = MyInt(23) py> type(n+1) This is tedious and error prone, but at least once it is done, subclasses of MyInt will Just Work: py> class MyOtherInt(MyInt): ... pass ... py> a = MyOtherInt(42) py> type(a + 1000) (At least, *in general* they will work. See below.) So, why not have int's methods use type(self) instead of hard coding int? The answer is that *some* subclasses might override the constructor, which would cause the __add__ method to fail: # this will fail if the constructor has a different signature o = type(self)(o) Okay, but changing the constructor signature is quite unusual. Mostly, people subclass to add new methods or attributes, or to override a specific method. The dict/defaultdict situation is relatively uncommon. Instead of requiring *every* subclass to override all the methods, couldn't we require the base classes (like int) to assume that the signature is unchanged and call type(self), and leave it up to the subclass to override all the methods *only* if the signature has changed? (Which they probably would have to do anyway.) As the MyInt example above shows, or datetime in the standard library, this actually works fine in practice: py> from datetime import datetime py> class MySpecialDateTime(datetime): ... pass ... py> t = MySpecialDateTime.today() py> type(t) Why can't int, str, list, tuple etc. be more like datetime? -- Steve _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Sat Feb 14 18:47:19 2015 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 14 Feb 2015 17:47:19 +0000 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Sat Feb 14 2015 at 3:17:51 AM Victor Stinner wrote: > 2015-02-14 11:57 GMT+01:00 Victor Stinner : > > I propose something else: a DirEntry.inode read-only property (...) > > Full DirEntry API: > > - name (str) attribute > - path (str) read-only property, created at the first call > - inode (int or None) attribute <=== my proposition > +1 we need to provide the inode (we shouldn't be throwing anything from the underlying directory entry away when possible). But... I think the "or None" semantics are a bad idea. It'd be better for this to raise AttributeError on Windows so that someone can't write the most natural form of code assuming that inode is valid and have it appear to work on Windows when in fact it'd do the wrong thing. > - is_dir(*, follow_symlinks=True) > - is_file(*, follow_symlinks=True) > - is_symlink(*, follow_symlinks=True) > - stat(*, follow_symlinks=True) > > is_dir(), is_file(), is_symlink() and stat() are method because they > may all require a syscall (os.stat or os.lstat). They all cache their > result. In some cases, the result is already known when DirEntry is > created. In most cases, a single call to os.stat() is required to fill > the result of all methods. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Sat Feb 14 19:26:36 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 14 Feb 2015 13:26:36 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <20150214122332.GV2498@ando.pearwood.info> References: <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <20150214122332.GV2498@ando.pearwood.info> Message-ID: On Sat, Feb 14, 2015 at 7:23 AM, Steven D'Aprano wrote: > Why can't int, str, list, tuple etc. be more like datetime? They are. In all these types, class methods call subclass constructors but instance methods don't. >>> class Int(int): ... pass ... >>> Int.from_bytes(bytes([1,2,3]), 'big') 66051 >>> type(_) >>> Int(1) + 1 2 >>> type(_) In the case of int, there is a good reason for this behavior - bool. In python, we want True + True == 2. In numpy, where binary operations preserve subclasses, you have >>> import numpy >>> numpy.bool_(1) + numpy.bool_(1) True I don't see a similar argument for the date class, however. Given date.{to|from}ordinal(), date subclasses are pretty much bound to have timedelta addition satisfy (d + td).toordinal() == d.toordinal() + td.days. Any other definition would be fighting the baseclass design and would be better implemented via containment. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Sat Feb 14 20:36:26 2015 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 14 Feb 2015 20:36:26 +0100 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <20150214122332.GV2498@ando.pearwood.info> Message-ID: On 02/14/2015 07:26 PM, Alexander Belopolsky wrote: > In the case of int, there is a good reason for this behavior - bool. In python, > we want True + True == 2. In numpy, where binary operations preserve > subclasses, you have > >>>> import numpy >>>> numpy.bool_(1) + numpy.bool_(1) > True I don't think numpy.bool_ subclasses some class like numpy.int_. Georg From mistersheik at gmail.com Sat Feb 14 21:15:26 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 14 Feb 2015 15:15:26 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: <20150214122332.GV2498@ando.pearwood.info> References: <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <20150214122332.GV2498@ando.pearwood.info> Message-ID: I think the __make_me__ pattern discussed earlier is still the most generic cooperative solution. Here it is with a classmethod version too: class C(D, E): def some_method(self): return __make_me__(self, C) def __make_me__(self, arg_cls, *args, **kwargs): if arg_cls is C: pass elif issubclass(D, arg_cls): args, kwargs = modified_args_for_D(args, kwargs) elif issubclass(E, arg_cls): args, kwargs = modified_args_for_D(args, kwargs) else: raise ValueError if self.__class__ == C: return C(*args, **kwargs) return self.__make_me__(C, *args, **kwargs) @classmethod def __make_me_cls__(cls, arg_cls, *args, **kwargs): if arg_cls is C: pass elif issubclass(D, arg_cls): args, kwargs = modified_args_for_D(args, kwargs) elif issubclass(E, arg_cls): args, kwargs = modified_args_for_D(args, kwargs) else: raise ValueError if cls == C: return C(*args, **kwargs) return cls.__make_me_cls__(C, *args, **kwargs) On Sat, Feb 14, 2015 at 7:23 AM, Steven D'Aprano wrote: > On Fri, Feb 13, 2015 at 06:03:35PM -0500, Neil Girdhar wrote: > > I personally don't think this is a big enough issue to warrant any > changes, > > but I think Serhiy's solution would be the ideal best with one additional > > parameter: the caller's type. Something like > > > > def __make_me__(self, cls, *args, **kwargs) > > > > and the idea is that any time you want to construct a type, instead of > > > > self.__class__(assumed arguments?) > > > > where you are not sure that the derived class' constructor knows the > right > > argument types, you do > > > > def SomeCls: > > def some_method(self, ...): > > return self.__make_me__(SomeCls, assumed arguments?) > > > > Now the derived class knows who is asking for a copy. > > What if you wish to return an instance from a classmethod? You don't > have a `self` available. > > class SomeCls: > def __init__(self, x, y, z): > ... > @classmethod > def from_spam(cls, spam): > x, y, z = process(spam) > return cls.__make_me__(self, cls, x, y, z) # oops, no self > > > Even if you are calling from an instance method, and self is available, > you cannot assume that the information needed for the subclass > constructor is still available. Perhaps that information is used in the > constructor and then discarded. > > The problem we wish to solve is that when subclassing, methods of some > base class blindly return instances of itself, instead of self's type: > > > py> class MyInt(int): > ... pass > ... > py> n = MyInt(23) > py> assert isinstance(n, MyInt) > py> assert isinstance(n+1, MyInt) > Traceback (most recent call last): > File "", line 1, in ? > AssertionError > > > The means that subclasses often have to override all the parent's > methods, just to ensure the type is correct: > > class MyInt(int): > def __add__(self, other): > o = super().__add__(other) > if o is not NotImplemented: > o = type(self)(o) > return o > > > Something like that, repeated for all the int methods, should work: > > py> n = MyInt(23) > py> type(n+1) > > > > This is tedious and error prone, but at least once it is done, > subclasses of MyInt will Just Work: > > > py> class MyOtherInt(MyInt): > ... pass > ... > py> a = MyOtherInt(42) > py> type(a + 1000) > > > > (At least, *in general* they will work. See below.) > > So, why not have int's methods use type(self) instead of hard coding > int? The answer is that *some* subclasses might override the > constructor, which would cause the __add__ method to fail: > > # this will fail if the constructor has a different signature > o = type(self)(o) > > > Okay, but changing the constructor signature is quite unusual. Mostly, > people subclass to add new methods or attributes, or to override a > specific method. The dict/defaultdict situation is relatively uncommon. > > Instead of requiring *every* subclass to override all the methods, > couldn't we require the base classes (like int) to assume that the > signature is unchanged and call type(self), and leave it up to the > subclass to override all the methods *only* if the signature has > changed? (Which they probably would have to do anyway.) > > As the MyInt example above shows, or datetime in the standard library, > this actually works fine in practice: > > py> from datetime import datetime > py> class MySpecialDateTime(datetime): > ... pass > ... > py> t = MySpecialDateTime.today() > py> type(t) > > > > Why can't int, str, list, tuple etc. be more like datetime? > > > > -- > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benhoyt at gmail.com Sat Feb 14 21:32:07 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Sat, 14 Feb 2015 15:32:07 -0500 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: > +1 we need to provide the inode (we shouldn't be throwing anything from the > underlying directory entry away when possible). But... > > I think the "or None" semantics are a bad idea. It'd be better for this to > raise AttributeError on Windows so that someone can't write the most natural > form of code assuming that inode is valid and have it appear to work on > Windows when in fact it'd do the wrong thing. +1 for inode support. I agree with the above -- it should either raise AttributeError on Windows if it's not going to be set ... or it should be more like Victor's original proposal where .inode() is a method that calls stat on Windows. I don't have strong feelings. -Ben From mistersheik at gmail.com Sat Feb 14 21:42:27 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 14 Feb 2015 15:42:27 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <20150214122332.GV2498@ando.pearwood.info> Message-ID: Oops, I meant to call super if necessary: @classmethod def __make_me_cls__(cls, arg_cls, *args, **kwargs): if arg_cls is C: pass elif arg_cls is D: args, kwargs = modified_args_for_D(args, kwargs) elif arg_cls is E: args, kwargs = modified_args_for_D(args, kwargs) else: return super().__make_me_cls__(arg_cls, args, kwargs) if cls is C: return C(*args, **kwargs) return cls.__make_me_cls__(C, *args, **kwargs) On Sat, Feb 14, 2015 at 3:15 PM, Neil Girdhar wrote: > I think the __make_me__ pattern discussed earlier is still the most > generic cooperative solution. Here it is with a classmethod version too: > > class C(D, E): > def some_method(self): > return __make_me__(self, C) > > def __make_me__(self, arg_cls, *args, **kwargs): > if arg_cls is C: > pass > elif issubclass(D, arg_cls): > args, kwargs = modified_args_for_D(args, kwargs) > elif issubclass(E, arg_cls): > args, kwargs = modified_args_for_D(args, kwargs) > else: > raise ValueError > > if self.__class__ == C: > return C(*args, **kwargs) > return self.__make_me__(C, *args, **kwargs) > > @classmethod > def __make_me_cls__(cls, arg_cls, *args, **kwargs): > if arg_cls is C: > pass > elif issubclass(D, arg_cls): > args, kwargs = modified_args_for_D(args, kwargs) > elif issubclass(E, arg_cls): > args, kwargs = modified_args_for_D(args, kwargs) > else: > raise ValueError > > if cls == C: > return C(*args, **kwargs) > return cls.__make_me_cls__(C, *args, **kwargs) > > > On Sat, Feb 14, 2015 at 7:23 AM, Steven D'Aprano > wrote: > >> On Fri, Feb 13, 2015 at 06:03:35PM -0500, Neil Girdhar wrote: >> > I personally don't think this is a big enough issue to warrant any >> changes, >> > but I think Serhiy's solution would be the ideal best with one >> additional >> > parameter: the caller's type. Something like >> > >> > def __make_me__(self, cls, *args, **kwargs) >> > >> > and the idea is that any time you want to construct a type, instead of >> > >> > self.__class__(assumed arguments?) >> > >> > where you are not sure that the derived class' constructor knows the >> right >> > argument types, you do >> > >> > def SomeCls: >> > def some_method(self, ...): >> > return self.__make_me__(SomeCls, assumed arguments?) >> > >> > Now the derived class knows who is asking for a copy. >> >> What if you wish to return an instance from a classmethod? You don't >> have a `self` available. >> >> class SomeCls: >> def __init__(self, x, y, z): >> ... >> @classmethod >> def from_spam(cls, spam): >> x, y, z = process(spam) >> return cls.__make_me__(self, cls, x, y, z) # oops, no self >> >> >> Even if you are calling from an instance method, and self is available, >> you cannot assume that the information needed for the subclass >> constructor is still available. Perhaps that information is used in the >> constructor and then discarded. >> >> The problem we wish to solve is that when subclassing, methods of some >> base class blindly return instances of itself, instead of self's type: >> >> >> py> class MyInt(int): >> ... pass >> ... >> py> n = MyInt(23) >> py> assert isinstance(n, MyInt) >> py> assert isinstance(n+1, MyInt) >> Traceback (most recent call last): >> File "", line 1, in ? >> AssertionError >> >> >> The means that subclasses often have to override all the parent's >> methods, just to ensure the type is correct: >> >> class MyInt(int): >> def __add__(self, other): >> o = super().__add__(other) >> if o is not NotImplemented: >> o = type(self)(o) >> return o >> >> >> Something like that, repeated for all the int methods, should work: >> >> py> n = MyInt(23) >> py> type(n+1) >> >> >> >> This is tedious and error prone, but at least once it is done, >> subclasses of MyInt will Just Work: >> >> >> py> class MyOtherInt(MyInt): >> ... pass >> ... >> py> a = MyOtherInt(42) >> py> type(a + 1000) >> >> >> >> (At least, *in general* they will work. See below.) >> >> So, why not have int's methods use type(self) instead of hard coding >> int? The answer is that *some* subclasses might override the >> constructor, which would cause the __add__ method to fail: >> >> # this will fail if the constructor has a different signature >> o = type(self)(o) >> >> >> Okay, but changing the constructor signature is quite unusual. Mostly, >> people subclass to add new methods or attributes, or to override a >> specific method. The dict/defaultdict situation is relatively uncommon. >> >> Instead of requiring *every* subclass to override all the methods, >> couldn't we require the base classes (like int) to assume that the >> signature is unchanged and call type(self), and leave it up to the >> subclass to override all the methods *only* if the signature has >> changed? (Which they probably would have to do anyway.) >> >> As the MyInt example above shows, or datetime in the standard library, >> this actually works fine in practice: >> >> py> from datetime import datetime >> py> class MySpecialDateTime(datetime): >> ... pass >> ... >> py> t = MySpecialDateTime.today() >> py> type(t) >> >> >> >> Why can't int, str, list, tuple etc. be more like datetime? >> >> >> >> -- >> Steve >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Feb 14 21:44:11 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Feb 2015 21:44:11 +0100 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20150214214411.1917b4d6@fsol> On Sat, 14 Feb 2015 15:32:07 -0500 Ben Hoyt wrote: > > +1 we need to provide the inode (we shouldn't be throwing anything from the > > underlying directory entry away when possible). But... > > > > I think the "or None" semantics are a bad idea. It'd be better for this to > > raise AttributeError on Windows so that someone can't write the most natural > > form of code assuming that inode is valid and have it appear to work on > > Windows when in fact it'd do the wrong thing. > > +1 for inode support. I agree with the above -- it should either raise > AttributeError on Windows if it's not going to be set ... or it should > be more like Victor's original proposal where .inode() is a method > that calls stat on Windows. I don't have strong feelings. The whole point of scandir is to expose low-level system calls in a cross-platform way. If you start raising some exceptions on some platforms then that quality disappears. Regards Antoine. From alexander.belopolsky at gmail.com Sat Feb 14 21:47:28 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 14 Feb 2015 15:47:28 -0500 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <75908557b011a.54de5f34@wiscmail.wisc.edu> <75a0ded3b060d.54de5f70@wiscmail.wisc.edu> <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <20150214122332.GV2498@ando.pearwood.info> Message-ID: On Sat, Feb 14, 2015 at 2:36 PM, Georg Brandl wrote: > > In the case of int, there is a good reason for this behavior - bool. In > python, > > we want True + True == 2. In numpy, where binary operations preserve > > subclasses, you have > > > >>>> import numpy > >>>> numpy.bool_(1) + numpy.bool_(1) > > True > > I don't think numpy.bool_ subclasses some class like numpy.int_. And numpy.bool_ subclasses don't preserve type in addition: >>> import numpy >>> class Bool(numpy.bool_): ... pass ... >>> numpy.bool_.mro() [, , ] >>> Bool(1) + Bool(1) True >>> type(_) So there goes my theory. :-) I think all these examples just highlight the need for a clear guidance when self.__class__() can be called in base classes to construct instances of derived classes. Apparently numpy has it both ways. One way for scalars (see above) and the other for arrays: >>> class Array(numpy.ndarray): ... pass ... >>> a = Array(1) >>> a[0] = 1 >>> a+a Array([ 2.]) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Sat Feb 14 22:11:25 2015 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 14 Feb 2015 21:11:25 +0000 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> <20150214214411.1917b4d6@fsol> Message-ID: That suggests the .inode() method approach makes more sense then. On Sat, Feb 14, 2015, 12:44 PM Antoine Pitrou wrote: > On Sat, 14 Feb 2015 15:32:07 -0500 > Ben Hoyt wrote: > > > +1 we need to provide the inode (we shouldn't be throwing anything > from the > > > underlying directory entry away when possible). But... > > > > > > I think the "or None" semantics are a bad idea. It'd be better for > this to > > > raise AttributeError on Windows so that someone can't write the most > natural > > > form of code assuming that inode is valid and have it appear to work on > > > Windows when in fact it'd do the wrong thing. > > > > +1 for inode support. I agree with the above -- it should either raise > > AttributeError on Windows if it's not going to be set ... or it should > > be more like Victor's original proposal where .inode() is a method > > that calls stat on Windows. I don't have strong feelings. > > The whole point of scandir is to expose low-level system calls in a > cross-platform way. If you start raising some exceptions on some > platforms then that quality disappears. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benhoyt at gmail.com Sat Feb 14 22:17:31 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Sat, 14 Feb 2015 16:17:31 -0500 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: <20150214214411.1917b4d6@fsol> References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> <20150214214411.1917b4d6@fsol> Message-ID: >> +1 for inode support. I agree with the above -- it should either raise >> AttributeError on Windows if it's not going to be set ... or it should >> be more like Victor's original proposal where .inode() is a method >> that calls stat on Windows. I don't have strong feelings. > > The whole point of scandir is to expose low-level system calls in a > cross-platform way. If you start raising some exceptions on some > platforms then that quality disappears. I agree with that! -Ben From marko at pacujo.net Sat Feb 14 22:14:26 2015 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 14 Feb 2015 23:14:26 +0200 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: <20150214214411.1917b4d6@fsol> (Antoine Pitrou's message of "Sat, 14 Feb 2015 21:44:11 +0100") References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> <20150214214411.1917b4d6@fsol> Message-ID: <87r3tsrya5.fsf@elektro.pacujo.net> Antoine Pitrou : > The whole point of scandir is to expose low-level system calls in a > cross-platform way. Cross-platform is great and preferable, but low-level system facilities should be made available even when they are unique to a particular OS. Marko From benhoyt at gmail.com Sat Feb 14 22:25:42 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Sat, 14 Feb 2015 16:25:42 -0500 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: <87r3tsrya5.fsf@elektro.pacujo.net> References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> <20150214214411.1917b4d6@fsol> <87r3tsrya5.fsf@elektro.pacujo.net> Message-ID: >> The whole point of scandir is to expose low-level system calls in a >> cross-platform way. > > Cross-platform is great and preferable, but low-level system facilities > should be made available even when they are unique to a particular OS. Yes, but this can be made cross-platform fairly easily, just like the other method calls. Just like DirEntry.stat() has different cross-platform operation (no stat call on Windows, a stat call on POSIX), DirEntry.inode() would have a different operation (stat call on Windows, no stat call on POSIX). -Ben From victor.stinner at gmail.com Sun Feb 15 00:03:54 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 15 Feb 2015 00:03:54 +0100 Subject: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method? In-Reply-To: References: <87a90hmd9c.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Le 14 f?vr. 2015 18:47, "Gregory P. Smith" a ?crit : > I think the "or None" semantics are a bad idea. Oh, in fact it shouldn't be None but 0 onWindows to be consistent with DirEntry.stat().st_ino which is also equal to 0. The value 0 is not a valid inode number. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sun Feb 15 01:54:14 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 15 Feb 2015 00:54:14 +0000 Subject: [Python-Dev] Can I replace distutils.msvccompiler? Message-ID: <1423961656813.23811@microsoft.com> I'm about to start doing a rework on distutils.msvc[9]compiler.MSVCCompiler for Python 3.5 to be able to handle forward-compatibility better. Right now I've tweaked msvc9compiler enough for VS 2015, but it won't handle later VS versions automatically. Is there any reason to keep the old msvccompiler and msvc9compiler modules around? If possible I'd like to drop them completely and just have one implementation (probably distutils.msvccompiler.MSVCCompiler again). Obviously it will be behaviourally compatible with 3.4 and compatible with the CCompiler interface, but it may not be function-for-function compatible with distutils.msvc9compiler (e.g. get_build_version is not useful). Do we need a deprecation period? msvccompiler is mentioned in the docs, but not documented, and msvc9compiler is not even mentioned. Thanks, Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Feb 15 03:04:06 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 15 Feb 2015 13:04:06 +1100 Subject: [Python-Dev] subclassing builtin data structures In-Reply-To: References: <75e095d7b6cdd.54de5fad@wiscmail.wisc.edu> <77408255b368c.54de5fe9@wiscmail.wisc.edu> <7590ea0db149b.54de6026@wiscmail.wisc.edu> <76b0c98db1162.54de6062@wiscmail.wisc.edu> <75909e0fb2b09.54de0c27@wiscmail.wisc.edu> <20150214122332.GV2498@ando.pearwood.info> Message-ID: <20150215020406.GC2498@ando.pearwood.info> On Sat, Feb 14, 2015 at 01:26:36PM -0500, Alexander Belopolsky wrote: > On Sat, Feb 14, 2015 at 7:23 AM, Steven D'Aprano > wrote: > > > Why can't int, str, list, tuple etc. be more like datetime? > > > They are. In all these types, class methods call subclass constructors but > instance methods don't. But in datetime, instance methods *do*. Sorry that my example with .today() was misleading. py> from datetime import datetime py> class MyDatetime(datetime): ... pass ... py> MyDatetime.today() MyDatetime(2015, 2, 15, 12, 45, 38, 429269) py> MyDatetime.today().replace(day=20) MyDatetime(2015, 2, 20, 12, 45, 53, 405889) > In the case of int, there is a good reason for this behavior - bool. In > python, we want True + True == 2. Sure. But bool is only one subclass. I expect that it should be bool's responsibility to override __add__ etc. to return an instance of the parent class (int) rather have nearly all subclasses have to override __add__ etc. to return instances of themselves. -- Steve From ncoghlan at gmail.com Sun Feb 15 06:33:15 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Feb 2015 15:33:15 +1000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 00:44, Paul Moore wrote: > I'm looking at putting together a patch for CPython to implement PEP > 441. In doing so, there are a few issues that I'd like to raise with > the PEP. These are all to do with the supporting app "pyzaa" (IIRC, > Nick proposed renaming this to "pyzapp", which I like, but it's not a > big deal either way). > > 1. The PEP says that the application will warn if you try to add C > extensions to the archive. In practice, this will be quite hard to do > - extensions vary across platforms, and it's not only C extensions > that won't work from an archive (consider a DLL with a pure-python > ctypes wrapper). We could check the filenames for "known" extensions > (maybe using importlib.machinery.EXTENSION_SUFFIXES, but that varies > by platform) but I think that's going to be fragile. IMO, the costs > aren't worth the benefits, and I'd like to remove this proposal and > simply document that applications packed up with pyzaa need to be > tested to ensure they work from a zipfile. > > 2. The option to compile Python source seems to me to be of limited > use. It ties the archive to a particular Python version (more > precisely, only one version gets the benefits of precompiled > bytecode). It's also tricky to implement, particularly the option to > compile an existing archive, because there standard library py_compile > and compileall modules work on the filesystem, and for compiling files > in an archive it would be better to build the bytecode in memory (no > need to use tempfiles). I'd prefer to say that if users want to > include bytecode, they can precompile on disk (using compileall) for > as many versions as they want. > > FWIW, Daniel's standalone pyzaa program doesn't currently include > either of the above features. "Do the simplest thing that could possibly work" applies, so I'd be OK with dropping both of those features. > As well as removing things, I'd like to add a programmable API for the > utility. At the moment I only have a rough idea of how that would > work, but I'm thinking of a PyZApp class, which can be created from a > filename or an open stream (like a ZipFile object) and which has > methods to set the shebang line, setting the main entry point, and > adding files or directories. Does that need to go into the PEP or is > it OK to review the API as part of the patch? I don't mind either way, > but my normal coding process tends to be to fiddle with the API as I > code it up and use it in the main app, so I'd normally leave > documenting the final API till it's written... I'd suggest including the API in the PEP, which can co-evolve with the reference implementation. The following suggestions are ones I came up with last time I looked closely at the draft PEP: * use "zipapp" for the support module name, distinct from the command line tool name (think venv vs pyvenv) * expose a zipapp.pack programmatic interface that takes all the input pieces and builds a suitable archive I'd definitely prefer a simple procedural API like that over offering a public stateful object-oriented API, as the latter significantly increases testing complexity without offering a sufficiently compelling gain in expressiveness to justify the additional upfront test development cost and the ongoing maintenance and support burden. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Feb 15 09:14:02 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 08:14:02 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 05:33, Nick Coghlan wrote: > I'd definitely prefer a simple procedural API like that over offering > a public stateful object-oriented API, as the latter significantly > increases testing complexity without offering a sufficiently > compelling gain in expressiveness to justify the additional upfront > test development cost and the ongoing maintenance and support burden. The main advantage of a class API is the ability to support building an archive from data in memory (or by collecting parts from different areas on the filesystem). That's something I've had a use for on a number of occasions. OTOH, if the pyz/pyzw extensions are already (or will be) registered by the installer then the only remaining feature of this PEP [1] is the archive creation app. And designing that as we go is probably the wrong way to get something in the stdlib. Maybe it would be better to put something on PyPI and let it develop outside the stdlib first? There's already at least pyzaa and pyzzer available... Paul [1] Apart from the benefit of publicising executable zip files as a means of deploying Python code by mentioning them in a "What's new" section :-) From p.f.moore at gmail.com Sun Feb 15 09:47:50 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 08:47:50 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 08:14, Paul Moore wrote: > Maybe it would be better to > put something on PyPI and let it develop outside the stdlib first? The only place where a ".pyz" file can't easily be manipulated with the stdlib zipfile module is in writing a shebang line to the start of the archive (i.e. adding "prefix" bytes before the start of the zipfile data). It would be nice if the ZipFile class supported this (because to do it properly you need access to the file object that the ZipFile object wraps). Would it be reasonable to add methods to the ZipFile class to read and write the prefix data? Paul From ncoghlan at gmail.com Sun Feb 15 09:59:17 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Feb 2015 18:59:17 +1000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 18:14, Paul Moore wrote: > OTOH, if the pyz/pyzw extensions are already (or will be) registered > by the installer then the only remaining feature of this PEP [1] is > the archive creation app. And designing that as we go is probably the > wrong way to get something in the stdlib. Maybe it would be better to > put something on PyPI and let it develop outside the stdlib first? > There's already at least pyzaa and pyzzer available... Perhaps rather than creating yet another implementation of this, we should just add some "See Also" links to the relevant parts of the command line usage guide and the runpy module documentation? Particularly if the modules are pip-installable and we add the cross-references to the 2.7 and 3.4 docs as well - we didn't install pip by default back when Daniel first wrote PEP 441. > [1] Apart from the benefit of publicising executable zip files as a > means of deploying Python code by mentioning them in a "What's new" > section :-) That will be covered to some degree by the mention of the file associations being registered on Windows, and we can try to persuade Larry it's worth mentioning in the release announcement itself :) The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what "pyz" and "pyzw" files are later. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Feb 15 14:21:07 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 13:21:07 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 08:59, Nick Coghlan wrote: > The other option would to cut PEP 441 way back to *just* be about > standardising and registering the file associations, and recommending > the use of pip to obtain the build machinery with (whether pyzaa, > pyzzer or Twitter's more comprehensive pex). It would be a short PEP, > but potentially still worth it for the improved visibility of the > decision when folks are trying to figure out what "pyz" and "pyzw" > files are later. Ok, thinking about this a little more. Getting the extension support is the key thing on Windows - at the moment, people are faced with adding their own file associations or putting binary data in a .py file, neither of which is a nice choice. Tooling is important, though - sure, you can zip the data up and put a header on, but it's fiddly. Which brings us full circle. A simple module, executable as "python -m zipapp" (see below re name) which exports a single function, pack() that creates the archive. If we want to provide a script to wrap the module, like pyvenv.py does for venv, I've no objection to that - presumably it would go in Tools/Scripts? If people (like me) want to experiment with a more programmatic API for building pyz files, they can do so on PyPI, and if such a thing becomes sufficiently mature we can look then at proposing it for inclusion in the stdlib, as an extension to the zipapp module. Regarding naming, I'm happy to go with zipapp if it's your preference. Presumably the wrapper in Tools/Scripts would be pyzipapp.py? So the usage would be something like python -m zipapp [options] dir_to_zip Options: -p The interpreter to use on the shebang line (defaulting to /usr/bin/env python) -o archive_name The name of the output file (defaulting to the source directory name with a .pyz extension) If the argument has no extension, add '.pyz' -m module:function The entry point to call (written to __main__.py) Using this is an error if there is a __main__.py, and mandatory if there isn't If you want anything more complex, it's easy enough to write your own script based on zipfile, or use one of the modules on PyPI. Does this sound reasonable? If it's OK, I'll go ahead and prepare an update to the PEP and an implementation. (Steve, looks like I may be learning how to maintain the wix files after all - wish me luck :-)) If I hear no objections in the next couple of days, I'll assume everyone's OK with it and I'll prepare a PEP update and a patch. Paul From p.f.moore at gmail.com Sun Feb 15 14:27:54 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 13:27:54 +0000 Subject: [Python-Dev] PEP 486: Make the Python Launcher aware of virtual environments In-Reply-To: References: <54DD034B.2030907@stoneleaf.us> Message-ID: On 13 February 2015 at 07:02, Nick Coghlan wrote: > Procedurally, since it's a Windows specific change, and assuming Guido > doesn't want to pronounce on this one himself, perhaps Steve Dower > would be an appropriate BDFL-Delegate? The PEP is now up to date and the patch is on the tracker at http://bugs.python.org/issue23465. There's not been any further questions or comments, so I think I'm happy with it. As Guido's now away (I gather) I'll wait till he gets back and then post a formal request for pronouncement (or appointment of a BDFL-Delegate to pronounce). If anyone has any issues with the PEP, please speak up before then! Paul From ncoghlan at gmail.com Sun Feb 15 14:47:17 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Feb 2015 23:47:17 +1000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 23:21, Paul Moore wrote: > On 15 February 2015 at 08:59, Nick Coghlan wrote: >> The other option would to cut PEP 441 way back to *just* be about >> standardising and registering the file associations, and recommending >> the use of pip to obtain the build machinery with (whether pyzaa, >> pyzzer or Twitter's more comprehensive pex). It would be a short PEP, >> but potentially still worth it for the improved visibility of the >> decision when folks are trying to figure out what "pyz" and "pyzw" >> files are later. > > Ok, thinking about this a little more. > > Getting the extension support is the key thing on Windows - at the > moment, people are faced with adding their own file associations or > putting binary data in a .py file, neither of which is a nice choice. > Tooling is important, though - sure, you can zip the data up and put a > header on, but it's fiddly. > > Which brings us full circle. A simple module, executable as "python -m > zipapp" (see below re name) which exports a single function, pack() > that creates the archive. If we want to provide a script to wrap the > module, like pyvenv.py does for venv, I've no objection to that - > presumably it would go in Tools/Scripts? If people (like me) want to > experiment with a more programmatic API for building pyz files, they > can do so on PyPI, and if such a thing becomes sufficiently mature we > can look then at proposing it for inclusion in the stdlib, as an > extension to the zipapp module. > > Regarding naming, I'm happy to go with zipapp if it's your preference. > Presumably the wrapper in Tools/Scripts would be pyzipapp.py? Or we just skip the wrapper and make "python -m zipapp" the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult. > > So the usage would be something like > > python -m zipapp [options] dir_to_zip > > Options: > -p The interpreter to use on the shebang line > (defaulting to /usr/bin/env python) > -o archive_name The name of the output file (defaulting to the > source directory name with a .pyz extension) > If the argument has no extension, add '.pyz' > -m module:function The entry point to call (written to __main__.py) > Using this is an error if there is a > __main__.py, and mandatory if there isn't > > If you want anything more complex, it's easy enough to write your own > script based on zipfile, or use one of the modules on PyPI. > > Does this sound reasonable? If it's OK, I'll go ahead and prepare an > update to the PEP and an implementation. (Steve, looks like I may be > learning how to maintain the wix files after all - wish me luck :-)) > If I hear no objections in the next couple of days, I'll assume > everyone's OK with it and I'll prepare a PEP update and a patch. Sounds good to me. Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates it as described? It seems a bit tidier than allocating a new PEP number and rejecting PEP 441, when the revised proposal is essentially just a simplified and renamed version of your original idea. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Feb 15 15:07:07 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 14:07:07 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 13:47, Nick Coghlan wrote: > Or we just skip the wrapper and make "python -m zipapp" the > recommended invocation style. Adding a wrapper later is fairly easy, > but removing it would be difficult. Fine with me - the wrappers are basically useless on Windows where Tools/Scripts isn't on PATH. Paul From storchaka at gmail.com Sun Feb 15 17:00:36 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 15 Feb 2015 18:00:36 +0200 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15.02.15 10:47, Paul Moore wrote: > On 15 February 2015 at 08:14, Paul Moore wrote: >> Maybe it would be better to >> put something on PyPI and let it develop outside the stdlib first? > > The only place where a ".pyz" file can't easily be manipulated with > the stdlib zipfile module is in writing a shebang line to the start of > the archive (i.e. adding "prefix" bytes before the start of the > zipfile data). It would be nice if the ZipFile class supported this > (because to do it properly you need access to the file object that the > ZipFile object wraps). Would it be reasonable to add methods to the > ZipFile class to read and write the prefix data? But the stdlib zipfile module supports this. with open(filename, 'wb') as f: f.write(shebang) with zipfile.PyZipFile(f, 'a') as zf: ... From thomas at python.org Sun Feb 15 17:21:08 2015 From: thomas at python.org (Thomas Wouters) Date: Sun, 15 Feb 2015 17:21:08 +0100 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: FWIW, we have a similar tool to 'pyzaa' at Google, although a lot older (it has a pure-python implementation of zipimport, because it had to work with Python 2.2 back in the day) that *does* support extension modules and other shared libraries (by extracting them to disk on program startup.) We're also working to replace that by loading the extension modules directly from the archive (using dlopen_with_offset as described by ppluzhnikov in https://sourceware.org/bugzilla/show_bug.cgi?id=11767, although it should also be possible using dlopen_ehdr/dlopen_phdr), which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything, but it turns out it's easy to hack around by overriding _writecheck :P) And yes, we really are in a position to modify glibc to make our life distributing Python applications easier, internally; the old code involves shell and Python bootstrap code that tries very hard to avoid race conditions and security problems, which is a pain to maintain. It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all. I'd be happy to discuss this in more detail at PyCon, at which time we should be deploying code internally using all of this. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Feb 15 17:27:10 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 16:27:10 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 16:00, Serhiy Storchaka wrote: >> Would it be reasonable to add methods to the >> ZipFile class to read and write the prefix data? > > > But the stdlib zipfile module supports this. > > with open(filename, 'wb') as f: > f.write(shebang) > with zipfile.PyZipFile(f, 'a') as zf: Good point. It's not quite as easy to read the prefix data. I think you can do it by reopening the file, and reading the bytes up to byte min(i.header_offset for i in zf.infolist()). You need to reopen the file, though, and that calculation doesn't work for an empty zipfile (where you need to read to the start of the central directory instead). But it's sort of possible, and it's a pretty specialised requirement anyway (I only really needed it for testing). Thanks, Paul From p.f.moore at gmail.com Sun Feb 15 17:43:19 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 16:43:19 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 16:15, Petr Viktorin wrote: > On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore wrote: >> So the usage would be something like >> >> python -m zipapp [options] dir_to_zip >> >> Options: >> -p The interpreter to use on the shebang line >> (defaulting to /usr/bin/env python) > > On many systems this default would mean Python 2. Even if the official > recommendation changes for 3.5, the status quo might linger for a few > years. > On the other hand, the number of distros that don't ship Python 3 is > small, and the reason they're slow-moving tends to be stability and/or > compliance, so they're not the target audience for zipapp. And the > python3 symlink is not going away any time soon. > So I'd suggest `/usr/bin/env python3` for the default. But that will fail for users who only have Python 2 installed. And it won't pick up an activated virtualenv (assuming virtualenvs on Unix don't include a python3 command, which I believe is true - it certainly won't on Windows). As a Windows user, I believe that the command "python" should run the version of Python that you choose to be your default. I know it's not quite that simple on Unix, and the system "python" command is typically Python 2, with "python3" for the system Python 3 version, but I also know that tools like pyenv work like that. Equally, I've seen lots of scripts with "#!/usr/bin/env python" as the shebang line, and none with "#!/usr/bin/env python3". That may just be my limited experience with Unix though. I don't really want "#!/usr/bin/env python3" as the default shebang on Windows (at a minimum, it would do the wrong thing for my current environment). I'm open to advice from the Unix users as to how to set things up better on Unix, but there's a whole new set of problems that will arise as soon as we have different defaults on Unix and Windows, so IMO there would need to be pretty significant benefits to justify doing that. And of course it *is* only a default - users can set whatever they want. (But getting defaults right is important, so that's not to dismiss the point). Paul From p.f.moore at gmail.com Sun Feb 15 17:49:59 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 16:49:59 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 16:21, Thomas Wouters wrote: > It might be nice to consider those use-cases in pyzapp as well, especially > once the glibc feature is released. It requires some fairly big changes to > zipimport (I ended up rewriting the whole thing) but we can easily > opensource the code, if there's any interest at all. I'd be happy to discuss > this in more detail at PyCon, at which time we should be deploying code > internally using all of this. Those cases sound more like things to consider for Python's "executable zip" support. The zipapp module is purely concerned with taking a bunch of stuff and making a zipfile-with-shebang "python-executable zipfile" out of it. Of course if Python needed certain files to be aligned a particular way in order to execute the zipfile, that *would* be something that zipapp should take into account. But I'd say let's wait till zipfile execution needs the feature before adding it to zipapp. Paul From encukou at gmail.com Sun Feb 15 17:15:54 2015 From: encukou at gmail.com (Petr Viktorin) Date: Sun, 15 Feb 2015 17:15:54 +0100 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore wrote: > So the usage would be something like > > python -m zipapp [options] dir_to_zip > > Options: > -p The interpreter to use on the shebang line > (defaulting to /usr/bin/env python) On many systems this default would mean Python 2. Even if the official recommendation changes for 3.5, the status quo might linger for a few years. On the other hand, the number of distros that don't ship Python 3 is small, and the reason they're slow-moving tends to be stability and/or compliance, so they're not the target audience for zipapp. And the python3 symlink is not going away any time soon. So I'd suggest `/usr/bin/env python3` for the default. From dholth at gmail.com Sun Feb 15 18:46:11 2015 From: dholth at gmail.com (Daniel Holth) Date: Sun, 15 Feb 2015 12:46:11 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: Go ahead, make my pep. I will appreciate seeing it happen. On Feb 15, 2015 8:47 AM, "Nick Coghlan" wrote: > On 15 February 2015 at 23:21, Paul Moore wrote: > > On 15 February 2015 at 08:59, Nick Coghlan wrote: > >> The other option would to cut PEP 441 way back to *just* be about > >> standardising and registering the file associations, and recommending > >> the use of pip to obtain the build machinery with (whether pyzaa, > >> pyzzer or Twitter's more comprehensive pex). It would be a short PEP, > >> but potentially still worth it for the improved visibility of the > >> decision when folks are trying to figure out what "pyz" and "pyzw" > >> files are later. > > > > Ok, thinking about this a little more. > > > > Getting the extension support is the key thing on Windows - at the > > moment, people are faced with adding their own file associations or > > putting binary data in a .py file, neither of which is a nice choice. > > Tooling is important, though - sure, you can zip the data up and put a > > header on, but it's fiddly. > > > > Which brings us full circle. A simple module, executable as "python -m > > zipapp" (see below re name) which exports a single function, pack() > > that creates the archive. If we want to provide a script to wrap the > > module, like pyvenv.py does for venv, I've no objection to that - > > presumably it would go in Tools/Scripts? If people (like me) want to > > experiment with a more programmatic API for building pyz files, they > > can do so on PyPI, and if such a thing becomes sufficiently mature we > > can look then at proposing it for inclusion in the stdlib, as an > > extension to the zipapp module. > > > > Regarding naming, I'm happy to go with zipapp if it's your preference. > > Presumably the wrapper in Tools/Scripts would be pyzipapp.py? > > Or we just skip the wrapper and make "python -m zipapp" the > recommended invocation style. Adding a wrapper later is fairly easy, > but removing it would be difficult. > > > > > So the usage would be something like > > > > python -m zipapp [options] dir_to_zip > > > > Options: > > -p The interpreter to use on the shebang line > > (defaulting to /usr/bin/env python) > > -o archive_name The name of the output file (defaulting to the > > source directory name with a .pyz extension) > > If the argument has no extension, add '.pyz' > > -m module:function The entry point to call (written to __main__.py) > > Using this is an error if there is a > > __main__.py, and mandatory if there isn't > > > > If you want anything more complex, it's easy enough to write your own > > script based on zipfile, or use one of the modules on PyPI. > > > > Does this sound reasonable? If it's OK, I'll go ahead and prepare an > > update to the PEP and an implementation. (Steve, looks like I may be > > learning how to maintain the wix files after all - wish me luck :-)) > > If I hear no objections in the next couple of days, I'll assume > > everyone's OK with it and I'll prepare a PEP update and a patch. > > Sounds good to me. > > Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates > it as described? It seems a bit tidier than allocating a new PEP > number and rejecting PEP 441, when the revised proposal is essentially > just a simplified and renamed version of your original idea. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sun Feb 15 19:06:12 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 15 Feb 2015 18:06:12 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: , Message-ID: "Go ahead, make my pep." We should make a python-dev t-shirt with this on it :) Top-posted from my Windows Phone ________________________________ From: Daniel Holth Sent: ?2/?15/?2015 9:46 To: Nick Coghlan Cc: Paul Moore; Steve Dower; Python Dev Subject: Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support Go ahead, make my pep. I will appreciate seeing it happen. On Feb 15, 2015 8:47 AM, "Nick Coghlan" > wrote: On 15 February 2015 at 23:21, Paul Moore > wrote: > On 15 February 2015 at 08:59, Nick Coghlan > wrote: >> The other option would to cut PEP 441 way back to *just* be about >> standardising and registering the file associations, and recommending >> the use of pip to obtain the build machinery with (whether pyzaa, >> pyzzer or Twitter's more comprehensive pex). It would be a short PEP, >> but potentially still worth it for the improved visibility of the >> decision when folks are trying to figure out what "pyz" and "pyzw" >> files are later. > > Ok, thinking about this a little more. > > Getting the extension support is the key thing on Windows - at the > moment, people are faced with adding their own file associations or > putting binary data in a .py file, neither of which is a nice choice. > Tooling is important, though - sure, you can zip the data up and put a > header on, but it's fiddly. > > Which brings us full circle. A simple module, executable as "python -m > zipapp" (see below re name) which exports a single function, pack() > that creates the archive. If we want to provide a script to wrap the > module, like pyvenv.py does for venv, I've no objection to that - > presumably it would go in Tools/Scripts? If people (like me) want to > experiment with a more programmatic API for building pyz files, they > can do so on PyPI, and if such a thing becomes sufficiently mature we > can look then at proposing it for inclusion in the stdlib, as an > extension to the zipapp module. > > Regarding naming, I'm happy to go with zipapp if it's your preference. > Presumably the wrapper in Tools/Scripts would be pyzipapp.py? Or we just skip the wrapper and make "python -m zipapp" the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult. > > So the usage would be something like > > python -m zipapp [options] dir_to_zip > > Options: > -p The interpreter to use on the shebang line > (defaulting to /usr/bin/env python) > -o archive_name The name of the output file (defaulting to the > source directory name with a .pyz extension) > If the argument has no extension, add '.pyz' > -m module:function The entry point to call (written to __main__.py) > Using this is an error if there is a > __main__.py, and mandatory if there isn't > > If you want anything more complex, it's easy enough to write your own > script based on zipfile, or use one of the modules on PyPI. > > Does this sound reasonable? If it's OK, I'll go ahead and prepare an > update to the PEP and an implementation. (Steve, looks like I may be > learning how to maintain the wix files after all - wish me luck :-)) > If I hear no objections in the next couple of days, I'll assume > everyone's OK with it and I'll prepare a PEP update and a patch. Sounds good to me. Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates it as described? It seems a bit tidier than allocating a new PEP number and rejecting PEP 441, when the revised proposal is essentially just a simplified and renamed version of your original idea. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Sun Feb 15 19:25:10 2015 From: encukou at gmail.com (Petr Viktorin) Date: Sun, 15 Feb 2015 19:25:10 +0100 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On Sun, Feb 15, 2015 at 5:43 PM, Paul Moore wrote: > On 15 February 2015 at 16:15, Petr Viktorin wrote: >> On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore wrote: >>> So the usage would be something like >>> >>> python -m zipapp [options] dir_to_zip >>> >>> Options: >>> -p The interpreter to use on the shebang line >>> (defaulting to /usr/bin/env python) >> >> On many systems this default would mean Python 2. Even if the official >> recommendation changes for 3.5, the status quo might linger for a few >> years. >> On the other hand, the number of distros that don't ship Python 3 is >> small, and the reason they're slow-moving tends to be stability and/or >> compliance, so they're not the target audience for zipapp. And the >> python3 symlink is not going away any time soon. >> So I'd suggest `/usr/bin/env python3` for the default. > > But that will fail for users who only have Python 2 installed. And it > won't pick up an activated virtualenv (assuming virtualenvs on Unix > don't include a python3 command, which I believe is true - it > certainly won't on Windows). Yes, it would fail with only Python 2 installed. I don't think that's a problem. On POSIXy systems the "python3" symlink is created in all venvs. I thought (perhaps na?vely) that Windows doesn't do shebangs natively, so there's some Python-specific mechanism around them, which should handle "python3". If the app was single-source compatible with both 2 and 3, then `/usr/bin/env python` would be a good choice. But I don't think apps (as opposed to libraries) have much incentive to keep the backwards compatibility. > As a Windows user, I believe that the command "python" should run the > version of Python that you choose to be your default. I know it's not > quite that simple on Unix, and the system "python" command is > typically Python 2, with "python3" for the system Python 3 version, > but I also know that tools like pyenv work like that. Equally, I've > seen lots of scripts with "#!/usr/bin/env python" as the shebang line, > and none with "#!/usr/bin/env python3". That may just be my limited > experience with Unix though. On Linux, what version "python" is depends on the distro. Currently the recommendation from Python upstream is for it to be Python 2, though, and I know of just one distro that does not do this (Arch, because it switched before the recommendation was put in place). If you're in a venv, then of "python" is that venv's Python, of course. But the default mode of operation for Python-unaware users is to not be in a venv. > I don't really want "#!/usr/bin/env python3" as the default shebang on > Windows (at a minimum, it would do the wrong thing for my current > environment). I'm open to advice from the Unix users as to how to set > things up better on Unix, but there's a whole new set of problems that > will arise as soon as we have different defaults on Unix and Windows, > so IMO there would need to be pretty significant benefits to justify > doing that. I'm not familiar with how the shebang works on Windows, but if "/usr/bin/env python3" breaks things, I find it highly unfortunate. > And of course it *is* only a default - users can set whatever they > want. (But getting defaults right is important, so that's not to > dismiss the point). I'm afraid the point is whether it's even possible to have a shebang that does the right thing on both platforms. From storchaka at gmail.com Sun Feb 15 19:45:00 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 15 Feb 2015 20:45:00 +0200 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15.02.15 18:21, Thomas Wouters wrote: > which requires that extension modules are stored uncompressed (simple) > and page-aligned (harder, as the zipfile.ZipFile class doesn't directly > support page-aligning anything It is possible to add this feature to ZipFile. It can be useful because will allow to mmap uncompressed files in ZIP file. From vinay_sajip at yahoo.co.uk Sun Feb 15 20:23:49 2015 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 15 Feb 2015 19:23:49 +0000 (UTC) Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows References: <2042003663.4217375.1423820835953.JavaMail.yahoo@mail.yahoo.com> Message-ID: Paul Moore gmail.com> writes: > 2. It would be a nice, although extremely obscure, feature to be able > to allow people who want to keep Python off their path to set > VIRTUAL_ENV but *not* set PATH, so that py.exe does the right thing > but there's *still* no python.exe on PATH. Limit the dependency on a > single environment variable rather than two, in other words. If this is the case, then your patch may not work if PATH isn't set. If a shebang is found in a script, it is processed in maybe_handle_shebang(), which means that your code to look in a venv won't be hit (though if you rely on the PATH being correctly set, then it will work as expected because the handling for e.g. /usr/bin/env python searches the PATH). Regards, Vinay Sajip From breamoreboy at yahoo.co.uk Sun Feb 15 20:42:09 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sun, 15 Feb 2015 19:42:09 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: , Message-ID: On 15/02/2015 18:06, Steve Dower wrote: > "Go ahead, make my pep." > > We should make a python-dev t-shirt with this on it :) > I'll buy one provided p&p isn't too high :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From p.f.moore at gmail.com Sun Feb 15 21:26:48 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 20:26:48 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <2042003663.4217375.1423820835953.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 15 February 2015 at 19:23, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> 2. It would be a nice, although extremely obscure, feature to be able >> to allow people who want to keep Python off their path to set >> VIRTUAL_ENV but *not* set PATH, so that py.exe does the right thing >> but there's *still* no python.exe on PATH. Limit the dependency on a >> single environment variable rather than two, in other words. > > If this is the case, then your patch may not work if PATH isn't set. If a > shebang is found in a script, it is processed in maybe_handle_shebang(), which > means that your code to look in a venv won't be hit (though if you rely on the > PATH being correctly set, then it will work as expected because the handling > for e.g. /usr/bin/env python searches the PATH). Yes, when I was actually coding this, I backed down on this (I updated the PEP accordingly). Now, running py.exe with no script will pick up VIRTUAL_ENV, but when you run a script, you need the virtualenv to be activated for this to work. The original PEP said nothing about a PATH search for #!/usr/bin/env python, so I mistakenly assumed there was no way currently to write a script that worked with an activated virtualenv. When I found that the code did this, I decided it was unnecessary to add extra tests and complexity just to cater for a case that nobody was ever likely to need. Paul From vinay_sajip at yahoo.co.uk Sun Feb 15 22:00:40 2015 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 15 Feb 2015 21:00:40 +0000 (UTC) Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: Message-ID: <2064944900.6494502.1424034040621.JavaMail.yahoo@mail.yahoo.com> ----- Original Message ----- From: Paul Moore > The original PEP said nothing about a PATH search for #!/usr/bin/env > python, so I mistakenly assumed there was no way currently to write a > script that worked with an activated virtualenv. Actually the path search was added later, by you! See [1] and [2]. Regards, Vinay Sajip [1] http://bugs.python.org/issue17903 [2] https://bitbucket.org/pmoore/pylauncher/commits/87d41c5a8f0fef5ef22f09570903e7aa4c850e96 From p.f.moore at gmail.com Sun Feb 15 22:15:41 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Feb 2015 21:15:41 +0000 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: <2064944900.6494502.1424034040621.JavaMail.yahoo@mail.yahoo.com> References: <2064944900.6494502.1424034040621.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 15 February 2015 at 21:00, Vinay Sajip wrote: >> The original PEP said nothing about a PATH search for #!/usr/bin/env >> python, so I mistakenly assumed there was no way currently to write a >> script that worked with an activated virtualenv. > > > Actually the path search was added later, by you! See [1] and [2]. Coo, that was clever of me :-) I guess I'll have to admit to going senile now... Paul From rosuav at gmail.com Mon Feb 16 00:00:11 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 16 Feb 2015 10:00:11 +1100 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DDCED7.9080606@g.nevcal.com> Message-ID: On Sat, Feb 14, 2015 at 11:07 AM, Nick Coghlan wrote: > OTOH, it may be time to reconsider our recommendation to distros as well, > suggesting that for Python 3.5+, we will consider it appropriate to have the > "python" symlink refer to "python3". If *all* distros provide a "python2" symlink, then the recommendation can become "use python if it's 2/3 compatible, or python2/python3 to choose", and then it won't hurt to switch python to be python3. Are there known distros in which current versions (or, those that will be current when 3.5 is released) aren't providing "python2"? ChrisA From rosuav at gmail.com Mon Feb 16 13:29:54 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 16 Feb 2015 23:29:54 +1100 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DDCED7.9080606@g.nevcal.com> Message-ID: On Mon, Feb 16, 2015 at 10:47 PM, Petr Viktorin wrote: >> If *all* distros provide a "python2" symlink, then the recommendation >> can become "use python if it's 2/3 compatible, or python2/python3 to >> choose", and then it won't hurt to switch python to be python3. >> Are there known distros in which current versions (or, those that will be >> current when 3.5 is released) aren't providing "python2"? > > That wouldn't be a good recommendation for the long term. > Fedora probably won't drop python2 from the default installation > before 3.5 is released, but hopes are high that it'll happen when 3.5 > is still current. Not sure about others. Clarification: If all distros *that provide Python 2.x* provide a "python2" symlink, etc, etc. ChrisA From encukou at gmail.com Mon Feb 16 12:47:09 2015 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 16 Feb 2015 12:47:09 +0100 Subject: [Python-Dev] PEP 370 - per-user scripts directory on Windows In-Reply-To: References: <54DA3DFB.1010203@stoneleaf.us> <1423590351443.13167@microsoft.com> <54DCE56E.2080408@stoneleaf.us> <54DDCED7.9080606@g.nevcal.com> Message-ID: On Mon, Feb 16, 2015 at 12:00 AM, Chris Angelico wrote: > On Sat, Feb 14, 2015 at 11:07 AM, Nick Coghlan wrote: >> OTOH, it may be time to reconsider our recommendation to distros as well, >> suggesting that for Python 3.5+, we will consider it appropriate to have the >> "python" symlink refer to "python3". > > If *all* distros provide a "python2" symlink, then the recommendation > can become "use python if it's 2/3 compatible, or python2/python3 to > choose", and then it won't hurt to switch python to be python3. > Are there known distros in which current versions (or, those that will be > current when 3.5 is released) aren't providing "python2"? That wouldn't be a good recommendation for the long term. Fedora probably won't drop python2 from the default installation before 3.5 is released, but hopes are high that it'll happen when 3.5 is still current. Not sure about others. From blaxxton at yahoo.com Mon Feb 16 16:20:29 2015 From: blaxxton at yahoo.com (Blaxton) Date: Mon, 16 Feb 2015 15:20:29 +0000 (UTC) Subject: [Python-Dev] Python 3.4 RPM on AIX Message-ID: <1657723749.7203522.1424100029433.JavaMail.yahoo@mail.yahoo.com> hi I am having problem building Python RPM on AIX and I think there are several bugs in Python .spec files. README file says download the bz2 version, but only gziped and xz version available from Python website so had to convert the .tgz file to bz2 and copy it over to SOURCES directory. %define config_binsuffix? was set to 2.6 and not sure why it should be 2.6 when I am working on 3.4 version. had to change it to to pass the error. %define config_binsuffix none Misc/cheatsheet seems to be decommissioned in new version of Python but still in spec file so commented it. The last error that I am receiving and can't figure the resolution is belowwhy it won't create the required direcotory ?Processing files: python-devel-3.4.2-AIX71 File not found by glob: /var/opt/freeware/tmp/python-3.4.2-root/usr/include/python3.4/*.h File not found: /var/opt/freeware/tmp/python-3.4.2-root/usr/lib/python3.4/config PreReq: rpmlib(VersionedDependencies) <= 3.0.3-1 python = 3.4.2 Processing files: python-tools-3.4.2-AIX71 File not found: /var/opt/freeware/tmp/python-3.4.2-root/usr/bin/pydoc File not found: /var/opt/freeware/tmp/python-3.4.2-root/usr/bin/smtpd.py PreReq: rpmlib(VersionedDependencies) <= 3.0.3-1 python = 3.4.2-AIX71 running rpm -bl option to see missing files are below: rpm -bl /opt/freeware/src/packages/SPECS/python-3.4.spec Processing files: python-3.4.2-AIX71 File not found by glob: /var/opt/freeware/tmp/python-3.4.2-root/opt/freeware/man/man1/python.1* File not found: /var/opt/freeware/tmp/python-3.4.2-root/usr/include/python3.4 Processing files: python-devel-3.4.2-AIX71 File not found by glob: /var/opt/freeware/tmp/python-3.4.2-root/usr/include/python3.4/*.h File not found: /var/opt/freeware/tmp/python-3.4.2-root/usr/lib/python3.4/config PreReq: rpmlib(VersionedDependencies) <= 3.0.3-1 python = 3.4.2 Processing files: python-tools-3.4.2-AIX71 File not found: /var/opt/freeware/tmp/python-3.4.2-root/usr/bin/pydoc File not found: /var/opt/freeware/tmp/python-3.4.2-root/usr/bin/smtpd.py PreReq: rpmlib(VersionedDependencies) <= 3.0.3-1 python = 3.4.2-AIX71 Any idea as how I can fix this ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Feb 16 17:16:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 16 Feb 2015 16:16:33 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 18:25, Petr Viktorin wrote: > On POSIXy systems the "python3" symlink is created in all venvs. I > thought (perhaps na?vely) that Windows doesn't do shebangs natively, > so there's some Python-specific mechanism around them, which should > handle "python3". Windows doesn't have "python2" or "python3" commands, just "python". To choose which version you use, set PATH or use an explicit path to the executable. The launcher offers shebang handling for scripts, and special-cases the following values for Unix compatibility: #!/usr/bin/env python #!/usr/bin/python #!/usr/local/bin/python They default to the "default Python" which is normally the latest Python 2 version available, but which can be configured by the user. They also support adding an explicit version (python2, python3, python2.7, python3.4, ...) As far as I know, this is *identical* behaviour to Unix - even to the incredibly annoying (to me) choice of Python 2 as a default. So I reconfigure the default in my personal settings to Python 3. Unix users can do this too (although it may involve a symlink in a ~/bin directory rather than an ini file change). I see no reason for the zipapp module to buck the trend that the default Python is "the latest 2.X version unless the user configures it differently themselves". Hence the default of "#!/usr/bin/env python". Certainly that means that users who use the default have to be prepared for the fact that they are subject to user configuration changes, and yes that includes running under either of Python 2 or 3. If you don't like that, force a particular version with -p, and be prepared for the *other* issues this may cause (user doesn't have that version, for example). The alternative, I guess, is to have *no* default and write no shebang unless -p is specified. That matches how .py files work (you have to type the shebang line in if you want it) and works the same way (on Unix, a file without a shebang must be run with an explicit "python" command, and on Windows it uses the default Python via the file type association). Honestly, I'd be happy with this - the Windows experience is identical, and it saves choosing something inappropriate ("in the face of ambiguity, refuse the temptation to guess" :-)) So, the options I see: 1. Stick with /usr/bin/env python 2. No shebang unless -p is specified 3. Unix users come up with a solution which is the same as the above for Windows users, but which suits them better. I don't see changing the Windows behaviour as an option. If nobody comes up with anything for (3) I'll take the consensus between (1) or (2). If there's no consensus, I'll pick something. Probably 1, as why change if nobody has a better idea? One final thought. This issue is no different from shebang lines in .py scripts - the only real difference is that it's easy to see and edit the shebang in a .py script. It may be that what's really needed here is an option for the user to display and update the shebang line in a pyz archive. I'll look at adding that to the PEP, regardless of the conclusion to this debate. Paul From ethan at stoneleaf.us Mon Feb 16 17:44:43 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 16 Feb 2015 08:44:43 -0800 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <54E21E7B.30206@stoneleaf.us> On 02/16/2015 08:16 AM, Paul Moore wrote: > The alternative, I guess, is to have *no* default and write no shebang > unless -p is specified. This seems to make sense. > So, the options I see: > > 1. Stick with /usr/bin/env python > 2. No shebang unless -p is specified > 3. Unix users come up with a solution which is the same as the above > for Windows users, but which suits them better. +1 on 2 +0.5 on 1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Mon Feb 16 17:42:04 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 16 Feb 2015 16:42:04 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 16 February 2015 at 16:34, Steve Dower wrote: >> As far as I know, this is *identical* behaviour to Unix - even to the incredibly >> annoying (to me) choice of Python 2 as a default. So I reconfigure the default >> in my personal settings to Python 3. Unix users can do this too (although it may >> involve a symlink in a ~/bin directory rather than an ini file change). > > We could also add special-cases for "#!/usr/...python3" in the launcher on Windows. The launcher handles that. It runs the same thing as "py -3" runs. Which may not be the same as what "py" runs (in my case it runs 3.5a1 where py runs 3.4). Arguably that's an odd choice, but it's simply that I only use "py" in the normal course of events so I only set the "python" default, not the "python3" default. My point is that on Windows, users typically don't change the executable name they use[1], but rather configure the "python" (or "py") command to do what they want. So I think that on Windows we should follow that convention and execute whatever "python"/"py" execute. Paul [1] As usual with anything like this, it's hard to get a sense of what's "typical" so if hordes of Windows users suddenly post saying they routinely use "python2" and "python3" commands, I'll happily concede I'm not the norm here and ask someone to step up and document recommended practices on Windows better, and I'll update the PEP to follow them :-) From marky1991 at gmail.com Mon Feb 16 18:00:52 2015 From: marky1991 at gmail.com (Mark Young) Date: Mon, 16 Feb 2015 12:00:52 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: I don't know what anyone else does, but in cases where I have both on my windows box, I do use python2(.x) and python3(.y) . If I only have one version on the box, I use the generic name of course. (I don't often have only one version on my boxes though. 2.x inevitably gets drug in in for some reason or another and I hardly ever uninstall old versions of 3.x) I don't use the launcher though, so I might be out-of-scope entirely. (in which case, sorry for the noise) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Mon Feb 16 17:34:12 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Mon, 16 Feb 2015 16:34:12 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: Paul Moore wrote: > On 15 February 2015 at 18:25, Petr Viktorin wrote: >> On POSIXy systems the "python3" symlink is created in all venvs. I >> thought (perhaps na?vely) that Windows doesn't do shebangs natively, >> so there's some Python-specific mechanism around them, which should >> handle "python3". > > Windows doesn't have "python2" or "python3" commands, just "python". > To choose which version you use, set PATH or use an explicit path to the > executable. > > The launcher offers shebang handling for scripts, and special-cases the > following values for Unix compatibility: > > #!/usr/bin/env python > #!/usr/bin/python > #!/usr/local/bin/python > > They default to the "default Python" which is normally the latest Python 2 > version available, but which can be configured by the user. > They also support adding an explicit version (python2, python3, python2.7, > python3.4, ...) > > As far as I know, this is *identical* behaviour to Unix - even to the incredibly > annoying (to me) choice of Python 2 as a default. So I reconfigure the default > in my personal settings to Python 3. Unix users can do this too (although it may > involve a symlink in a ~/bin directory rather than an ini file change). We could also add special-cases for "#!/usr/...python3" in the launcher on Windows. Cheers, Steve From p.f.moore at gmail.com Mon Feb 16 18:10:17 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 16 Feb 2015 17:10:17 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 16 February 2015 at 16:42, Paul Moore wrote: > My point is that on Windows, users typically don't change the > executable name they use[1], but rather configure the "python" (or > "py") command to do what they want. So I think that on Windows we > should follow that convention and execute whatever "python"/"py" > execute. One other thought. We could add "short form" options for -p to the zipapp command: -p X[.Y] Use "#!/usr/bin/env pythonX[.Y]" as the shebang. I'd definitely want the default to be "python" in that case, though, as it's the one you can't specify in a short form otherwise. Paul From p.f.moore at gmail.com Mon Feb 16 18:11:46 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 16 Feb 2015 17:11:46 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 16 February 2015 at 17:00, Mark Young wrote: > I don't know what anyone else does, but in cases where I have both on my > windows box, I do use python2(.x) and python3(.y) . If I only have one > version on the box, I use the generic name of course. (I don't often have > only one version on my boxes though. 2.x inevitably gets drug in in for some > reason or another and I hardly ever uninstall old versions of 3.x) I don't > use the launcher though, so I might be out-of-scope entirely. (in which > case, sorry for the noise) No, that's good to know. One clarification, though - if you only have Python 3, I get the impression you use "python" and not "python3" then...? Paul From marky1991 at gmail.com Mon Feb 16 18:14:22 2015 From: marky1991 at gmail.com (Mark Young) Date: Mon, 16 Feb 2015 12:14:22 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: If I only have one version on my box, yes, I only use "python". But like I said, for me personally, that situation doesn't last very long. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Mon Feb 16 17:34:24 2015 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 16 Feb 2015 17:34:24 +0100 Subject: [Python-Dev] [Python-checkins] cpython (3.4): Issue #23446: Use PyMem_New instead of PyMem_Malloc to avoid possible integer In-Reply-To: <20150216113449.60738.84196@psf.io> References: <20150216113449.60738.84196@psf.io> Message-ID: <20150216163424.GA3056@bytereef.org> On Mon, Feb 16, 2015 at 11:35:52AM +0000, serhiy.storchaka wrote: > diff --git a/Modules/_testbuffer.c b/Modules/_testbuffer.c > --- a/Modules/_testbuffer.c > +++ b/Modules/_testbuffer.c > @@ -850,7 +850,7 @@ > Py_ssize_t *dest; > Py_ssize_t x, i; > > - dest = PyMem_Malloc(len * (sizeof *dest)); > + dest = PyMem_New(Py_ssize_t, len); > if (dest == NULL) { > PyErr_NoMemory(); > return NULL; This, too, was already protected by len == ndim <= 64. Stefan Krah From greg.ewing at canterbury.ac.nz Mon Feb 16 21:40:43 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 Feb 2015 09:40:43 +1300 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <54E255CB.3000505@canterbury.ac.nz> Paul Moore wrote: > The alternative, I guess, is to have *no* default and write no shebang > unless -p is specified. +1. That sounds like a very good idea to me. -- Greg From victor.stinner at gmail.com Mon Feb 16 22:14:59 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 16 Feb 2015 22:14:59 +0100 Subject: [Python-Dev] [Python-checkins] cpython (3.4): Issue #23446: Use PyMem_New instead of PyMem_Malloc to avoid possible integer In-Reply-To: <20150216163424.GA3056@bytereef.org> References: <20150216113449.60738.84196@psf.io> <20150216163424.GA3056@bytereef.org> Message-ID: 2015-02-16 17:34 GMT+01:00 Stefan Krah : > > On Mon, Feb 16, 2015 at 11:35:52AM +0000, serhiy.storchaka wrote: >> diff --git a/Modules/_testbuffer.c b/Modules/_testbuffer.c >> --- a/Modules/_testbuffer.c >> +++ b/Modules/_testbuffer.c >> @@ -850,7 +850,7 @@ >> Py_ssize_t *dest; >> Py_ssize_t x, i; >> >> - dest = PyMem_Malloc(len * (sizeof *dest)); >> + dest = PyMem_New(Py_ssize_t, len); >> if (dest == NULL) { >> PyErr_NoMemory(); >> return NULL; > > This, too, was already protected by len == ndim <= 64. I don't understand why you don't want to use PyMem_New() even if it cannot overflow. PyMem_New() is more readable no? Victor From p.f.moore at gmail.com Tue Feb 17 00:21:55 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 16 Feb 2015 23:21:55 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 15 February 2015 at 17:46, Daniel Holth wrote: > Go ahead, make my pep. > > I will appreciate seeing it happen. Here is a draft update for PEP 441. It's still a work in progress - in particular I want to wait for consensus on the issue of the default interpreter before finalising it. But I thought it would be worth having a full spec available for people. PEP: 441 Title: Improving Python ZIP Application Support Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth , Paul Moore Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 March 2013 Post-History: 30 March 2013, 1 April 2013, 16 February 2015 Improving Python ZIP Application Support ======================================== Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6 [1]_. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the __main__ module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules. This feature is not as popular as it should be mainly because it was not promoted as part of Python 2.6 [2]_, so that it is relatively unknown, but also because the Windows installer does not register a file extension (other than .py) for this format of file, to associate with the launcher. This PEP proposes to fix these problems by re-publicising the feature, defining the .pyz and .pyzw extensions as ?Python ZIP Applications? and ?Windowed Python ZIP Applications?, and providing some simple tooling to manage the format. A New Python ZIP Application Extension ====================================== The Python 3.5 installer will associate .pyz and .pyzw ?Python ZIP Applications? with the platform launcher so they can be executed. A .pyz archive is a console application and a .pyzw archive is a windowed application, indicating whether the console should appear when running the app. For UNIX users, .pyz applications should typically be prefixed with a #! line pointing to the correct Python interpreter and an optional explanation:: #!/usr/bin/env python3 # Python application packed with zipapp module (binary contents of archive) However, it is always possible to execute a .pyz application by supplying the filename to the Python interpreter directly. As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work. Minimal Tooling: The zipapp Module ================================== This PEP also proposes including a module for working with these archives. The module will contain functions for working with Python zip application archives, and a command line interface (via ``python -m zipapp``) for their creation and manipulation. Module Interface ---------------- The zipapp module will provide the following functions: ``pack(target, directory, interpreter=None, main=None)`` Writes an application archive called ``target``, containing the contents of ``directory``. If ``interpreter`` is specified, it will be written to the start of the archive as a shebang line and the file will be made executable (if no interpreter is specified, the shebang line will be omitted). If the directory contains no ``__main__.py`` file, the function will construct a ``__main__.py`` which calls the function specified in the ``main`` argument (which should be in the form ``"pkg.mod:fn"``). It is an error to specify ``main`` if the directory contains a ``__main__.py``, or to omit ``main`` when there is no ``__main__.py`` (as that will result in an archive which has no main function and so cannot be executed). ``get_interpreter(archive)`` Returns the interpreter specified in the shebang line of the archive. If there is no shebang, the function returns None. ``set_interpreter(archive, new_archive, interpreter=None)`` Modifies the archive's shebang line to contain the specified interpreter, and writes the updated archive to new_archive. If the interpreter is None, removes the shebang line. Command Line Usage ------------------ The zipapp module can be run with the python -m flag. The command line interface is as follows:: python -m zipapp [options] directory Create an archive from the contents of the given directory. By default, an archive will be created with the same name as the source directory, with a .pyz extension. The following options can be specified: -o archive The destination archive will have the specified name. -p interpreter The given interpreter will be written to the shebang line of the archive. If this option is not given, the archive will have no shebang line. -m pkg.mod:fn The source directory must not have a __main__.py file. The archiver will write a __main__.py file into the target which calls fn from the module pkg.mod. The behaviour of the command line interface matches that of ``zipapp.pack()``. As noted, the archives are standard zip files, and so can be unpacked using any standard ZIP utility or Python?s zipfile module. FAQ --- Are you sure a standard ZIP utility can handle #! at the beginning? Absolutely. The zipfile specification allows for arbitrary data to be prepended to a zipfile. This feature is commonly used by "self-extracting zip" programs. If your archive program can't handle this, it is a bug in your archive program. Isn?t zipapp just a very thin wrapper over the zipfile module? Yes. If you prefer to build your own Python zip application archives using other tools, they will work just as well. The zipapp module is a convenience, nothing more. Why not use just use a .zip or .py extension? Users expect a .zip file to be opened with an archive tool, and expect a .py file to contain readable text. Both would be confusing for this use case. How does this compete with existing package formats? The sdist, bdist and wheel formats are designed for packaging of modules to be installed into an existing Python installation. They are not intended to be used without installing. The executable zip format is specifically designed for standalone use, without needing to be installed. They are in effect a multi-file version of a standalone Python script. Rejected Proposals ================== Convenience Values for Shebang Lines ------------------------------------ Is it worth having "convenience" forms for any of the common interpreter values? For example, ``-p 3`` meaning the same as ``-p "/usr/bin/env python3"``. It would save a lot of typing for the common cases, as well as giving cross-platform options for people who don't want or need to understand the intricacies of shebang handling on "other" platforms. Downsides are that it's not obvious how to translate the abbreviations. For example, should "3" mean "/usr/bin/env python3", "/usr/bin/python3", "python3", or something else? Also, there is no obvious short form for the key case of "/usr/bin/env python" (any available version of Python), which could easily result in scripts being written with overly-restrictive shebang lines. Overall, this seems like there are more problems than benefits, and as a result has been dropped from consideration. Open Questions ============== Default Interpreter ------------------- The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter. Unix users have problems with this behaviour, as the default for the python command on many distributions is Python 2, and it is felt that this PEP should prefer Python 3 by default. However, using a command of ``python3`` can result in unexpected behaviour for Windows users, where the default behaviour of the launcher for the command "python" is commonly customised by users, but the behaviour of "python3" may not be modified to match. Currently, the principle "in the face of ambiguity, refuse to guess" has been invoked, and archives have no shebang line unless explicitly requested. On Windows, the archives will still be run (with the default Python) by the launcher, and on Unix, the archives can be run by explicitly invoking the desired Python interpreter. This issue is currently under active discussion on python-dev, and the results will be reflected here when consensus has been reached. Command Line Tool to Manage Shebang Lines ----------------------------------------- It is conceivable that users would want to modify the shebang line for an existing archive, or even just display the current shebang line. This is tricky to do so with existing tools (zip programs typically ignore prepended data totally, and text editors can have trouble editing files containing binary data). The zipapp module provides functions to handle the shebang line, but should these be exposed via the command line interface? At the moment, the PEP proposes *not* to provide a command line interface for these functions, as it is not clear how to provide one without the resulting interface being over-complex and potentially confusing. References ========== .. [1] ?Allow interpreter to execute a zip file? (http://bugs.python.org/issue1739468) .. [2] ?Feature is not documented? (http://bugs.python.org/issue17359) Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From stephen at xemacs.org Tue Feb 17 05:02:49 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 17 Feb 2015 13:02:49 +0900 Subject: [Python-Dev] Python 3.4 RPM on AIX In-Reply-To: <1657723749.7203522.1424100029433.JavaMail.yahoo@mail.yahoo.com> References: <1657723749.7203522.1424100029433.JavaMail.yahoo@mail.yahoo.com> Message-ID: <874mqlkwwm.fsf@uwakimon.sk.tsukuba.ac.jp> Blaxton writes: > I am having problem building Python RPM on AIX and I think there > are several bugs in Python .spec files. This list is for development *of* Python, not development *with* Python. In this case, you would probably be better off with an AIX list devoted to packaging, an RPM-specific list, or failing those two, a Red Hat/Fedora list on packaging. All of the issues you list are related to generic distro packaging issues, and not to Python itself. From blaxxton at yahoo.com Tue Feb 17 05:50:14 2015 From: blaxxton at yahoo.com (Blaxton) Date: Tue, 17 Feb 2015 04:50:14 +0000 (UTC) Subject: [Python-Dev] Python 3.4 RPM on AIX In-Reply-To: <874mqlkwwm.fsf@uwakimon.sk.tsukuba.ac.jp> References: <874mqlkwwm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <753109360.8384872.1424148614550.JavaMail.yahoo@mail.yahoo.com> I am using the spec file that comes with Python source code which downloaded from python.org website source file is set on spec file to file with bz2 format while there is only .xz and zipped are available to download. I thought somehow some one from development team has wrote that .spec fileand maybe supporting it. This is like no support. From: Stephen J. Turnbull To: Blaxton Cc: "python-dev at python.org" Sent: Monday, February 16, 2015 10:02 PM Subject: [Python-Dev] Python 3.4 RPM on AIX Blaxton writes: > I am having problem building Python RPM on AIX and I think there > are several bugs in Python .spec files. This list is for development *of* Python, not development *with* Python.? In this case, you would probably be better off with an AIX list devoted to packaging, an RPM-specific list, or failing those two, a Red Hat/Fedora list on packaging.? All of the issues you list are related to generic distro packaging issues, and not to Python itself. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Tue Feb 17 06:54:57 2015 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Mon, 16 Feb 2015 23:54:57 -0600 Subject: [Python-Dev] Python 3.4 RPM on AIX In-Reply-To: <753109360.8384872.1424148614550.JavaMail.yahoo@mail.yahoo.com> References: <874mqlkwwm.fsf@uwakimon.sk.tsukuba.ac.jp> <753109360.8384872.1424148614550.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Mon, Feb 16, 2015 at 10:50 PM, Blaxton wrote: > I am using the spec file that comes with Python source code which downloaded > from python.org website > source file is set on spec file to file with bz2 format while there is only > .xz and zipped are available to download. > > > I thought somehow some one from development team has wrote that .spec file > and maybe supporting it. > > This is like no support. The spec file in Misc/RPM has not been maintained in quite some time; the last change to that file that wasn't just a version bump was one [1] whose commit message was "#5776: fix mistakes in python specfile. (Nobody probably uses it anyway.)" and was committed four and a half years ago. It's not unlikely that you're the first person to use the file since then :). We should probably just remove the file, but if you manage to get it working and would like to provide a patch on bugs.python.org, it would probably be accepted. [1] https://hg.python.org/cpython/rev/ef75ecd0e1a7 -- Zach From stephen at xemacs.org Tue Feb 17 09:14:27 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 17 Feb 2015 17:14:27 +0900 Subject: [Python-Dev] Python 3.4 RPM on AIX In-Reply-To: <753109360.8384872.1424148614550.JavaMail.yahoo@mail.yahoo.com> References: <874mqlkwwm.fsf@uwakimon.sk.tsukuba.ac.jp> <753109360.8384872.1424148614550.JavaMail.yahoo@mail.yahoo.com> Message-ID: <87twylj6os.fsf@uwakimon.sk.tsukuba.ac.jp> Blaxton writes: > I am using the spec file that comes with Python source code which > downloaded from python.org website Ah, sorry, I didn't realize that. My advice still stands; while there are Red Hat/Fedora/other-RPM-based distro workers here, I don't know if any of them actually have anything to do with RPM-building. And I've been following this list for over 10 years -- this is the first I've heard of problems with the .spec in the Python sources, so I suspect Zach is right: you're the first person to use the thing in half a decade. The Python project itself doesn't distribute RPMs as far as I can see. > This is like no support. Unfortunately, yes. I know I don't have the knowledge to help fix the .spec file, and I don't even know who to tell you to ask in the project. From bkabrda at redhat.com Tue Feb 17 09:33:51 2015 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Tue, 17 Feb 2015 03:33:51 -0500 (EST) Subject: [Python-Dev] Python 3.4 RPM on AIX In-Reply-To: <87twylj6os.fsf@uwakimon.sk.tsukuba.ac.jp> References: <874mqlkwwm.fsf@uwakimon.sk.tsukuba.ac.jp> <753109360.8384872.1424148614550.JavaMail.yahoo@mail.yahoo.com> <87twylj6os.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <902237152.16634682.1424162031187.JavaMail.zimbra@redhat.com> ----- Original Message ----- > Blaxton writes: > > > I am using the spec file that comes with Python source code which > > downloaded from python.org website > > Ah, sorry, I didn't realize that. My advice still stands; while there > are Red Hat/Fedora/other-RPM-based distro workers here, I don't know > if any of them actually have anything to do with RPM-building. And > I've been following this list for over 10 years -- this is the first > I've heard of problems with the .spec in the Python sources, so I > suspect Zach is right: you're the first person to use the thing in > half a decade. The Python project itself doesn't distribute RPMs as > far as I can see. Yes, there are people here who do everything around distro RPM-building here :) AFAIK no RPM based distribution uses the specfile from the Python tarball. Our specfiles (Fedora/Red Hat Enterprise Linux) are much more complex to satisfy all the downstream policies and such. In case you're interested, you can have a look at our current python3.spec [1]. Feel free to reuse it (all Fedora specfiles are licensed under MIT license unless noted otherwise). > > This is like no support. > > Unfortunately, yes. I know I don't have the knowledge to help fix the > .spec file, and I don't even know who to tell you to ask in the project. I can help with RPM related questions (it'd probably be best to ask them privately, since their offtopic for this list), but I have pretty much zero knowledge of AIX - so I won't be able to help you with AIX-specific issues. -- Regards, Slavek Kabrda [1] http://pkgs.fedoraproject.org/cgit/python3.git/tree/python3.spec From barry at python.org Tue Feb 17 19:44:49 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 13:44:49 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <20150217134449.28fa4592@anarchist.wooz.org> I'm just now catching up on this thread, so hopefully these thoughts are still relevant. IIRC, the PEP has been scaled back to mostly a convenience around packing an existing directory into a .pyz file, and (on Windows) adding an association for those as executable Python zip file. To the extent that I care about Windows, +1. I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible. The broader question of pex, pyzaa, etc. is an important one for Python, IMHO. Having a standard single-executable distribution story will help Python continue to compete on platforms that work beyond the distribution models we've come to think about. App models, and transactional systems such as Ubuntu Core, etc. are gaining mindshare and Python's traditional way of deploying applications doesn't really fit very well, and that may discourage developers from using Python. Whether Python itself wants to put a stake in the ground today for that is an open question, but adding support in Python for extension modules in zips (whether via save-to-disk or new operating system APIs) would be useful to explore. I happen to like pex, but what I really want is something like: $ pyvenv foo $ source foo/bin/activate $ pip install coolthing $ pyzip -o coolthing.pyz . $ deactivate $ ./coolthing.pyz This is outside the scope of PEP 441, but if anybody at Pycon wants to explore this further, I'm in. Cheers, -Barry From barry at python.org Tue Feb 17 19:47:42 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 13:47:42 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <20150217134742.22c474d5@anarchist.wooz.org> On Feb 15, 2015, at 05:15 PM, Petr Viktorin wrote: >So I'd suggest `/usr/bin/env python3` for the default. I really don't want `/usr/bin/env anything`. In general, using /usr/bin/env is a fine tool for developers, but a terrible tool for deployment. It's just too easy for end users to break an application by mucking with their $PATH. In Debian/Ubuntu, we don't allow these shebangs in system scripts, and in fact the package build tools will rewrite shebangs to avoid them. Cheers, -Barry From barry at python.org Tue Feb 17 19:52:56 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 13:52:56 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <20150217135256.1512523e@anarchist.wooz.org> On Feb 16, 2015, at 04:16 PM, Paul Moore wrote: >So, the options I see: > >1. Stick with /usr/bin/env python >2. No shebang unless -p is specified >3. Unix users come up with a solution which is the same as the above >for Windows users, but which suits them better. #2 seems to me to be the most reasonable alternative. The resulting pyz files (built w/o -p) would still be explicitly executable, just like .py files. However, -p must be able to accept any number of strings, including "/usr/bin/env python3" if the user wants that. Probably the best thing to do (on *nix at least) is, if the path is absolute, use the given string verbatim. If the path is relative, search for the given executable on $PATH and use the first one found. If nothing is found, use what's given explicitly. I think that will give all the reasonable use cases. Cheers, -Barry From barry at python.org Tue Feb 17 19:56:48 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 13:56:48 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <20150217135648.1f494957@anarchist.wooz.org> On Feb 16, 2015, at 11:21 PM, Paul Moore wrote: >Default Interpreter >------------------- > >The initial draft of this PEP proposed using ``/usr/bin/env python`` >as the default interpreter. The other reasonable alternative for a default shebang is sys.executable. Cheers, -Barry From barry at python.org Tue Feb 17 20:00:38 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 14:00:38 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <20150217140038.73a487ae@anarchist.wooz.org> On Feb 15, 2015, at 05:21 PM, Thomas Wouters wrote: >FWIW, we have a similar tool to 'pyzaa' at Google, although a lot older (it >has a pure-python implementation of zipimport, because it had to work with >Python 2.2 back in the day) that *does* support extension modules and other >shared libraries (by extracting them to disk on program startup.) We're >also working to replace that by loading the extension modules directly from >the archive (using dlopen_with_offset as described by ppluzhnikov in >https://sourceware.org/bugzilla/show_bug.cgi?id=11767, although it should >also be possible using dlopen_ehdr/dlopen_phdr), which requires that >extension modules are stored uncompressed (simple) and page-aligned >(harder, as the zipfile.ZipFile class doesn't directly support >page-aligning anything, but it turns out it's easy to hack around by >overriding _writecheck :P) And yes, we really are in a position to modify >glibc to make our life distributing Python applications easier, internally; >the old code involves shell and Python bootstrap code that tries very hard >to avoid race conditions and security problems, which is a pain to maintain. So, let's add this to zipimport so we can remove an unfortunate limitation and avoid everyone having to reinvent the wheel! Oh, and the dlopen extension sound pretty cool. Can't wait until Google gets that into glibc so us mere mortals can use it. Seems reasonable to add this to zipimport too, with whatever feature guards are necessary. You'll want to give your future self an opportunity to trot out the keys to the time machine. :) >It might be nice to consider those use-cases in pyzapp as well, especially >once the glibc feature is released. It requires some fairly big changes to >zipimport (I ended up rewriting the whole thing) but we can easily >opensource the code, if there's any interest at all. +1 >I'd be happy to discuss this in more detail at PyCon, at which time we should >be deploying code internally using all of this. +1 Cheers, -Barry From p.f.moore at gmail.com Tue Feb 17 21:44:58 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 17 Feb 2015 20:44:58 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217134449.28fa4592@anarchist.wooz.org> References: <20150217134449.28fa4592@anarchist.wooz.org> Message-ID: On 17 February 2015 at 18:44, Barry Warsaw wrote: > The broader question of pex, pyzaa, etc. is an important one for Python, > IMHO. Having a standard single-executable distribution story will help Python > continue to compete on platforms that work beyond the distribution models > we've come to think about. App models, and transactional systems such as > Ubuntu Core, etc. are gaining mindshare and Python's traditional way of > deploying applications doesn't really fit very well, and that may discourage > developers from using Python. > > Whether Python itself wants to put a stake in the ground today for that is an > open question, but adding support in Python for extension modules in zips > (whether via save-to-disk or new operating system APIs) would be useful to > explore. > > I happen to like pex, but what I really want is something like: > > $ pyvenv foo > $ source foo/bin/activate > $ pip install coolthing > $ pyzip -o coolthing.pyz . > $ deactivate > $ ./coolthing.pyz > > This is outside the scope of PEP 441, but if anybody at Pycon wants to explore > this further, I'm in. Barry, thanks for your comments. I'll pick up on the ones directly related to the PEP separately, but I'd like to add some thoughts on this. (I'm not at Pycon so dumping stuff in an email is the best I can do :-)) I'd like to see a good single-file bundled application format for Python (on Windows, there's py2exe, which is fantastic, but often I simply don't *want* to bundle Python and the stdlib with my code). The executable zip format is that solution, IMO. It's been around for ages, and has core support. There are certain things that are needed to make a compelling story, though. And it's definitely not a well known feature, which I hope is something PEP 441 will address, at least a bit. Handling of C extensions in zipfiles isn't so much an executable zip issue as a zipimport one. It would be nice to see something like this, but I would much rather it be an update to zipimport than be bolted on in whichever application does it. To my mind, having the core support the mechanisms is a much more robust approach (and avoids the possibility of the solution not being implemented cross-platform - look at py2exe which solves the extensions in a zipfile issue, but only for Windows). Also, modules still fail to be zip-safe for other reasons, such as needing data files to be real filesystem files (for example requests' certificate bundle). Adding more support for packages to be zip-safe would be fantastic, and would help in a lot more places than just the executable zip format. (Although promoting the executable zip format would help persuade projects to *use* those features rather than assuming a filesystem). Donald Stufft has a proposal on resource APIs for importlib that would help with this. As far as issues specific to the executable zip format are concerned, the main one is tool support. The zipapp module is intended as nothing more than the bare minimum. I think that more complex tools should live on PyPI, not in the core, for all the same reasons that pip was developed outside of the core - the ideal tool needs time to develop, needs a faster release cycle than core Python, etc, etc. At the moment, the only substantial tools I know of are pyzzer and pex. pyzzer is pretty good, IMO, but doesn't do much beyond the basic zip building. I tried pex, but it didn't work on Windows at the time - the problem I found is fixed, but I haven't really had the inclination to try again (there's only so many projects I can cope with being the Windows early adopter for :-)) One thing I would like to see, in the light of PEP 441, would be for pex to switch to using ".pyz" as its file extension, because that's what pex files are. (The documentation also refers to the support of __main__.py as "a quirk in the Python importer" which I find particularly irritating, as it's not a quirk, it's a deliberate feature implemented for precisely this sort of use case!!!) I agree that something that grabs all of the non-stdlib dependencies of a package and bundles them up automatically would be brilliant. I've intended to write something like that for a while now, but haven't had the time. But I certainly think your "what I really want" scenario above is entirely achievable. At least to the extent that the package code and its dependencies support running from a non-filesystem importer... A final thought - I wonder how easy it would be to write an "unpacking zipimporter" that worked like zipimport, but unpacked the zipfile to a temporary directory at the start and cleaned up on program termination? It would give the single-file benefits of zipimport, but would be guaranteed 100% compatible with filesystem imports, at a cost of worse startup/teardown performance, and extra disk space. It might make a nice fallback importer for troublesome packages. Paul From p.f.moore at gmail.com Tue Feb 17 21:52:50 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 17 Feb 2015 20:52:50 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217135256.1512523e@anarchist.wooz.org> References: <20150217135256.1512523e@anarchist.wooz.org> Message-ID: On 17 February 2015 at 18:52, Barry Warsaw wrote: >>So, the options I see: >> >>1. Stick with /usr/bin/env python >>2. No shebang unless -p is specified >>3. Unix users come up with a solution which is the same as the above >>for Windows users, but which suits them better. > > #2 seems to me to be the most reasonable alternative. The resulting pyz files > (built w/o -p) would still be explicitly executable, just like .py files. I'm pretty sure that's the way the general feeling is going. > However, -p must be able to accept any number of strings, including > "/usr/bin/env python3" if the user wants that. The code simply writes '#!{}\n'.format(p_option).encode(sys.filesystemencoding()) to the file, so you can put whatever you want in. Given that it isn't the name of the Python executable, maybe the option should be --interpreter instead? Oh, and am I right that the shebang line should be encoded using the filesystem encoding on Unix? I know 99.999% of use cases will be ascii, but someone could have a Python interpreter in /home/l?on/.local/bin/python... > Probably the best thing to do (on *nix at least) is, if the path is absolute, > use the given string verbatim. If the path is relative, search for the given > executable on $PATH and use the first one found. If nothing is found, use > what's given explicitly. I'm not quite sure what you mean here, but maybe you're thinking that the -p option is the executable name rather than what gets put in the #! line directly? Let me know if it's not covered by what I've already said. Paul From p.f.moore at gmail.com Tue Feb 17 21:53:52 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 17 Feb 2015 20:53:52 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217135648.1f494957@anarchist.wooz.org> References: <20150217135648.1f494957@anarchist.wooz.org> Message-ID: On 17 February 2015 at 18:56, Barry Warsaw wrote: >>The initial draft of this PEP proposed using ``/usr/bin/env python`` >>as the default interpreter. > > The other reasonable alternative for a default shebang is sys.executable. That's instantly non-portable. On my PC, it'd give C:\Apps\Python34\python.exe. Which won't work on Unix, or on PCs that use the standard install location, or when I upgrade to Python 3.5... Paul From p.f.moore at gmail.com Tue Feb 17 21:58:12 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 17 Feb 2015 20:58:12 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217140038.73a487ae@anarchist.wooz.org> References: <20150217140038.73a487ae@anarchist.wooz.org> Message-ID: On 17 February 2015 at 19:00, Barry Warsaw wrote: >>It might be nice to consider those use-cases in pyzapp as well, especially >>once the glibc feature is released. It requires some fairly big changes to >>zipimport (I ended up rewriting the whole thing) but we can easily >>opensource the code, if there's any interest at all. > > +1 +1 from me also. Anything that makes it easier for packages to work under zipimport. But I would say that anything that is added to zipimport should be cross-platform. Having support for C extensions in zipimport on Unix only will just add another way in which Python applications can inadvertantly be non-portable... (It should be possible to support both Windows and Unix, though, as py2exe has the Windows side of things covered so presumably the code used in py2exe could be ported to zipimport). Paul From barry at python.org Tue Feb 17 22:10:12 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 16:10:12 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <20150217140038.73a487ae@anarchist.wooz.org> Message-ID: <20150217161012.4d03aacc@anarchist.wooz.org> On Feb 17, 2015, at 08:58 PM, Paul Moore wrote: >But I would say that anything that is added to zipimport should be >cross-platform. Having support for C extensions in zipimport on Unix >only will just add another way in which Python applications can >inadvertantly be non-portable... (It should be possible to support >both Windows and Unix, though, as py2exe has the Windows side of >things covered so presumably the code used in py2exe could be ported >to zipimport). The actual "import an extension module" support in zipimport does need to be cross-platform, but it can work differently depending on the platform. For example, if extended-dlopen is available on your Linux machine, zipimport could just use that. For a traditional-dlopen-only machine, it would copy-to-filesystem. It would do whatever Windowsy thing makes sense over there too. I'm much less concerned about the cross-platform portability of the resulting pyz files. There's lots of reasons why an application built in (or for) my Linux machine might not work properly on your Windows machine, but FWIW if it were the only problem, doing a native rebuild on the target platform (or via supported cross-building) would be fine with me. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Tue Feb 17 22:13:46 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 16:13:46 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <20150217135648.1f494957@anarchist.wooz.org> Message-ID: <20150217161346.6fc81fa1@anarchist.wooz.org> On Feb 17, 2015, at 08:53 PM, Paul Moore wrote: >On 17 February 2015 at 18:56, Barry Warsaw wrote: >>>The initial draft of this PEP proposed using ``/usr/bin/env python`` >>>as the default interpreter. >> >> The other reasonable alternative for a default shebang is sys.executable. > >That's instantly non-portable. On my PC, it'd give >C:\Apps\Python34\python.exe. Which won't work on Unix, or on PCs that >use the standard install location, or when I upgrade to Python 3.5... But it doesn't matter (at least to me). The scenarios I want to support don't include building a pyz on my Linux machine, giving it to you, and having you able to run it natively on your Windows machine. There are lots of interesting use cases we could still support. I could build a single-file application and deploy it into my OS's system archives. I could deploy that thing into my platform's app store. I could hand it over to a colleague running an identical version of the OS. I could deploy it into my data center. I could install it on my phone. Don't discount sys.executable to quickly. :) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Tue Feb 17 22:19:20 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 17 Feb 2015 21:19:20 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217134449.28fa4592@anarchist.wooz.org> References: <20150217134449.28fa4592@anarchist.wooz.org> Message-ID: On 17 February 2015 at 18:44, Barry Warsaw wrote: > I don't know exactly what the procedure would be to claim .pyz for *nix, > e.g. updating /etc/mime.types, but I think the PEP should at least mention > this. I think we want to get as official support for .pyz files on *nix as > possible. I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately. But absolutely, yes, the intention is that ".pyz" is officially the standard extension for Python zip applications. (And for that matter, I'd like the PEP to promote the specific term "Python Zip Application" as the correct way to refer to these things - without a concrete term to use, it's very hard for the idea to get traction). Paul From barry at python.org Tue Feb 17 22:25:28 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 16:25:28 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <20150217135256.1512523e@anarchist.wooz.org> Message-ID: <20150217162528.04da71df@anarchist.wooz.org> On Feb 17, 2015, at 08:52 PM, Paul Moore wrote: >I'm pretty sure that's the way the general feeling is going. Though the more I think about it, the more I like sys.executable. :) >> However, -p must be able to accept any number of strings, including >> "/usr/bin/env python3" if the user wants that. > >The code simply writes >'#!{}\n'.format(p_option).encode(sys.filesystemencoding()) to the file, so >you can put whatever you want in. Given that it isn't the name of the Python >executable, maybe the option should be --interpreter instead? (Quoting out of order.) >I'm not quite sure what you mean here, but maybe you're thinking that >the -p option is the executable name rather than what gets put in the >#! line directly? Let me know if it's not covered by what I've already >said. I was thinking it would be useful to mimic virtualenv's -p/--python option, but I think that doesn't actually do the $PATH parsing, so maybe taking -p verbatim is fine. >Oh, and am I right that the shebang line should be encoded using the >filesystem encoding on Unix? I know 99.999% of use cases will be >ascii, but someone could have a Python interpreter in >/home/l?on/.local/bin/python... Well, actually probably utf-8 in most cases, at least for Python 3 on *nix. I'm not sure sys.getfilesystemencoding() is the right encoding, rather than sys.getdefaultencoding(), if you're talking about the encoding of the shebang line rather than the encoding of the resulting pyz filename. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Tue Feb 17 22:25:38 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 17 Feb 2015 13:25:38 -0800 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217161346.6fc81fa1@anarchist.wooz.org> References: <20150217135648.1f494957@anarchist.wooz.org> <20150217161346.6fc81fa1@anarchist.wooz.org> Message-ID: <54E3B1D2.3030402@stoneleaf.us> On 02/17/2015 01:13 PM, Barry Warsaw wrote: > There are lots of interesting use cases we could still support. I could build > a single-file application and deploy it into my OS's system archives. I could > deploy that thing into my platform's app store. I could hand it over to a > colleague running an identical version of the OS. I could deploy it into my > data center. I could install it on my phone. All those locations have Python installed in the same place? Even your phone?? > Don't discount sys.executable too quickly. :) I suppose it could be an option. ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From v+python at g.nevcal.com Tue Feb 17 22:20:19 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 17 Feb 2015 13:20:19 -0800 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217135256.1512523e@anarchist.wooz.org> References: <20150217135256.1512523e@anarchist.wooz.org> Message-ID: <54E3B093.6050804@g.nevcal.com> On 2/17/2015 10:52 AM, Barry Warsaw wrote: > Probably the best thing to do (on *nix at least) is, if the path is absolute, > use the given string verbatim. If the path is relative, search for the given > executable on $PATH and use the first one found. If nothing is found, use > what's given explicitly. I think searching would be inappropriate. Just use what is given, as a string. If the user wants a search, they can use `which`. Otherwise, setting to a relative path reliably would be hard (and sometimes desirable). -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Feb 17 22:40:19 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 16:40:19 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <20150217134449.28fa4592@anarchist.wooz.org> Message-ID: <20150217164019.2423bb37@anarchist.wooz.org> On Feb 17, 2015, at 08:44 PM, Paul Moore wrote: >I'd like to see a good single-file bundled application format for >Python (on Windows, there's py2exe, which is fantastic, but often I >simply don't *want* to bundle Python and the stdlib with my code). I agree, for deployment to environments where you can guarantee that the basic Python infrastructure is available. It may not be though, and perhaps we can relegate that to py2exe, cx_freeze, and friends, although I would say that having such support built into Python would be very cool, and of high value to our users. That's not to play down the usefulness of good executable zip support, pared of the stdlib and interpreter. OTOH, it would be nice to think ahead, where the same tool could generate both types of single-file executables. >Handling of C extensions in zipfiles isn't so much an executable zip >issue as a zipimport one. Agreed. >(Although promoting the executable zip format would help persuade projects to >*use* those features rather than assuming a filesystem). Donald Stufft has a >proposal on resource APIs for importlib that would help with this. Yep, which I support. :) pkg_resources.resource_filename() is the way to do that today and it abstracts away all the nastiness. Mostly agree with your other comments, though I haven't played with pyzzer and pex is quite nice, for the pared down use case, although it's not quite integrated with virtualenv (I've had some discussions with pex's author about that). OTOH, I think it *would* be useful to have this built into Python. We almost have too many choices, and may people I've talked to lately (including experienced Pythonistas) have trouble choosing the right tool. But that can come later; let's get some basic support into Python first, and a good start IMHO is executable zips (including support for extension modules) with the pkg_resources/importlib API additions. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Tue Feb 17 22:44:33 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 16:44:33 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54E3B1D2.3030402@stoneleaf.us> References: <20150217135648.1f494957@anarchist.wooz.org> <20150217161346.6fc81fa1@anarchist.wooz.org> <54E3B1D2.3030402@stoneleaf.us> Message-ID: <20150217164433.17c842e5@anarchist.wooz.org> On Feb 17, 2015, at 01:25 PM, Ethan Furman wrote: >On 02/17/2015 01:13 PM, Barry Warsaw wrote: > >> There are lots of interesting use cases we could still support. I could >> build a single-file application and deploy it into my OS's system archives. >> I could deploy that thing into my platform's app store. I could hand it >> over to a colleague running an identical version of the OS. I could deploy >> it into my data center. I could install it on my phone. > >All those locations have Python installed in the same place? Even your >phone?? But of course! :) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Tue Feb 17 22:45:32 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 17 Feb 2015 21:45:32 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217161012.4d03aacc@anarchist.wooz.org> References: <20150217140038.73a487ae@anarchist.wooz.org> <20150217161012.4d03aacc@anarchist.wooz.org> Message-ID: On 17 February 2015 at 21:10, Barry Warsaw wrote: > The actual "import an extension module" support in zipimport does need to be > cross-platform, but it can work differently depending on the platform. For > example, if extended-dlopen is available on your Linux machine, zipimport > could just use that. For a traditional-dlopen-only machine, it would > copy-to-filesystem. It would do whatever Windowsy thing makes sense over > there too. Oh, absolutely. As long as the functionality is supported everywhere, the implementation can be whatever is needed. > I'm much less concerned about the cross-platform portability of the resulting > pyz files. There's lots of reasons why an application built in (or for) my > Linux machine might not work properly on your Windows machine, but FWIW if it > were the only problem, doing a native rebuild on the target platform (or via > supported cross-building) would be fine with me. Yeah, it's not at all clear to me what people's intended use cases are for pyz files. Personally, I want them for writing scripts I can copy between my various PCs without hassle. (So single file saves me from having to make sure I have dependencies installed everywhere, but sys.executable ties me to a specific install location). Your use case seems similar (but on Unix, where there's less likelihood that Python will be installed in different places). I don't know if anyone is seeing them as a distribution format (i.e., they put their software up on their website for anyone to use). If Python zip applications got the ability to include binary extensions, they would *definitely* not be portable (we don't want to go down the route of wheel-like compatibility tags for an application file format). Currently, I'm trying to leave this aspect of the PEP open, so the PEP describes what the format is and how it works, but doesn't try to imply particular use cases. That way people can use it how they prefer, and the PEP doesn't put people off with a lot of caveats. My current draft of the documentation page for the zipapp module includes an example of setting a shebang line, with some notes on issues to consider for a portable shebang line if you propose distributing your application - but that's as far as I want to go. Paul From p.f.moore at gmail.com Tue Feb 17 22:46:32 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 17 Feb 2015 21:46:32 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217161346.6fc81fa1@anarchist.wooz.org> References: <20150217135648.1f494957@anarchist.wooz.org> <20150217161346.6fc81fa1@anarchist.wooz.org> Message-ID: On 17 February 2015 at 21:13, Barry Warsaw wrote: > Don't discount sys.executable to quickly. :) I used pyzzer when it defaulted to sys.executable, and the experience was painful. That's on Windows, and Unix may be different, but let's just say I don't want to go there :-) Paul From p.f.moore at gmail.com Tue Feb 17 22:52:17 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 17 Feb 2015 21:52:17 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217164019.2423bb37@anarchist.wooz.org> References: <20150217134449.28fa4592@anarchist.wooz.org> <20150217164019.2423bb37@anarchist.wooz.org> Message-ID: On 17 February 2015 at 21:40, Barry Warsaw wrote: > OTOH, I think it *would* be useful to have this built into Python. We almost > have too many choices, and may people I've talked to lately (including > experienced Pythonistas) have trouble choosing the right tool. But that can > come later; let's get some basic support into Python first, and a good start > IMHO is executable zips (including support for extension modules) with the > pkg_resources/importlib API additions. I see no problem with either having zipapp develop into a much fuller-featured solution, or having an externally developed tool brought into the stdlib and replacing zipapp. I just don't think there's anything mature enough for that role out there yet, and I don't feel that we know enough yet to design the right solution from scratch in the PEP. As you say, though, that can come later. If 3.5 includes PEP 441 and Donald's proposed importlib additions, I'll be happy. If it also includes zipimport support for C extensions I'll be ecstatic! (But I suspect that one's more a 3.6 timeframe) Paul From barry at python.org Tue Feb 17 23:05:21 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 17:05:21 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <20150217134449.28fa4592@anarchist.wooz.org> <20150217164019.2423bb37@anarchist.wooz.org> Message-ID: <20150217170521.2d8ea97f@anarchist.wooz.org> On Feb 17, 2015, at 09:52 PM, Paul Moore wrote: >I see no problem with either having zipapp develop into a much >fuller-featured solution, or having an externally developed tool >brought into the stdlib and replacing zipapp. I just don't think >there's anything mature enough for that role out there yet, and I >don't feel that we know enough yet to design the right solution from >scratch in the PEP. > >As you say, though, that can come later. If 3.5 includes PEP 441 and >Donald's proposed importlib additions, I'll be happy. If it also >includes zipimport support for C extensions I'll be ecstatic! (But I >suspect that one's more a 3.6 timeframe) Agreed! The only difference between executable zips and the full-featured self-contained app is the goop you put at the front of the file, right? :) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From dholth at gmail.com Tue Feb 17 23:06:18 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 17 Feb 2015 17:06:18 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On Sun, Feb 15, 2015 at 1:45 PM, Serhiy Storchaka wrote: > On 15.02.15 18:21, Thomas Wouters wrote: >> >> which requires that extension modules are stored uncompressed (simple) >> and page-aligned (harder, as the zipfile.ZipFile class doesn't directly >> support page-aligning anything > > > It is possible to add this feature to ZipFile. It can be useful because will > allow to mmap uncompressed files in ZIP file. One of the cool things about ZIP files is that they ignore any data that is not referenced by the index (which is at the end of the file). That is why we can put a #! at the beginning of the file, prefix .exe self-extractors, or put extra data between files in the archive. You could probably even overlap archive members if you were motivated. But it would be best to let the uncompressed extensions be standard ZIP members. From barry at python.org Tue Feb 17 23:07:35 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Feb 2015 17:07:35 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <20150217140038.73a487ae@anarchist.wooz.org> <20150217161012.4d03aacc@anarchist.wooz.org> Message-ID: <20150217170735.39ad5f48@anarchist.wooz.org> On Feb 17, 2015, at 09:45 PM, Paul Moore wrote: >If Python zip applications got the ability to include binary extensions, they >would *definitely* not be portable (we don't want to go down the route of >wheel-like compatibility tags for an application file format). Agreed! >Currently, I'm trying to leave this aspect of the PEP open, so the PEP >describes what the format is and how it works, but doesn't try to imply >particular use cases. That way people can use it how they prefer, and the PEP >doesn't put people off with a lot of caveats. My current draft of the >documentation page for the zipapp module includes an example of setting a >shebang line, with some notes on issues to consider for a portable shebang >line if you propose distributing your application - but that's as far as I >want to go. I think that's totally reasonable. Admittedly, I was hijacking your PEP discussion for some pie-in-the-sky musing. :) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From v+python at g.nevcal.com Wed Feb 18 00:29:34 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 17 Feb 2015 15:29:34 -0800 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217164019.2423bb37@anarchist.wooz.org> References: <20150217134449.28fa4592@anarchist.wooz.org> <20150217164019.2423bb37@anarchist.wooz.org> Message-ID: <54E3CEDE.2070906@g.nevcal.com> On 2/17/2015 1:40 PM, Barry Warsaw wrote: > I agree, for deployment to environments where you can guarantee that the basic > Python infrastructure is available. It may not be though, and perhaps we can > relegate that to py2exe, cx_freeze, and friends, although I would say that > having such support built into Python would be very cool, and of high value to > our users. > > That's not to play down the usefulness of good executable zip support, pared > of the stdlib and interpreter. OTOH, it would be nice to think ahead, where > the same tool could generate both types of single-file executables. Thinking ahead? Novel idea! My dream scheme... A scheme for putting all the pure- Python modules (mostly user code) into a .pyz, and a concurrent scheme for putting all the non-pure- Python modules (maybe even Python itself, for platforms that are missing the necessary version of Python 3), into some other platform-dependent bundle. User instructions then become... try the .pyz, if it doesn't work, install the platform-dependent bundle and then it will. So one could build up a platform-dependent bundle that contains all stuff used in development of various projects, and have easy user instructions for deployment of various .pyz apps. Longer instructions for people that think they know what they are doing would list the version of Python and the versions of the various dependencies in the platform-dependent bundle, if they wish to install them manually, etc. I suppose licensing might, in some cases, conflict with making a single bundle for some modules and platforms. I'm already doing the .pyz apps, using .py extensions, but don't have the other piece in place.... it's harder, and I haven't acquired the skill set for putting the binary chunks together... other than unzipping to the filesystem. Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From blaxxton at yahoo.com Wed Feb 18 05:06:09 2015 From: blaxxton at yahoo.com (Blaxton) Date: Wed, 18 Feb 2015 04:06:09 +0000 (UTC) Subject: [Python-Dev] Python 3.4 RPM on AIX In-Reply-To: References: Message-ID: <2090530108.214248.1424232369189.JavaMail.yahoo@mail.yahoo.com> Hi Zach, I think it is best to remove the .spec file if it is not supported by Python developers.It is misleading to ship unsupported file within source tree. Thanks for reply. From: Zachary Ware To: "python-dev at python.org" Cc: Blaxton Sent: Monday, February 16, 2015 11:54 PM Subject: Re: [Python-Dev] Python 3.4 RPM on AIX On Mon, Feb 16, 2015 at 10:50 PM, Blaxton wrote: > I am using the spec file that comes with Python source code which downloaded > from python.org website > source file is set on spec file to file with bz2 format while there is only > .xz and zipped are available to download. > > > I thought somehow some one from development team has wrote that .spec file > and maybe supporting it. > > This is like no support. The spec file in Misc/RPM has not been maintained in quite some time; the last change to that file that wasn't just a version bump was one [1] whose commit message was "#5776: fix mistakes in python specfile. (Nobody probably uses it anyway.)" and was committed four and a half years ago.? It's not unlikely that you're the first person to use the file since then :). We should probably just remove the file, but if you manage to get it working and would like to provide a patch on bugs.python.org, it would probably be accepted. [1] https://hg.python.org/cpython/rev/ef75ecd0e1a7 -- Zach -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Feb 18 08:45:56 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 18 Feb 2015 09:45:56 +0200 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150217162528.04da71df@anarchist.wooz.org> References: <20150217135256.1512523e@anarchist.wooz.org> <20150217162528.04da71df@anarchist.wooz.org> Message-ID: On 17.02.15 23:25, Barry Warsaw wrote: > I'm not sure sys.getfilesystemencoding() is the right encoding, rather than > sys.getdefaultencoding(), if you're talking about the encoding of the shebang > line rather than the encoding of the resulting pyz filename. On POSIX sys.getfilesystemencoding() is the right encoding because the shebang is read by system loader which doesn't encode/decode, but uses a file name as raw bytes string. On Mac OS always is UTF-8, but sys.getdefaultencoding() can be ASCII. From benjamin at python.org Wed Feb 18 14:57:48 2015 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 18 Feb 2015 08:57:48 -0500 Subject: [Python-Dev] Python 3.4 RPM on AIX In-Reply-To: <2090530108.214248.1424232369189.JavaMail.yahoo@mail.yahoo.com> References: <2090530108.214248.1424232369189.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1424267868.788959.229212373.496F2381@webmail.messagingengine.com> On Tue, Feb 17, 2015, at 23:06, Blaxton wrote: > Hi Zach, > I think it is best to remove the .spec file if it is not supported by > Python developers.It is misleading to ship unsupported file within source > tree. I agree, so I just removed it. From jimjjewett at gmail.com Wed Feb 18 21:48:30 2015 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Wed, 18 Feb 2015 12:48:30 -0800 (PST) Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: Message-ID: <54e4fa9e.0d98e00a.2452.6e29@mx.google.com> Barry Warsaw wrote: >> I don't know exactly what the procedure would be to claim .pyz for *nix, >> e.g. updating /etc/mime.types, but I think the PEP should at least mention >> this. I think we want to get as official support for .pyz files on *nix as >> possible. Paul Moore wrote: > I'll add a note to the PEP, but I have no idea how we would even go > about that, so that's all I can do, unfortunately. Are you just looking for http://www.iana.org/assignments/media-types/media-types.xhtml and its references, including the registration procedures http://tools.ietf.org/html/rfc6838#section-4.2.5 and the application form at http://www.iana.org/form/media-types ? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From p.f.moore at gmail.com Wed Feb 18 22:16:04 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 18 Feb 2015 21:16:04 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54e4fa9e.0d98e00a.2452.6e29@mx.google.com> References: <54e4fa9e.0d98e00a.2452.6e29@mx.google.com> Message-ID: On 18 February 2015 at 20:48, Jim J. Jewett wrote: > Barry Warsaw wrote: >>> I don't know exactly what the procedure would be to claim .pyz for *nix, >>> e.g. updating /etc/mime.types, but I think the PEP should at least mention >>> this. I think we want to get as official support for .pyz files on *nix as >>> possible. > > Paul Moore wrote: >> I'll add a note to the PEP, but I have no idea how we would even go >> about that, so that's all I can do, unfortunately. > > Are you just looking for > > http://www.iana.org/assignments/media-types/media-types.xhtml > > and its references, including the registration procedures > > http://tools.ietf.org/html/rfc6838#section-4.2.5 > > and the application form at > > http://www.iana.org/form/media-types > > ? That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about. Actually, I guess the first question is whether ".py" is reserved for Python scripts. If it is, then certainly /pyz should be reserved in a similar way for Python Zip Applications. If there's no formal registration of .py, then it seems a bit unreasonable to bother for .pyz... Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address... Paul From barry at python.org Wed Feb 18 22:41:44 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 18 Feb 2015 16:41:44 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54e4fa9e.0d98e00a.2452.6e29@mx.google.com> Message-ID: <20150218164144.1377586f@anarchist.wooz.org> On Feb 18, 2015, at 09:16 PM, Paul Moore wrote: >That covers mime types, but not file extensions, so it's not really >what *I* thought Barry was talking about. I was actually thinking about both. :) The Wikipedia page has several external links to collections of file extensions, but I don't know that there's *an* official registry. http://en.wikipedia.org/wiki/File_extension Picking FileInfo.com at random, and searching for a few .py* extensions, you find the usual suspects, plus a few others, but all are Python related, and there's no .pyz extension currently registered. >Also, I don't think reserving anything is something I, as an >individual (and specifically a non-Unix user) should do. It probably >should be handled by the PSF, as the process seems to need a contact >email address... Probably so. Seems like they could pick one or more of these registries and register pyz for this purpose. Cheers, -Barry From encukou at gmail.com Thu Feb 19 10:19:57 2015 From: encukou at gmail.com (Petr Viktorin) Date: Thu, 19 Feb 2015 10:19:57 +0100 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54e4fa9e.0d98e00a.2452.6e29@mx.google.com> Message-ID: On Wed, Feb 18, 2015 at 10:16 PM, Paul Moore wrote: > On 18 February 2015 at 20:48, Jim J. Jewett wrote: >> Barry Warsaw wrote: >>>> I don't know exactly what the procedure would be to claim .pyz for *nix, >>>> e.g. updating /etc/mime.types, but I think the PEP should at least mention >>>> this. I think we want to get as official support for .pyz files on *nix as >>>> possible. >> >> Paul Moore wrote: >>> I'll add a note to the PEP, but I have no idea how we would even go >>> about that, so that's all I can do, unfortunately. >> >> Are you just looking for >> >> http://www.iana.org/assignments/media-types/media-types.xhtml >> >> and its references, including the registration procedures >> >> http://tools.ietf.org/html/rfc6838#section-4.2.5 >> >> and the application form at >> >> http://www.iana.org/form/media-types >> >> ? > > That covers mime types, but not file extensions, so it's not really > what *I* thought Barry was talking about. > > Actually, I guess the first question is whether ".py" is reserved for > Python scripts. If it is, then certainly /pyz should be reserved in a > similar way for Python Zip Applications. If there's no formal > registration of .py, then it seems a bit unreasonable to bother for > .pyz... > > Also, I don't think reserving anything is something I, as an > individual (and specifically a non-Unix user) should do. It probably > should be handled by the PSF, as the process seems to need a contact > email address... There is no extension registry for Unixy systems, because extensions traditionally don't matter. File types are identified by their contents; there is a "magic"* database that maps file contents to types (MIME types, nowadays). The `file` command is the CLI interface for that. Some programs will fall back on the extension if that doesn't give a result, but that would depend on the program. For a pyz file to run when you "open" it (for most values of "open"), it needs to have the executable bit set, and have a shebang line. * /usr/share/misc/magic ? it originally contained just magic numbers, such as those you might find at the beginning of a .pyc file. From jimjjewett at gmail.com Thu Feb 19 19:11:42 2015 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Thu, 19 Feb 2015 13:11:42 -0500 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54e4fa9e.0d98e00a.2452.6e29@mx.google.com> Message-ID: On Wed, Feb 18, 2015 at 4:16 PM, Paul Moore wrote: > On 18 February 2015 at 20:48, Jim J. Jewett wrote: >> Barry Warsaw wrote: >>>> I don't know exactly what the procedure would be to claim .pyz for *nix, >>>> e.g. updating /etc/mime.types, ... >> Are you just looking for >> http://www.iana.org/assignments/media-types/media-types.xhtml and ... > That covers mime types, but not file extensions, so it's not really > what *I* thought Barry was talking about. Question 13 at http://www.iana.org/form/media-types asks for additional information, and specifically calls out Magic Number and File Extension, among others. I doubt there is any more official repository for file extension meaning within MIME or unix. > Also, I don't think reserving anything is something I, as an > individual (and specifically a non-Unix user) should do. It probably > should be handled by the PSF, as the process seems to need a contact > email address... Ideally, it would be a long-lasting organizational address, such as pep-editor at python.org. But often, it isn't. -jJ From p.f.moore at gmail.com Thu Feb 19 19:33:31 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 19 Feb 2015 18:33:31 +0000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54e4fa9e.0d98e00a.2452.6e29@mx.google.com> Message-ID: On 19 February 2015 at 18:11, Jim J. Jewett wrote: >> Also, I don't think reserving anything is something I, as an >> individual (and specifically a non-Unix user) should do. It probably >> should be handled by the PSF, as the process seems to need a contact >> email address... > > Ideally, it would be a long-lasting organizational address, such as > pep-editor at python.org. But often, it isn't. Given that .py isn't on that list in any form I could locate, I don't think it's reasonable to add .pyz. If someone wants to propose adding both, I'd be fine with that, but it can be done independently of PEP 441. I'll add a note to the PEP to that effect. On that note, I intend to post the latest version of the PEP, and the reference implementation, tomorrow. I'll get the official version of the PEP on python.org updated at the same time. Paul From storchaka at gmail.com Thu Feb 19 23:50:43 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 20 Feb 2015 00:50:43 +0200 Subject: [Python-Dev] TypeError messages Message-ID: Different patterns for TypeError messages are used in the stdlib: expected X, Y found expected X, found Y expected X, but Y found expected X instance, Y found X expected, not Y expect X, not Y need X, Y found X is required, not Y Z must be X, not Y Z should be X, not Y and more. What the pattern is most preferable? Some messages use the article before X or Y. Should the article be used or omitted? Some messages (only in C) truncate actual type name (%.50s, %.80s, %.200s, %.500s). Should type name be truncated at all and for how limit? Type names newer truncated in TypeError messages raised in Python code. Some messages enclose actual type name with single quotes ('%s', '%.200s'). Should type name be quoted? It is uncommon if type name contains spaces. From python at mrabarnett.plus.com Fri Feb 20 00:57:10 2015 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 19 Feb 2015 23:57:10 +0000 Subject: [Python-Dev] TypeError messages In-Reply-To: References: Message-ID: <54E67856.2090806@mrabarnett.plus.com> On 2015-02-19 22:50, Serhiy Storchaka wrote:> Different patterns for TypeError messages are used in the stdlib: > > expected X, Y found > expected X, found Y > expected X, but Y found > expected X instance, Y found > X expected, not Y > expect X, not Y > need X, Y found > X is required, not Y > Z must be X, not Y > Z should be X, not Y > > and more. > > What the pattern is most preferable? > Stylistically, if the first part is in the active voice, then the second part should also be in the active voice: expected X, but found Y The active voice tends to be simpler and easier to parse than the passive voice. I think that the word "but" adds clarity here. Strictly speaking, that message is OK only if it's expecting X itself; if, in fact, it's expecting an instance of X, then you should really be saying something along the lines of: expected X instance, but found Y instance or: expected instance of X, but found instance of Y > Some messages use the article before X or Y. Should the article be > used or omitted? > Messages tend not to be complete sentences anyway, so I think that it would be fitting to omit articles. > Some messages (only in C) truncate actual type name (%.50s, %.80s, > %.200s, %.500s). Should type name be truncated at all and for how > limit? Type names newer truncated in TypeError messages raised in > Python code. > Truncating type names is probably not a good idea. > Some messages enclose actual type name with single quotes ('%s', > '%.200s'). Should type name be quoted? It is uncommon if type name > contains spaces. > I think that it should be quoted only if it's expecting those characters, e.g. if it's expecting a closing parenthesis, then it should say ')'. If, on the other hand, it's expecting a certain type, then it should give that type unquoted. From rob.cliffe at btinternet.com Fri Feb 20 02:02:43 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 20 Feb 2015 01:02:43 +0000 Subject: [Python-Dev] TypeError messages In-Reply-To: <54E67856.2090806@mrabarnett.plus.com> References: <54E67856.2090806@mrabarnett.plus.com> Message-ID: <54E687B3.1000807@btinternet.com> On 19/02/2015 23:57, MRAB wrote: > On 2015-02-19 22:50, Serhiy Storchaka wrote:> Different patterns for > TypeError messages are used in the stdlib: > > > > expected X, Y found > > expected X, found Y > > expected X, but Y found > > expected X instance, Y found > > X expected, not Y > > expect X, not Y > > need X, Y found > > X is required, not Y > > Z must be X, not Y > > Z should be X, not Y > > > > and more. > > > > What the pattern is most preferable? > > > Stylistically, if the first part is in the active voice, then the > second part should also be in the active voice: > > expected X, but found Y > > The active voice tends to be simpler and easier to parse than the > passive voice. > > I think that the word "but" adds clarity here. > > Strictly speaking, that message is OK only if it's expecting X itself; > if, in fact, it's expecting an instance of X, then you should really be > saying something along the lines of: > > expected X instance, but found Y instance > > or: > > expected instance of X, but found instance of Y To me that is a purist argument, but in practice expected int, but found tuple is perfectly clear (I have received such messages myself!), and extra verbiage is just clutter. Identifying Z may be helpful however, so my feeling would be to stick to 2 patterns - the one above and Z should be int, not tuple (You may prefer e.g. "got" to "found", or "must" to "should". But ideally the usage should be the same throughout.) Rob Cliffe > > > > Some messages use the article before X or Y. Should the article be > > used or omitted? > > > Messages tend not to be complete sentences anyway, so I think that it > would be fitting to omit articles. > > > Some messages (only in C) truncate actual type name (%.50s, %.80s, > > %.200s, %.500s). Should type name be truncated at all and for how > > limit? Type names newer truncated in TypeError messages raised in > > Python code. > > > Truncating type names is probably not a good idea. > > > Some messages enclose actual type name with single quotes ('%s', > > '%.200s'). Should type name be quoted? It is uncommon if type name > > contains spaces. > > > I think that it should be quoted only if it's expecting those > characters, e.g. if it's expecting a closing parenthesis, then it > should say ')'. If, on the other hand, it's expecting a certain type, > then it should give that type unquoted. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com > > > ----- > No virus found in this message. > Checked by AVG - www.avg.com > Version: 2014.0.4800 / Virus Database: 4257/9145 - Release Date: 02/19/15 > > From solipsis at pitrou.net Fri Feb 20 11:47:09 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 20 Feb 2015 11:47:09 +0100 Subject: [Python-Dev] cpython: Improve struct cache locality by bring commonly accessed fields close together. References: <20150220085009.10184.75445@psf.io> Message-ID: <20150220114709.331acbcd@fsol> On Fri, 20 Feb 2015 08:50:10 +0000 raymond.hettinger wrote: > https://hg.python.org/cpython/rev/0ba4bcc0a7a6 > changeset: 94696:0ba4bcc0a7a6 > user: Raymond Hettinger > date: Fri Feb 20 00:50:04 2015 -0800 > summary: > Improve struct cache locality by bring commonly accessed fields close together. > > files: > Modules/_randommodule.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Modules/_randommodule.c b/Modules/_randommodule.c > --- a/Modules/_randommodule.c > +++ b/Modules/_randommodule.c > @@ -78,8 +78,8 @@ > > typedef struct { > PyObject_HEAD > + int index; > unsigned long state[N]; > - int index; > } RandomObject; I may be missing something something, but I don't see how that improves cache locality. The index is only ever looked up when the state is also looked up, and since the state is used sequentially there are equal chances for the state lookup to be near the end rather than near the front. Regards Antoine. From brett at python.org Fri Feb 20 15:05:11 2015 From: brett at python.org (Brett Cannon) Date: Fri, 20 Feb 2015 14:05:11 +0000 Subject: [Python-Dev] TypeError messages References: Message-ID: On Thu Feb 19 2015 at 5:52:07 PM Serhiy Storchaka wrote: > Different patterns for TypeError messages are used in the stdlib: > > expected X, Y found > expected X, found Y > expected X, but Y found > expected X instance, Y found > X expected, not Y > expect X, not Y > need X, Y found > X is required, not Y > Z must be X, not Y > Z should be X, not Y > > and more. > > What the pattern is most preferable? > My preference is for "expected X, but found Y". > > Some messages use the article before X or Y. Should the article be used > or omitted? > > Some messages (only in C) truncate actual type name (%.50s, %.80s, > %.200s, %.500s). Should type name be truncated at all and for how limit? > I assume this is over some worry of string size blowing out memory allocation or something? If someone can show that's an actual worry then fine, otherwise I say don't truncate. > Type names newer truncated in TypeError messages raised in Python code. > > Some messages enclose actual type name with single quotes ('%s', > '%.200s'). Should type name be quoted? It is uncommon if type name > contains spaces. > I agree that type names don't need to be quoted. -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlosjosepita at gmail.com Thu Feb 19 18:15:32 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Thu, 19 Feb 2015 14:15:32 -0300 Subject: [Python-Dev] Idea: __length_hint__ wrapper in itertools Message-ID: Hi all, python now supports the __length_hint__ method but not for every use case that could potentially benefit from it, v.g.: 1) Some common builtins like map don't set the hint. 2) Generators. 3) Application specific hints that, by definition, can't be predicted by a general strategy. I know 1 and 2 were discussed at some length in the past, but still there is no agreement about what to do with them. In the meantime, and because of 3, what do you think about adding a simple iterator wrapper to itertools which would allow to provide the hint by the user without any significant performance lost (__next__ in the wrapper could be just the original __next__). AFAIK there is no such thing in the standard library or anywhere else and it seems pretty easy to implement, although maybe I'm completely wrong. Say: hinted_iter = it.length_hint(iter, hint) Cheers -- Carlos From Stefan.Richthofer at gmx.de Fri Feb 20 16:25:04 2015 From: Stefan.Richthofer at gmx.de (Stefan Richthofer) Date: Fri, 20 Feb 2015 16:25:04 +0100 Subject: [Python-Dev] TypeError messages In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From brett at python.org Fri Feb 20 16:40:17 2015 From: brett at python.org (Brett Cannon) Date: Fri, 20 Feb 2015 15:40:17 +0000 Subject: [Python-Dev] TypeError messages References: Message-ID: On Fri Feb 20 2015 at 10:27:35 AM Stefan Richthofer < Stefan.Richthofer at gmx.de> wrote: > Honestly, the right solution would be to have a function or macro that > generates the TypeError messages > from X, Y, Z arguments. (Until now I actually believed this would be > already the case) > - type errors would be of uniform style > - if for some reoson the format should be changed, this can be done in one > central place > - if someone would want to parse the error message this would be feasible > I suppose it should be straight forward to find error message creations in > the source by searching for "TypeError" or something. > Maybe this kind of cleanup could be done along with the implementation of > PEP 484? > Actually PEP 473 covers standardizing error messages by introducing keyword-only arguments which would lead to a standardized message being generated. From the C side there can be a function provided to make it easy to get the same result as constructing the exception with the keyword-only argument. -Brett > > -Stefan > > *Gesendet:* Freitag, 20. Februar 2015 um 15:05 Uhr > *Von:* "Brett Cannon" > *An:* "Serhiy Storchaka" , python-dev at python.org > *Betreff:* Re: [Python-Dev] TypeError messages > > On Thu Feb 19 2015 at 5:52:07 PM Serhiy Storchaka > wrote: >> >> Different patterns for TypeError messages are used in the stdlib: >> >> expected X, Y found >> expected X, found Y >> expected X, but Y found >> expected X instance, Y found >> X expected, not Y >> expect X, not Y >> need X, Y found >> X is required, not Y >> Z must be X, not Y >> Z should be X, not Y >> >> and more. >> >> What the pattern is most preferable? > > > My preference is for "expected X, but found Y". > > >> >> Some messages use the article before X or Y. Should the article be used >> or omitted? >> >> Some messages (only in C) truncate actual type name (%.50s, %.80s, >> %.200s, %.500s). Should type name be truncated at all and for how limit? > > > I assume this is over some worry of string size blowing out memory > allocation or something? If someone can show that's an actual worry then > fine, otherwise I say don't truncate. > > >> Type names newer truncated in TypeError messages raised in Python code. >> >> Some messages enclose actual type name with single quotes ('%s', >> '%.200s'). Should type name be quoted? It is uncommon if type name >> contains spaces. > > > I agree that type names don't need to be quoted. > _______________________________________________ Python-Dev mailing list > Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.de > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Fri Feb 20 17:11:25 2015 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 20 Feb 2015 11:11:25 -0500 Subject: [Python-Dev] TypeError messages In-Reply-To: References: Message-ID: <54E75CAD.7010309@trueblade.com> On 02/20/2015 09:05 AM, Brett Cannon wrote: > Some messages (only in C) truncate actual type name (%.50s, %.80s, > %.200s, %.500s). Should type name be truncated at all and for how limit? > > > I assume this is over some worry of string size blowing out memory > allocation or something? If someone can show that's an actual worry then > fine, otherwise I say don't truncate. I asked about this years ago, and was told it was in case the type name pointer was bad, and to limit the amount of garbage printed. Whether that's an actual problem or not, I can't say. It seems more likely that you'd get a segfault, but maybe if it was pointing to reused memory it could be useful. Eric. From demianbrecht at gmail.com Fri Feb 20 17:38:17 2015 From: demianbrecht at gmail.com (Demian Brecht) Date: Fri, 20 Feb 2015 08:38:17 -0800 Subject: [Python-Dev] str(IntEnum) Message-ID: <396CC6AB-6181-4B7F-AC1F-9EC5D898C7CA@gmail.com> While working on a bug in the issue tracker, I came across something that I thought was a little odd around the behaviour of IntEnum. Should the behaviour of an instance of an IntEnum not be symmetric to an int where possible? For example: >>> class MyEnum(IntEnum): ... FOO = 1 ... >>> MyEnum.FOO == 1 True >>> MyEnum.FOO * 3 == 3 True >>> str(MyEnum.FOO) == str(1) False In my mind, the string representation here should be ?1? and not the label. Was this simply an oversight of the specialized IntEnum implementation, or was there a concrete reason for this that I?m not seeing? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Fri Feb 20 17:54:20 2015 From: brett at python.org (Brett Cannon) Date: Fri, 20 Feb 2015 16:54:20 +0000 Subject: [Python-Dev] str(IntEnum) References: <396CC6AB-6181-4B7F-AC1F-9EC5D898C7CA@gmail.com> Message-ID: On Fri Feb 20 2015 at 11:39:11 AM Demian Brecht wrote: > While working on a bug in the issue tracker, I came across something that > I thought was a little odd around the behaviour of IntEnum. Should the > behaviour of an instance of an IntEnum not be symmetric to an int where > possible? For example: > > >>> class MyEnum(IntEnum): > ... FOO = 1 > ... > >>> MyEnum.FOO == 1 > True > >>> MyEnum.FOO * 3 == 3 > True > >>> str(MyEnum.FOO) == str(1) > False > > In my mind, the string representation here should be ?1? and not the > label. Was this simply an oversight of the specialized IntEnum > implementation, or was there a concrete reason for this that I?m not seeing? > Concrete reason. The string is 'MyEnum.FOO' which is much more readable and obvious where the value came from. The fact that it can be treated as an int is the same as the reason True and False are subclasses of int; it made practical sense for compatibility with what they typically replaced, but where it made more sense to diverge and introduce new behaviour then we did so. -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Feb 20 18:08:17 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 20 Feb 2015 18:08:17 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20150220170817.8C16E561DC@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-02-13 - 2015-02-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4770 (-20) closed 30484 (+54) total 35254 (+34) Open issues with patches: 2239 Issues opened (24) ================== #23018: Add version info to python http://bugs.python.org/issue23018 reopened by serhiy.storchaka #23460: Decimals do not obey ':g' exponential notation formatting rule http://bugs.python.org/issue23460 opened by ikelly #23462: All os.exec*e variants crash on Windows http://bugs.python.org/issue23462 opened by nicolaje #23464: Remove or deprecate JoinableQueue in asyncio docs http://bugs.python.org/issue23464 opened by emptysquare #23465: Implement PEP 486 - Make the Python Launcher aware of virtual http://bugs.python.org/issue23465 opened by pmoore #23466: PEP 461: Inconsistency between str and bytes formatting of int http://bugs.python.org/issue23466 opened by serhiy.storchaka #23467: Improve byte formatting compatibility between Py2 and Py3 http://bugs.python.org/issue23467 opened by serhiy.storchaka #23469: Delete Misc/*.wpr files http://bugs.python.org/issue23469 opened by berker.peksag #23470: OpenBSD buildbot uses wrong stdlib http://bugs.python.org/issue23470 opened by serhiy.storchaka #23471: 404 Not Found when downloading Python 3.4.3rc1 Documentation http://bugs.python.org/issue23471 opened by stringsonfire #23472: Setup locales on buildbots http://bugs.python.org/issue23472 opened by serhiy.storchaka #23476: SSL cert verify fail for "www.verisign.com" http://bugs.python.org/issue23476 opened by nagle #23477: Increase coverage for wsgiref module http://bugs.python.org/issue23477 opened by ashkop #23484: SemLock acquire() keyword arg 'blocking' is invalid http://bugs.python.org/issue23484 opened by td #23485: PEP 475: handle EINTR in the select and selectors module http://bugs.python.org/issue23485 opened by haypo #23486: Enum member lookup is 20x slower than normal class attribute l http://bugs.python.org/issue23486 opened by craigh #23487: argparse: add_subparsers 'action' broken http://bugs.python.org/issue23487 opened by derks #23488: Random objects twice as big as necessary on 64-bit builds http://bugs.python.org/issue23488 opened by rhettinger #23489: atexit handlers are not executed when using multiprocessing.Po http://bugs.python.org/issue23489 opened by juj #23490: allocation (and overwrite) of a 0 byte buffer http://bugs.python.org/issue23490 opened by pkt #23491: PEP 441 - Improving Python Zip Application Support http://bugs.python.org/issue23491 opened by pmoore #23492: Argument Clinic: improve generated parser for 1-argument funct http://bugs.python.org/issue23492 opened by serhiy.storchaka #23493: optimize sort_keys in json module by using operator.itemgetter http://bugs.python.org/issue23493 opened by wbolster #433028: SRE: (?flag:...) is not supported http://bugs.python.org/issue433028 reopened by serhiy.storchaka Most recent 15 issues with no replies (15) ========================================== #23493: optimize sort_keys in json module by using operator.itemgetter http://bugs.python.org/issue23493 #23492: Argument Clinic: improve generated parser for 1-argument funct http://bugs.python.org/issue23492 #23487: argparse: add_subparsers 'action' broken http://bugs.python.org/issue23487 #23485: PEP 475: handle EINTR in the select and selectors module http://bugs.python.org/issue23485 #23484: SemLock acquire() keyword arg 'blocking' is invalid http://bugs.python.org/issue23484 #23470: OpenBSD buildbot uses wrong stdlib http://bugs.python.org/issue23470 #23469: Delete Misc/*.wpr files http://bugs.python.org/issue23469 #23464: Remove or deprecate JoinableQueue in asyncio docs http://bugs.python.org/issue23464 #23459: Linux: expose the new execveat() syscall http://bugs.python.org/issue23459 #23456: asyncio: add missing @coroutine decorators http://bugs.python.org/issue23456 #23455: file iterator "deemed broken"; can resume after StopIteration http://bugs.python.org/issue23455 #23444: HCI Bluetooth socket bind error on an arm crosscompiled enviro http://bugs.python.org/issue23444 #23443: XMLRPCLIB Exception uses str not class or instance http://bugs.python.org/issue23443 #23436: xml.dom.minidom.Element.ownerDocument is hiden http://bugs.python.org/issue23436 #23423: XPath Support in ElementTree doc omission http://bugs.python.org/issue23423 Most recent 15 issues waiting for review (15) ============================================= #23492: Argument Clinic: improve generated parser for 1-argument funct http://bugs.python.org/issue23492 #23491: PEP 441 - Improving Python Zip Application Support http://bugs.python.org/issue23491 #23490: allocation (and overwrite) of a 0 byte buffer http://bugs.python.org/issue23490 #23488: Random objects twice as big as necessary on 64-bit builds http://bugs.python.org/issue23488 #23486: Enum member lookup is 20x slower than normal class attribute l http://bugs.python.org/issue23486 #23477: Increase coverage for wsgiref module http://bugs.python.org/issue23477 #23469: Delete Misc/*.wpr files http://bugs.python.org/issue23469 #23466: PEP 461: Inconsistency between str and bytes formatting of int http://bugs.python.org/issue23466 #23465: Implement PEP 486 - Make the Python Launcher aware of virtual http://bugs.python.org/issue23465 #23458: [2.7] random: make the file descriptor non-inheritable (on POS http://bugs.python.org/issue23458 #23456: asyncio: add missing @coroutine decorators http://bugs.python.org/issue23456 #23451: Deprecation warnings building 3.5 Visual Studio Windows 8.1 64 http://bugs.python.org/issue23451 #23444: HCI Bluetooth socket bind error on an arm crosscompiled enviro http://bugs.python.org/issue23444 #23441: rlcompleter: tab on empty prefix => insert spaces http://bugs.python.org/issue23441 #23440: Extend http.server.SimpleHTTPRequestHandler testing http://bugs.python.org/issue23440 Top 10 most discussed issues (10) ================================= #22107: tempfile module misinterprets access denied error on Windows http://bugs.python.org/issue22107 16 msgs #21998: asyncio: support fork http://bugs.python.org/issue21998 8 msgs #22928: HTTP header injection in urrlib2/urllib/httplib/http.client http://bugs.python.org/issue22928 8 msgs #23465: Implement PEP 486 - Make the Python Launcher aware of virtual http://bugs.python.org/issue23465 7 msgs #23486: Enum member lookup is 20x slower than normal class attribute l http://bugs.python.org/issue23486 7 msgs #21257: Document parse_headers function of http.client http://bugs.python.org/issue21257 6 msgs #3566: httplib persistent connections violate MUST in RFC2616 sec 8.1 http://bugs.python.org/issue3566 5 msgs #20699: Document that binary IO classes work with bytes-likes objects http://bugs.python.org/issue20699 5 msgs #23425: Windows getlocale unix-like with french, german, portuguese, s http://bugs.python.org/issue23425 5 msgs #23452: Build errors using VS Express 2013 in win32 mode http://bugs.python.org/issue23452 5 msgs Issues closed (53) ================== #4773: HTTPMessage not documented and has inconsistent API across Py2 http://bugs.python.org/issue4773 closed by berker.peksag #9099: multiprocessing/win32: WindowsError: [Error 0] Success on Pipe http://bugs.python.org/issue9099 closed by davin #9119: Python download page needs to mention crypto code in Windows i http://bugs.python.org/issue9119 closed by berker.peksag #9913: Misc/SpecialBuilds.txt is out of date http://bugs.python.org/issue9913 closed by belopolsky #12782: Multiple context expressions do not support parentheses for co http://bugs.python.org/issue12782 closed by barry #14559: (2.7.3 Regression) PC\8.0 directory can no longer be used to http://bugs.python.org/issue14559 closed by steve.dower #14672: Windows installer: add desktop shortcut(s) http://bugs.python.org/issue14672 closed by steve.dower #14910: argparse: disable abbreviation http://bugs.python.org/issue14910 closed by berker.peksag #16111: Python 2.7.3 Windows MSI installer installs the VC++ 9 dlls di http://bugs.python.org/issue16111 closed by loewis #16361: HTTPS/TLS Problem in Python 3.3 http://bugs.python.org/issue16361 closed by loewis #17033: RPM spec file has old config_binsuffix value http://bugs.python.org/issue17033 closed by benjamin.peterson #17753: test_zipfile: requires write access to test and email.test http://bugs.python.org/issue17753 closed by serhiy.storchaka #19105: pprint doesn't use all width http://bugs.python.org/issue19105 closed by serhiy.storchaka #19681: test_repr (test.test_functools.TestPartialC) failures http://bugs.python.org/issue19681 closed by serhiy.storchaka #19792: pathlib does not support symlink in Windows XP http://bugs.python.org/issue19792 closed by steve.dower #20833: scripts\pydocgui.pyw out of date http://bugs.python.org/issue20833 closed by berker.peksag #21548: pydoc -k IndexError on empty docstring http://bugs.python.org/issue21548 closed by berker.peksag #21849: Fix multiprocessing for non-ascii data http://bugs.python.org/issue21849 closed by serhiy.storchaka #21898: .hgignore: Missing ignores for Eclipse/pydev http://bugs.python.org/issue21898 closed by berker.peksag #22235: httplib: TypeError with file() object in ssl.py http://bugs.python.org/issue22235 closed by r.david.murray #22633: Memory disclosure/buffer overread via bug in Py_FrozenMain http://bugs.python.org/issue22633 closed by python-dev #22735: Fix various crashes exposed through mro() customization http://bugs.python.org/issue22735 closed by berker.peksag #22844: test_gdb failure on Debian Wheezy for Z http://bugs.python.org/issue22844 closed by serhiy.storchaka #22883: Get rid of references to PyInt in Py3 sources http://bugs.python.org/issue22883 closed by serhiy.storchaka #22885: Arbitrary code execution vulnerability due to unchecked eval() http://bugs.python.org/issue22885 closed by serhiy.storchaka #23096: Implementation-depended pickling floats with protocol 0 http://bugs.python.org/issue23096 closed by serhiy.storchaka #23146: Incosistency in pathlib between / and \ http://bugs.python.org/issue23146 closed by pitrou #23169: Reflect that PreReq and BuildPreReq are deprecated in the late http://bugs.python.org/issue23169 closed by benjamin.peterson #23233: TypeError in ./setup.py http://bugs.python.org/issue23233 closed by serhiy.storchaka #23239: SSL match_hostname does not accept IP Address http://bugs.python.org/issue23239 closed by pitrou #23408: Pickle tests use incorrect test data http://bugs.python.org/issue23408 closed by serhiy.storchaka #23410: Document more BaseHTTPRequestHandler attributes http://bugs.python.org/issue23410 closed by python-dev #23418: Keep http.server.__all__ up to date http://bugs.python.org/issue23418 closed by berker.peksag #23422: Clarify docs for importlib.import_module() http://bugs.python.org/issue23422 closed by brett.cannon #23424: Unicode character ends interactive session http://bugs.python.org/issue23424 closed by terry.reedy #23437: Make user scripts directory versioned on Windows http://bugs.python.org/issue23437 closed by python-dev #23439: Fixed http.client.__all__ and added a test http://bugs.python.org/issue23439 closed by berker.peksag #23442: http.client.REQUEST_HEADER_FIELDS_TOO_LARGE renamed in 3.5 http://bugs.python.org/issue23442 closed by berker.peksag #23446: Use PyMem_New instead of PyMem_Malloc http://bugs.python.org/issue23446 closed by serhiy.storchaka #23450: Possible loss of data warnings building 3.5 Visual Studio Wind http://bugs.python.org/issue23450 closed by serhiy.storchaka #23457: make test failures http://bugs.python.org/issue23457 closed by dcrs6000 at gmail.com #23461: Building on windows modifies importlib.h http://bugs.python.org/issue23461 closed by steve.dower #23463: Incorrect behaviour when opening files containing colons on Wi http://bugs.python.org/issue23463 closed by tim.golden #23468: ISO 8601 datetime process http://bugs.python.org/issue23468 closed by belopolsky #23473: Allow namedtuple to be JSON encoded as dict http://bugs.python.org/issue23473 closed by serhiy.storchaka #23474: Enhance locale testing http://bugs.python.org/issue23474 closed by serhiy.storchaka #23475: asyncio: reference leak in test_close_kill_running() http://bugs.python.org/issue23475 closed by haypo #23478: A list arg with default value inside function got appended eac http://bugs.python.org/issue23478 closed by benjamin.peterson #23479: str.format() breaks object duck typing http://bugs.python.org/issue23479 closed by r.david.murray #23480: Minor typo http://bugs.python.org/issue23480 closed by SilentGhost #23481: SSL module should not offer RC4 based cipher suites for client http://bugs.python.org/issue23481 closed by python-dev #23482: sqlite3_d.dll is not included in installer http://bugs.python.org/issue23482 closed by python-dev #23483: python 2.7 builds using icc http://bugs.python.org/issue23483 closed by SilentGhost From demianbrecht at gmail.com Fri Feb 20 19:24:53 2015 From: demianbrecht at gmail.com (Demian Brecht) Date: Fri, 20 Feb 2015 10:24:53 -0800 Subject: [Python-Dev] Fwd: str(IntEnum) References: <5D84B808-5AB3-4FD4-94F1-A997548D7353@gmail.com> Message-ID: <5C3F4038-92F6-47DB-A9E8-6A811AB04924@gmail.com> > On Feb 20, 2015, at 8:54 AM, Brett Cannon wrote: > Concrete reason. The string is 'MyEnum.FOO' which is much more readable and obvious where the value came from. The fact that it can be treated as an int is the same as the reason True and False are subclasses of int; it made practical sense for compatibility with what they typically replaced, but where it made more sense to diverge and introduce new behaviour then we did so. Thanks Brett, that makes sense and was pretty much what I assumed. Reading the docs for __str__ does clarify things a bit in that the intention is to be a string representation of the object and not the value. That said, implementations seem to vary throughout the standard library and builtins: >>> str(uuid.uuid4()) 'd467d97c-fc09-4dc9-bea5-aeebdad8df8d? >>> str(int()) ?0' >>> str(datetime.datetime.now()) '2015-02-20 09:31:28.385539? >>> str(IPv4Address('127.0.0.1')) '127.0.0.1' These and other implementations return a string representation of the instance?s value, not a string representation of the object itself. Whereas elsewhere in the standard library: >>> str(ProtocolError('url', 42, 'msg', '')) '? >>> str(URLError('reason')) '? >>> str(Cookie(0, '', '', '4', '', '', '', '', '', '', '', 0, '', '', '', '')) '' The specific problem that I encountered was when swapping an IntEnum implementation for ints in http.client, which caused a change in logging output (from int.__str__ to Enum.__str__), which was a bit of a surprise, especially given the value is a builtin type. I think that a decent rule around the usage of __str__ is that it should be a string representation of the value, not of the object. Failing the ability to logically coerce the value to a string, it should simply fall back to repr(obj). Of course, I realize that the chances of this change being made to such a fundamental (and largely inconsequential) feature are likely nil, but I thought I?d share my thoughts anyways. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ethan at stoneleaf.us Fri Feb 20 19:44:45 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 20 Feb 2015 10:44:45 -0800 Subject: [Python-Dev] Fwd: str(IntEnum) In-Reply-To: <5C3F4038-92F6-47DB-A9E8-6A811AB04924@gmail.com> References: <5D84B808-5AB3-4FD4-94F1-A997548D7353@gmail.com> <5C3F4038-92F6-47DB-A9E8-6A811AB04924@gmail.com> Message-ID: <54E7809D.7020000@stoneleaf.us> On 02/20/2015 10:24 AM, Demian Brecht wrote: > I think that a decent rule around the usage of __str__ is that it should > be a string representation of the value, not of the object. Failing the > ability to logically coerce the value to a string, it should simply fall > back to repr(obj). There are two "stringy" methods for objects: __repr__ and __str__; if __str__ has not been defined __repr__ is automatically used. One of the motivating forces behind Enum is that often the name is more important (for humans) than the actual value, so __str__ is defined as being the Enum class name plus the Enum member name. The __repr__ has that as well as the underlying value (occasionally it's useful to know that, too ;) . -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From me at the-compiler.org Fri Feb 20 19:36:02 2015 From: me at the-compiler.org (Florian Bruhin) Date: Fri, 20 Feb 2015 19:36:02 +0100 Subject: [Python-Dev] Fwd: str(IntEnum) In-Reply-To: <5C3F4038-92F6-47DB-A9E8-6A811AB04924@gmail.com> References: <5D84B808-5AB3-4FD4-94F1-A997548D7353@gmail.com> <5C3F4038-92F6-47DB-A9E8-6A811AB04924@gmail.com> Message-ID: <20150220183602.GA429@tonks> * Demian Brecht [2015-02-20 10:24:53 -0800]: > These and other implementations return a string representation of the instance?s value, not a string representation of the object itself. Whereas elsewhere in the standard library: > > >>> str(ProtocolError('url', 42, 'msg', '')) > '? > >>> str(URLError('reason')) > '? > >>> str(Cookie(0, '', '', '4', '', '', '', '', '', '', '', 0, '', '', '', '')) > '' > > The specific problem that I encountered was when swapping an IntEnum implementation for ints in http.client, which caused a change in logging output (from int.__str__ to Enum.__str__), which was a bit of a surprise, especially given the value is a builtin type. > > I think that a decent rule around the usage of __str__ is that it should be a string representation of the value, not of the object. Failing the ability to logically coerce the value to a string, it should simply fall back to repr(obj). > > Of course, I realize that the chances of this change being made to such a fundamental (and largely inconsequential) feature are likely nil, but I thought I?d share my thoughts anyways. >>> foo = object() >>> str(foo) '' >>> repr(foo) '' This is exactly what you see above. ;) Florian -- http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From bcannon at gmail.com Fri Feb 20 19:47:30 2015 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 20 Feb 2015 18:47:30 +0000 Subject: [Python-Dev] Update to PEP 11 to clarify garnering platform support References: Message-ID: I just realized I actually never committed this change. Assuming no new objections I'll commit this in the near future (promise this time =). On Fri May 16 2014 at 1:51:00 PM Brett Cannon wrote: > Here is some proposed wording. Since it is more of a clarification of what > it takes to garner support -- which is just a new section -- rather than a > complete rewrite I'm including just the diff to make it easier to read the > changes. > > > *diff -r 49d18bb47ebc pep-0011.txt* > > *--- a/pep-0011.txt Wed May 14 11:18:22 2014 -0400* > > *+++ b/pep-0011.txt Fri May 16 13:48:30 2014 -0400* > > @@ -2,22 +2,21 @@ > > Title: Removing support for little used platforms > > Version: $Revision$ > > Last-Modified: $Date$ > > -Author: martin at v.loewis.de (Martin von L?wis) > > +Author: Martin von L?wis , > > + Brett Cannon > > Status: Active > > Type: Process > > Content-Type: text/x-rst > > Created: 07-Jul-2002 > > Post-History: 18-Aug-2007 > > + 16-May-2014 > > > > > > Abstract > > -------- > > > > -This PEP documents operating systems (platforms) which are not > > -supported in Python anymore. For some of these systems, > > -supporting code might be still part of Python, but will be removed > > -in a future release - unless somebody steps forward as a volunteer > > -to maintain this code. > > +This PEP documents how an operating system (platform) garners > > +support in Python as well as documenting past support. > > > > > > Rationale > > @@ -37,16 +36,53 @@ > > change to the Python source code will work on all supported > > platforms. > > > > -To reduce this risk, this PEP proposes a procedure to remove code > > -for platforms with no Python users. > > +To reduce this risk, this PEP specifies what is required for a > > +platform to be considered supported by Python as well as providing a > > +procedure to remove code for platforms with little or no Python > > +users. > > > > +Supporting platforms > > +-------------------- > > + > > +Gaining official platform support requires two things. First, a core > > +developer needs to volunteer to maintain platform-specific code. This > > +core developer can either already be a member of the Python > > +development team or be given contributor rights on the basis of > > +maintaining platform support (it is at the discretion of the Python > > +development team to decide if a person is ready to have such rights > > +even if it is just for supporting a specific platform). > > + > > +Second, a stable buildbot must be provided [2]_. This guarantees that > > +platform support will not be accidentally broken by a Python core > > +developer who does not have personal access to the platform. For a > > +buildbot to be considered stable it requires that the machine be > > +reliably up and functioning (but it is up to the Python core > > +developers to decide whether to promote a buildbot to being > > +considered stable). > > + > > +This policy does not disqualify supporting other platforms > > +indirectly. Patches which are not platform-specific but still done to > > +add platform support will be considered for inclusion. For example, > > +if platform-independent changes were necessary in the configure > > +script which was motivated to support a specific platform that would > > +be accepted. Patches which add platform-specific code such as the > > +name of a specific platform to the configure script will generally > > +not be accepted without the platform having official support. > > + > > +CPU architecture and compiler support are viewed in a similar manner > > +as platforms. For example, to consider the ARM architecture supported > > +a buildbot running on ARM would be required along with support from > > +the Python development team. In general it is not required to have > > +a CPU architecture run under every possible platform in order to be > > +considered supported. > > > > Unsupporting platforms > > ---------------------- > > > > -If a certain platform that currently has special code in it is > > -deemed to be without Python users, a note must be posted in this > > -PEP that this platform is no longer actively supported. This > > +If a certain platform that currently has special code in Python is > > +deemed to be without Python users or lacks proper support from the > > +Python development team and/or a buildbot, a note must be posted in > > +this PEP that this platform is no longer actively supported. This > > note must include: > > > > - the name of the system > > @@ -69,8 +105,8 @@ > > forward and offer maintenance. > > > > > > -Resupporting platforms > > ----------------------- > > +Re-supporting platforms > > +----------------------- > > > > If a user of a platform wants to see this platform supported > > again, he may volunteer to maintain the platform support. Such an > > @@ -101,7 +137,7 @@ > > release is made. Developers of extension modules will generally need > > to use the same Visual Studio release; they are concerned both with > > the availability of the versions they need to use, and with keeping > > -the zoo of versions small. The Python source tree will keep > > +the zoo of versions small. The Python source tree will keep > > unmaintained build files for older Visual Studio releases, for which > > patches will be accepted. Such build files will be removed from the > > source tree 3 years after the extended support for the compiler has > > @@ -223,6 +259,7 @@ > > ---------- > > > > .. [1] http://support.microsoft.com/lifecycle/ > > +.. [2] http://buildbot.python.org/3.x.stable/ > > > > Copyright > > --------- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Sat Feb 21 04:03:37 2015 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Fri, 20 Feb 2015 21:03:37 -0600 Subject: [Python-Dev] Fwd: str(IntEnum) In-Reply-To: <20150220183602.GA429@tonks> References: <5D84B808-5AB3-4FD4-94F1-A997548D7353@gmail.com> <5C3F4038-92F6-47DB-A9E8-6A811AB04924@gmail.com> <20150220183602.GA429@tonks> Message-ID: On Fri, Feb 20, 2015 at 12:36 PM, Florian Bruhin wrote: > * Demian Brecht [2015-02-20 10:24:53 -0800]: >> These and other implementations return a string representation of the instance?s value, not a string representation of the object itself. Whereas elsewhere in the standard library: >> >> >>> str(ProtocolError('url', 42, 'msg', '')) >> '? >> >>> str(URLError('reason')) >> '? >> >>> str(Cookie(0, '', '', '4', '', '', '', '', '', '', '', 0, '', '', '', '')) >> '' >> >> The specific problem that I encountered was when swapping an IntEnum implementation for ints in http.client, which caused a change in logging output (from int.__str__ to Enum.__str__) , which was a bit of a surprise, especially given the value is a builtin type. >> >> I think that a decent rule around the usage of __str__ is that it should be a string representation of the value, not of the object. Failing the ability to logically coerce the value to a string, it should simply fall back to repr(obj). >> >> Of course, I realize that the chances of this change being made to such a fundamental (and largely inconsequential) feature are likely nil, but I thought I?d share my thoughts anyways. > > >>> foo = object() > >>> str(foo) > '' > >>> repr(foo) > '' > > This is exactly what you see above. ;) > > Florian > > -- > http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) > GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc > I love long mails! | http://email.is-not-s.ms/ There's a semantic difference between an int and an IntEnum (or subclass thereof). MyEnum.FOO is a MyEnum type. IntEnums just happen to behave like an int under certain circumstances. That doesn't mean it needs to act like it in all. Further, it can be turned into an int if you want to represent it as an int, e.g., str(int(MyEnum.FOO)) == str(1). I hope this helps. From rosuav at gmail.com Sat Feb 21 06:56:25 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 Feb 2015 16:56:25 +1100 Subject: [Python-Dev] Any volunteers to implement PEP 479? In-Reply-To: References: Message-ID: On Wed, Jan 7, 2015 at 2:48 PM, Guido van Rossum wrote: > There's a proof of concept patch in http://bugs.python.org/issue22906, but > it doesn't have the __future__ import and probably gets other details wrong. > > Reference: > PEP 479 -- Change StopIteration handling inside generators > > -- > --Guido van Rossum (python.org/~guido) For the benefit of people who haven't been following the tracker issue: There is now a patch which *does* create the __future__ directive and so on. It applies cleanly to current tip (I just tested it again today), and the test suite passes on Debian AMD64. Can other people please try this, on other platforms, and see how it goes? ChrisA From kushaldas at gmail.com Sat Feb 21 07:01:40 2015 From: kushaldas at gmail.com (Kushal Das) Date: Sat, 21 Feb 2015 11:31:40 +0530 Subject: [Python-Dev] [X-POST] GSoC 2015 ideas Message-ID: Hi all, In case anyone wants to mentor a student for GSoC 2015, or have an idea on which students can work on during GSoC, please start putting them up in [1]. I am yet to get the edit access, I will put up the initial template text there. You can even send the idea as a reply to this thread. The template for the ideas is given below: == Project name == * '''Project description''': Make sure you have a high-level description that any student can understand, as well as deeper details * '''Skills''': programming languages? specific domain knowledge? * '''Difficulty level''': Easy/Intermediate/Hard classification (students ask for this info frequently to help them narrow down their choices) * '''Related Readings/Links''': was there a mailing list discussion about this topic? standards you want the students to read first? bugs/feature requests? * '''Potential mentors''': A list of mentors likely to be involved with this project, so students know who to look for on IRC/mailing lists if they have questions. (If you've had trouble with students overwhelming specific mentors, feel free to re-iterate here if students should contact the mailing list to reach all mentors.) [1] https://wiki.python.org/moin/SummerOfCode/2015/python-core Kushal -- Fedora Cloud Engineer CPython Core Developer Director Python Software Foundation http://kushaldas.in From guido at python.org Sat Feb 21 09:03:16 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Feb 2015 00:03:16 -0800 Subject: [Python-Dev] TypeError messages In-Reply-To: <54E75CAD.7010309@trueblade.com> References: <54E75CAD.7010309@trueblade.com> Message-ID: IIRC there's a limited buffer used for the formatting. Also, if a dynamically created type name is 1000000 characters long I'd rather see it truncated than blow up my shell window. On Friday, February 20, 2015, Eric V. Smith wrote: > On 02/20/2015 09:05 AM, Brett Cannon wrote: > > Some messages (only in C) truncate actual type name (%.50s, %.80s, > > %.200s, %.500s). Should type name be truncated at all and for how > limit? > > > > > > I assume this is over some worry of string size blowing out memory > > allocation or something? If someone can show that's an actual worry then > > fine, otherwise I say don't truncate. > > I asked about this years ago, and was told it was in case the type name > pointer was bad, and to limit the amount of garbage printed. Whether > that's an actual problem or not, I can't say. It seems more likely that > you'd get a segfault, but maybe if it was pointing to reused memory it > could be useful. > > Eric. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Feb 21 10:31:30 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 21 Feb 2015 09:31:30 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 486 - Make the Python Launcher aware of virtual environments Message-ID: The discussion on PEP 486 (started at https://mail.python.org/pipermail/python-dev/2015-February/138171.html, following from the thread at https://mail.python.org/pipermail/python-dev/2015-February/138160.html) seems to have died down. There's an implementation at http://bugs.python.org/issue23465. So, I think this is ready for pronouncement. Thanks, Paul From ncoghlan at gmail.com Sat Feb 21 15:05:07 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Feb 2015 00:05:07 +1000 Subject: [Python-Dev] PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54e4fa9e.0d98e00a.2452.6e29@mx.google.com> Message-ID: On 19 February 2015 at 07:16, Paul Moore wrote: > On 18 February 2015 at 20:48, Jim J. Jewett wrote: >> Barry Warsaw wrote: >>>> I don't know exactly what the procedure would be to claim .pyz for *nix, >>>> e.g. updating /etc/mime.types, but I think the PEP should at least mention >>>> this. I think we want to get as official support for .pyz files on *nix as >>>> possible. >> >> Paul Moore wrote: >>> I'll add a note to the PEP, but I have no idea how we would even go >>> about that, so that's all I can do, unfortunately. >> >> Are you just looking for >> >> http://www.iana.org/assignments/media-types/media-types.xhtml >> >> and its references, including the registration procedures >> >> http://tools.ietf.org/html/rfc6838#section-4.2.5 >> >> and the application form at >> >> http://www.iana.org/form/media-types >> >> ? > > That covers mime types, but not file extensions, so it's not really > what *I* thought Barry was talking about. FWIW, registering the vnd.python MIME prefix with IANA has been vaguely kicking around on my todo list since we switched the PyPA metadata PEPs over to using JSON. If anyone did decide to follow up on that idea, the PSF Infrastructure working group mailing list would likely be a suitable contact address to use (Infra already handles relationships with other internet registries for DNS et al, and I can't think of another group better suited to the task). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Feb 21 15:09:34 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Feb 2015 00:09:34 +1000 Subject: [Python-Dev] Idea: __length_hint__ wrapper in itertools In-Reply-To: References: Message-ID: On 20 February 2015 at 03:15, Carlos Pita wrote: > Hi all, > > python now supports the __length_hint__ method but not for every use > case that could potentially benefit from it, v.g.: > > 1) Some common builtins like map don't set the hint. > 2) Generators. > 3) Application specific hints that, by definition, can't be predicted > by a general strategy. > > I know 1 and 2 were discussed at some length in the past, but still > there is no agreement about what to do with them. > > In the meantime, and because of 3, what do you think about adding a > simple iterator wrapper to itertools which would allow to provide the > hint by the user without any significant performance lost (__next__ in > the wrapper could be just the original __next__). AFAIK there is no > such thing in the standard library or anywhere else and it seems > pretty easy to implement, although maybe I'm completely wrong. Seamless wrappers for arbitrary types are actually fairly tricky to implement, but Graham Dumpleton's wrapt library hides most of those messy details: https://pypi.python.org/pypi/wrapt A pure iterator-only wrapper would be a fair bit easier to implement, but it's not clear it's worth it for cases like this where data processing and memory reallocation time dominates your runtime costs sufficiently for __length_hint__ to make a big difference. Writing your own wrapper using a generic wrapper library like wrapt may be a better choice (and has the advantage of working today rather than only in 3.5+) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Feb 21 15:11:05 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Feb 2015 00:11:05 +1000 Subject: [Python-Dev] Update to PEP 11 to clarify garnering platform support In-Reply-To: References: Message-ID: On 21 February 2015 at 04:47, Brett Cannon wrote: > I just realized I actually never committed this change. Assuming no new > objections I'll commit this in the near future (promise this time =). Looks good to me :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Feb 21 15:26:23 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Feb 2015 00:26:23 +1000 Subject: [Python-Dev] TypeError messages In-Reply-To: References: Message-ID: On 21 February 2015 at 00:05, Brett Cannon wrote: > > > On Thu Feb 19 2015 at 5:52:07 PM Serhiy Storchaka > wrote: >> >> Different patterns for TypeError messages are used in the stdlib: >> >> expected X, Y found >> expected X, found Y >> expected X, but Y found >> expected X instance, Y found >> X expected, not Y >> expect X, not Y >> need X, Y found >> X is required, not Y >> Z must be X, not Y >> Z should be X, not Y >> >> and more. >> >> What the pattern is most preferable? > > > My preference is for "expected X, but found Y". Likewise, although as Rob noted, it's sometimes to desirable to use the "Z should be X, not Y" form if the shorter form would be ambiguous. Perhaps this should be a recommendation in both PEP 7 & 8? It's exactly the kind of issue where having a recommended way to do it can save a fair bit of time considering the exact phrasing of error messages, as well as improving the developer experience by providing more consistent feedback on error details. >> Some messages use the article before X or Y. Should the article be used >> or omitted? >> >> Some messages (only in C) truncate actual type name (%.50s, %.80s, >> %.200s, %.500s). Should type name be truncated at all and for how limit? > > I assume this is over some worry of string size blowing out memory > allocation or something? If someone can show that's an actual worry then > fine, otherwise I say don't truncate. This is C, assuming strings are correctly NULL terminated is almost always a recipe for future disaster :) >> Type names newer truncated in TypeError messages raised in Python code. Assuming the "newer" was intended to be "never", yes, that's due to the main concern being with not blowing up in the face of missing terminating NULLs in purported C strings. That concern doesn't exist at the Python level. >> Some messages enclose actual type name with single quotes ('%s', >> '%.200s'). Should type name be quoted? It is uncommon if type name >> contains spaces. > > I agree that type names don't need to be quoted. As a user, I actually appreciate the quotes, because they make the error pattern easier for me to parse. Compare: int expected, but found str float expected, but found int 'int' expected, but found 'str' 'float' expected, but found 'int' The variable information in this pattern (i.e. the types) gets highlighted visually in the second version, as it's in the sections surrounded by single quotes. For me, that helps the structural scaffolding (the "expected, but found" part) to fade more readily into the background as a familiar piece of text that primes my pattern recognition, but doesn't change much across different error messages. Regards, Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sat Feb 21 16:13:33 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 21 Feb 2015 17:13:33 +0200 Subject: [Python-Dev] TypeError messages In-Reply-To: <54E67856.2090806@mrabarnett.plus.com> References: <54E67856.2090806@mrabarnett.plus.com> Message-ID: On 20.02.15 01:57, MRAB wrote: > Messages tend not to be complete sentences anyway, so I think that it > would be fitting to omit articles. Thanks. This post was aroused by your note about articles on the tracker. From storchaka at gmail.com Sat Feb 21 16:17:50 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 21 Feb 2015 17:17:50 +0200 Subject: [Python-Dev] TypeError messages In-Reply-To: <54E75CAD.7010309@trueblade.com> References: <54E75CAD.7010309@trueblade.com> Message-ID: On 20.02.15 18:11, Eric V. Smith wrote: > I asked about this years ago, and was told it was in case the type name > pointer was bad, and to limit the amount of garbage printed. Whether > that's an actual problem or not, I can't say. It seems more likely that > you'd get a segfault, but maybe if it was pointing to reused memory it > could be useful. Thank you. This makes sense and explains why type names are not truncated in Python code. From storchaka at gmail.com Sat Feb 21 16:20:07 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 21 Feb 2015 17:20:07 +0200 Subject: [Python-Dev] TypeError messages In-Reply-To: References: <54E75CAD.7010309@trueblade.com> Message-ID: On 21.02.15 10:03, Guido van Rossum wrote: > IIRC there's a limited buffer used for the formatting. For now formatting buffer is not limited. But other arguments are valid. From storchaka at gmail.com Sat Feb 21 16:35:07 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 21 Feb 2015 17:35:07 +0200 Subject: [Python-Dev] TypeError messages In-Reply-To: References: Message-ID: On 21.02.15 16:26, Nick Coghlan wrote: > Likewise, although as Rob noted, it's sometimes to desirable to use > the "Z should be X, not Y" form if the shorter form would be > ambiguous. Z is not always available. > Perhaps this should be a recommendation in both PEP 7 & 8? It's > exactly the kind of issue where having a recommended way to do it can > save a fair bit of time considering the exact phrasing of error > messages, as well as improving the developer experience by providing > more consistent feedback on error details. It would be great. I'm going to change standard messages in PyArg_Parse* and common converting functions (as PyLong_AsLong and PyObject_GetBuffer) to make them uniform. From rdmurray at bitdance.com Sat Feb 21 16:57:49 2015 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 21 Feb 2015 10:57:49 -0500 Subject: [Python-Dev] TypeError messages In-Reply-To: References: Message-ID: <20150221155750.49F58250F93@webabinitio.net> On Sun, 22 Feb 2015 00:26:23 +1000, Nick Coghlan wrote: > On 21 February 2015 at 00:05, Brett Cannon wrote: > > I agree that type names don't need to be quoted. > > As a user, I actually appreciate the quotes, because they make the > error pattern easier for me to parse. Compare: > > int expected, but found str > float expected, but found int > > 'int' expected, but found 'str' > 'float' expected, but found 'int' It's not a big deal to me either way, but I find the version with the quotes makes me think it is looking for the literal string 'int', but found the literal string 'str', whereas in the first case it seems clear it is looking for objects of that type. Perhaps this is just because I am used to the existing messages, but I think it goes beyond that. --David From stefan_ml at behnel.de Sat Feb 21 17:06:55 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 21 Feb 2015 17:06:55 +0100 Subject: [Python-Dev] TypeError messages In-Reply-To: References: Message-ID: Serhiy Storchaka schrieb am 21.02.2015 um 16:35: > I'm going to change standard messages in PyArg_Parse* > and common converting functions (as PyLong_AsLong and PyObject_GetBuffer) > to make them uniform. *sigh* - and along came another heap of doctest adaptations... Stefan From demianbrecht at gmail.com Sat Feb 21 17:39:20 2015 From: demianbrecht at gmail.com (Demian Brecht) Date: Sat, 21 Feb 2015 08:39:20 -0800 Subject: [Python-Dev] str(IntEnum) In-Reply-To: References: <5D84B808-5AB3-4FD4-94F1-A997548D7353@gmail.com> <5C3F4038-92F6-47DB-A9E8-6A811AB04924@gmail.com> <20150220183602.GA429@tonks> Message-ID: <93B631C3-DC57-4DB7-9DCA-8AFCD6D23FBE@gmail.com> > > On Feb 20, 2015, at 7:03 PM, Ian Cordasco wrote: > I hope this helps. It does, as do the other replies, thanks all. To be clear, my first gripe has stemmed into two related (but very minor) problems: 1. IntEnum.__str__. I understand the reasoning behind the current implementation. Personally I?d still prefer it to be consistent with int (and other types as shown above) and if I wanted to know where it came from, I could use repr(), but I understand that this /was/ previously thought out and is contrary to the decision made. I?m fine with being in the minority (or on my own as it seems in this case) and leaving it alone (with the only caveat being the next point). 2. Consistency of __str__ semantics across the standard library and builtins. I believe that the value of __str__ is something that I, as a user, should be able to infer when using disparate types. Unfortunately, some represent the values while other represent the object themselves, nearly the same problem that __repr__ solves minus the requirement of being a valid Python expression where possible. In my mind, a clear separation between __str__ representing the value of an instance and __repr__ representing the object, or where the value came from (and perhaps /not/ having the auto-fallback) makes sense, but would only be one potential solution to promoting consistency. In any event, there are many larger problems to be solved (that is, if anyone else /does/ consider this a problem) and I very well may be on my own with this thinking. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From solipsis at pitrou.net Sat Feb 21 18:12:37 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 21 Feb 2015 18:12:37 +0100 Subject: [Python-Dev] TypeError messages References: <20150221155750.49F58250F93@webabinitio.net> Message-ID: <20150221181237.39a60d61@fsol> On Sat, 21 Feb 2015 10:57:49 -0500 "R. David Murray" wrote: > On Sun, 22 Feb 2015 00:26:23 +1000, Nick Coghlan wrote: > > On 21 February 2015 at 00:05, Brett Cannon wrote: > > > I agree that type names don't need to be quoted. > > > > As a user, I actually appreciate the quotes, because they make the > > error pattern easier for me to parse. Compare: > > > > int expected, but found str > > float expected, but found int > > > > 'int' expected, but found 'str' > > 'float' expected, but found 'int' > > It's not a big deal to me either way, but I find the version with the > quotes makes me think it is looking for the literal string 'int', but > found the literal string 'str', whereas in the first case it seems > clear it is looking for objects of that type. Perhaps this is just > because I am used to the existing messages, but I think it goes > beyond that. Agreed. Regards Antoine. From solipsis at pitrou.net Sat Feb 21 18:14:22 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 21 Feb 2015 18:14:22 +0100 Subject: [Python-Dev] TypeError messages References: Message-ID: <20150221181422.1062c547@fsol> On Fri, 20 Feb 2015 14:05:11 +0000 Brett Cannon wrote: > On Thu Feb 19 2015 at 5:52:07 PM Serhiy Storchaka > wrote: > > > Different patterns for TypeError messages are used in the stdlib: > > > > expected X, Y found > > expected X, found Y > > expected X, but Y found > > expected X instance, Y found > > X expected, not Y > > expect X, not Y > > need X, Y found > > X is required, not Y > > Z must be X, not Y > > Z should be X, not Y > > > > and more. > > > > What the pattern is most preferable? > > > > My preference is for "expected X, but found Y". If we are busy nitpicking, why are we saying "found Y"? Nothing was *found* by the callee, it just *got* an argument. So it should be "expected X, but got Y". Personally, I think the "but" is superfluous: the contradiction is already implied, so "expected X, got Y" is terser and conveys the meaning just as well. Regards Antoine. From victor.stinner at gmail.com Sat Feb 21 18:52:44 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 21 Feb 2015 18:52:44 +0100 Subject: [Python-Dev] TypeError messages In-Reply-To: References: Message-ID: Le 21 f?vr. 2015 15:27, "Nick Coghlan" a ?crit : > int expected, but found str > float expected, but found int > > 'int' expected, but found 'str' > 'float' expected, but found 'int' I would prefer quotes with the word type: 'float' type expected, but got 'int' type Or no quotes. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From facundobatista at gmail.com Sat Feb 21 19:48:41 2015 From: facundobatista at gmail.com (Facundo Batista) Date: Sat, 21 Feb 2015 15:48:41 -0300 Subject: [Python-Dev] Enum is really serializable using json? Message-ID: We have this issue closed as resolved: http://bugs.python.org/issue18264 It's called "enum.IntEnum is not compatible with JSON serialisation", and it looks that after a long conversation they added proper support for it in Py3.4. However, this is still failing: Python 3.4.2 (default, Oct 8 2014, 13:18:07) [GCC 4.9.1] on linux >>> import enum, json >>> Foo = enum.Enum('Foo', 'bar baz') >>> json.dumps(Foo.bar) Traceback (most recent call last): ... TypeError: is not JSON serializable Am I failing to understand the bug tracker saying the patch was applied for Py3.4? (thought it was more polite to ask here than to reopen the whole thread in the issue tracker) Thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From storchaka at gmail.com Sat Feb 21 19:49:26 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 21 Feb 2015 20:49:26 +0200 Subject: [Python-Dev] Generate all Argument Clinic code into separate files Message-ID: Currently for some files converted to use Argument Clinic the generated code is written into a separate file, for other files it is inlined. I'm going to make a number of small enhancement to Argument Clinic to generate faster code for parsing arguments (see for example issue23492). Every such step will produce large diffs for generated code and will create code churn if it is inlined and mixed up with handwritten code. It would be better when only generated files will be changed. So I suggest to move all inlined generated code in separate file. What are you think? From benjamin at python.org Sat Feb 21 19:50:41 2015 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 21 Feb 2015 13:50:41 -0500 Subject: [Python-Dev] Enum is really serializable using json? In-Reply-To: References: Message-ID: <1424544641.3310894.230580221.32DB84BC@webmail.messagingengine.com> On Sat, Feb 21, 2015, at 13:48, Facundo Batista wrote: > We have this issue closed as resolved: > > http://bugs.python.org/issue18264 > > It's called "enum.IntEnum is not compatible with JSON serialisation", > and it looks that after a long conversation they added proper support > for it in Py3.4. > > However, this is still failing: > > Python 3.4.2 (default, Oct 8 2014, 13:18:07) > [GCC 4.9.1] on linux > >>> import enum, json > >>> Foo = enum.Enum('Foo', 'bar baz') > >>> json.dumps(Foo.bar) > Traceback (most recent call last): > ... > TypeError: is not JSON serializable > > Am I failing to understand the bug tracker saying the patch was > applied for Py3.4? As the issue title suggests, IntEnum but not Enum supports JSON serialization. From facundobatista at gmail.com Sat Feb 21 20:10:17 2015 From: facundobatista at gmail.com (Facundo Batista) Date: Sat, 21 Feb 2015 16:10:17 -0300 Subject: [Python-Dev] Enum is really serializable using json? In-Reply-To: <1424544641.3310894.230580221.32DB84BC@webmail.messagingengine.com> References: <1424544641.3310894.230580221.32DB84BC@webmail.messagingengine.com> Message-ID: On Sat, Feb 21, 2015 at 3:50 PM, Benjamin Peterson wrote: >> Am I failing to understand the bug tracker saying the patch was >> applied for Py3.4? > > As the issue title suggests, IntEnum but not Enum supports JSON > serialization. Arghhh.. stupid of me. Sorry for the noise, and thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From storchaka at gmail.com Sat Feb 21 20:28:21 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 21 Feb 2015 21:28:21 +0200 Subject: [Python-Dev] Prefixes and namespaces Message-ID: /* Namespaces are one honking great idea -- let's do more of those! */ There are two ways to avoid name conflicts: prefixes and namespaces. Programming languages that lacks namespaces (such as C) need to use prefixes. For example: PROTOCOL_SSLv2, PROTOCOL_SSLv3, PROTOCOL_SSLv23. Python used the same prefixed names when reflect C constants to module level Python globals. But when convert integer constants to IntEnum, is it needed to preserve common prefix? Or may be we can drop it, because enum class name plays its role? class Protocol(IntEnum): PROTOCOL_SSLv2 = ... PROTOCOL_SSLv3 = ... PROTOCOL_SSLv23 = ... or class Protocol(IntEnum): SSLv2 = ... SSLv3 = ... SSLv23 = ... ? Protocol.PROTOCOL_SSLv2 or Protocol.SSLv2? From graffatcolmingov at gmail.com Sat Feb 21 20:49:47 2015 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Sat, 21 Feb 2015 13:49:47 -0600 Subject: [Python-Dev] Prefixes and namespaces In-Reply-To: References: Message-ID: On Sat, Feb 21, 2015 at 1:28 PM, Serhiy Storchaka wrote: > /* Namespaces are one honking great idea -- let's do more of those! */ > > There are two ways to avoid name conflicts: prefixes and namespaces. > Programming languages that lacks namespaces (such as C) need to use > prefixes. For example: PROTOCOL_SSLv2, PROTOCOL_SSLv3, PROTOCOL_SSLv23. > Python used the same prefixed names when reflect C constants to module level > Python globals. But when convert integer constants to IntEnum, is it needed > to preserve common prefix? Or may be we can drop it, because enum class name > plays its role? > > class Protocol(IntEnum): > PROTOCOL_SSLv2 = ... > PROTOCOL_SSLv3 = ... > PROTOCOL_SSLv23 = ... > > or > > class Protocol(IntEnum): > SSLv2 = ... > SSLv3 = ... > SSLv23 = ... > > ? Protocol.PROTOCOL_SSLv2 or Protocol.SSLv2? So I like the latter (Protocol.SSLv2) but would qualify that with the request that ssl.PROTOCOL_SSLv2 continue to work until Python 2 is dead and libraries like requests, urllib3, httplib2, etc. no longer need to support those versions. From ethan at stoneleaf.us Sat Feb 21 21:09:04 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 21 Feb 2015 12:09:04 -0800 Subject: [Python-Dev] Prefixes and namespaces In-Reply-To: References: Message-ID: <54E8E5E0.3080006@stoneleaf.us> On 02/21/2015 11:49 AM, Ian Cordasco wrote: > On Sat, Feb 21, 2015 at 1:28 PM, Serhiy Storchaka wrote: >> ? Protocol.PROTOCOL_SSLv2 or Protocol.SSLv2 ? > > So I like the latter (Protocol.SSLv2) but would qualify that with the > request that ssl.PROTOCOL_SSLv2 continue to work until Python 2 is > dead and libraries like requests, urllib3, httplib2, etc. no longer > need to support those versions. Good idea, and easy enough to do since Enum supports aliases. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From storchaka at gmail.com Sat Feb 21 21:09:15 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 21 Feb 2015 22:09:15 +0200 Subject: [Python-Dev] Prefixes and namespaces In-Reply-To: References: Message-ID: On 21.02.15 21:49, Ian Cordasco wrote: > So I like the latter (Protocol.SSLv2) but would qualify that with the > request that ssl.PROTOCOL_SSLv2 continue to work until Python 2 is > dead and libraries like requests, urllib3, httplib2, etc. no longer > need to support those versions. Of course, ssl.PROTOCOL_SSLv2 will continue to work until Python 2.7 and 3.4 are dead. From solipsis at pitrou.net Sat Feb 21 21:27:04 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 21 Feb 2015 21:27:04 +0100 Subject: [Python-Dev] Prefixes and namespaces References: Message-ID: <20150221212704.65374892@fsol> On Sat, 21 Feb 2015 21:28:21 +0200 Serhiy Storchaka wrote: > /* Namespaces are one honking great idea -- let's do more of those! */ > > There are two ways to avoid name conflicts: prefixes and namespaces. > Programming languages that lacks namespaces (such as C) need to use > prefixes. For example: PROTOCOL_SSLv2, PROTOCOL_SSLv3, PROTOCOL_SSLv23. > Python used the same prefixed names when reflect C constants to module > level Python globals. Python still uses the names, and they are still globals. There's no reason to change that. The enum is simply there to improve debugging when printing the values. Regards Antoine. From brett at python.org Sat Feb 21 19:03:05 2015 From: brett at python.org (Brett Cannon) Date: Sat, 21 Feb 2015 18:03:05 +0000 Subject: [Python-Dev] TypeError messages References: <20150221181422.1062c547@fsol> Message-ID: On Sat Feb 21 2015 at 12:15:25 PM Antoine Pitrou wrote: > On Fri, 20 Feb 2015 14:05:11 +0000 > Brett Cannon wrote: > > On Thu Feb 19 2015 at 5:52:07 PM Serhiy Storchaka > > wrote: > > > > > Different patterns for TypeError messages are used in the stdlib: > > > > > > expected X, Y found > > > expected X, found Y > > > expected X, but Y found > > > expected X instance, Y found > > > X expected, not Y > > > expect X, not Y > > > need X, Y found > > > X is required, not Y > > > Z must be X, not Y > > > Z should be X, not Y > > > > > > and more. > > > > > > What the pattern is most preferable? > > > > > > > My preference is for "expected X, but found Y". > > If we are busy nitpicking, why are we saying "found Y"? Nothing was > *found* by the callee, it just *got* an argument. > > So it should be "expected X, but got Y". > > Personally, I think the "but" is superfluous: the contradiction is > already implied, so "expected X, got Y" is terser and conveys the > meaning just as well. > I'm also fine with the terser version. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Feb 21 21:58:10 2015 From: brett at python.org (Brett Cannon) Date: Sat, 21 Feb 2015 20:58:10 +0000 Subject: [Python-Dev] Generate all Argument Clinic code into separate files References: Message-ID: On Sat Feb 21 2015 at 1:50:32 PM Serhiy Storchaka wrote: > Currently for some files converted to use Argument Clinic the generated > code is written into a separate file, for other files it is inlined. > > I'm going to make a number of small enhancement to Argument Clinic to > generate faster code for parsing arguments (see for example issue23492). > Every such step will produce large diffs for > generated code and will create code churn if it is inlined and mixed up > with handwritten code. It would be better when only generated files will > be changed. So I suggest to move all inlined generated code in separate > file. What are you think? > +1 to moving to a separate file for all .c files. Might be painful now but the long-terms benefits are worth it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From benhoyt at gmail.com Sat Feb 21 22:22:33 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Sat, 21 Feb 2015 16:22:33 -0500 Subject: [Python-Dev] [Python-checkins] cpython: Issue #23152: Implement _Py_fstat() to support files larger than 2 GB on Message-ID: When merging some changes while working on scandir, I noticed a minor issue with this commit: https://hg.python.org/cpython/rev/4f6f4aa0d80f The definition of "struct win32_stat" has been moved to fileutils.h and renamed to "struct _Py_stat_struct", which is fine -- however, the old "struct win32_stat" definition is still present (but unused) in posixmodule.c. So I think the old "struct win32_stat { ... }" definition can simply be removed from posixmodule.c now. Also, unrelated to this commit, I notice the _Py_attribute_data_to_stat function (was attribute_data_to_stat) can't fail and always returns 0, and all callers ignore its return value anyway. Can it be changed to return void? -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sat Feb 21 22:39:32 2015 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 21 Feb 2015 21:39:32 +0000 Subject: [Python-Dev] TypeError messages In-Reply-To: <20150221181422.1062c547@fsol> References: <20150221181422.1062c547@fsol> Message-ID: <54E8FB14.5050600@mrabarnett.plus.com> On 2015-02-21 17:14, Antoine Pitrou wrote: > On Fri, 20 Feb 2015 14:05:11 +0000 > Brett Cannon wrote: >> On Thu Feb 19 2015 at 5:52:07 PM Serhiy Storchaka >> wrote: >> >> > Different patterns for TypeError messages are used in the stdlib: >> > >> > expected X, Y found >> > expected X, found Y >> > expected X, but Y found >> > expected X instance, Y found >> > X expected, not Y >> > expect X, not Y >> > need X, Y found >> > X is required, not Y >> > Z must be X, not Y >> > Z should be X, not Y >> > >> > and more. >> > >> > What the pattern is most preferable? >> > >> >> My preference is for "expected X, but found Y". > > If we are busy nitpicking, why are we saying "found Y"? Nothing was > *found* by the callee, it just *got* an argument. > Well, it depends on the reason for the message. If you're passing an argument, then 'found' is the wrong word, but if you're parsing, say, a regex, then 'got' is the wrong word. > So it should be "expected X, but got Y". > > Personally, I think the "but" is superfluous: the contradiction is > already implied, so "expected X, got Y" is terser and conveys the > meaning just as well. > If you wanted a message to cover both argument-passing and parsing, then "expected Y, not Y" would do. From solipsis at pitrou.net Sat Feb 21 22:44:01 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 21 Feb 2015 22:44:01 +0100 Subject: [Python-Dev] TypeError messages References: <20150221181422.1062c547@fsol> <54E8FB14.5050600@mrabarnett.plus.com> Message-ID: <20150221224401.0be51e46@fsol> On Sat, 21 Feb 2015 21:39:32 +0000 MRAB wrote: > On 2015-02-21 17:14, Antoine Pitrou wrote: > > On Fri, 20 Feb 2015 14:05:11 +0000 > > Brett Cannon wrote: > >> On Thu Feb 19 2015 at 5:52:07 PM Serhiy Storchaka > >> wrote: > >> > >> > Different patterns for TypeError messages are used in the stdlib: > >> > > >> > expected X, Y found > >> > expected X, found Y > >> > expected X, but Y found > >> > expected X instance, Y found > >> > X expected, not Y > >> > expect X, not Y > >> > need X, Y found > >> > X is required, not Y > >> > Z must be X, not Y > >> > Z should be X, not Y > >> > > >> > and more. > >> > > >> > What the pattern is most preferable? > >> > > >> > >> My preference is for "expected X, but found Y". > > > > If we are busy nitpicking, why are we saying "found Y"? Nothing was > > *found* by the callee, it just *got* an argument. > > > Well, it depends on the reason for the message. > > If you're passing an argument, then 'found' is the wrong word, but if > you're parsing, say, a regex, then 'got' is the wrong word. I don't think parsing would raise a TypeError, would it? Regards Antoine. From brett at python.org Sat Feb 21 23:11:54 2015 From: brett at python.org (Brett Cannon) Date: Sat, 21 Feb 2015 22:11:54 +0000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #23152: Implement _Py_fstat() to support files larger than 2 GB on References: Message-ID: On Sat Feb 21 2015 at 4:23:16 PM Ben Hoyt wrote: > When merging some changes while working on scandir, I noticed a minor > issue with this commit: > > https://hg.python.org/cpython/rev/4f6f4aa0d80f > > The definition of "struct win32_stat" has been moved to fileutils.h and > renamed to "struct _Py_stat_struct", which is fine -- however, the old > "struct win32_stat" definition is still present (but unused) in > posixmodule.c. > > So I think the old "struct win32_stat { ... }" definition can simply be > removed from posixmodule.c now. > I don't think win32_stat is part of the stable ABI so as long as everything keeps working then I don't see why it needs to stick around. > > Also, unrelated to this commit, I notice the _Py_attribute_data_to_stat > function (was attribute_data_to_stat) can't fail and always returns 0, and > all callers ignore its return value anyway. Can it be changed to return > void? > Don't see why not since it's a private API. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat Feb 21 23:13:49 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 22 Feb 2015 00:13:49 +0200 Subject: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat In-Reply-To: <20150221180424.3865.41447@psf.io> References: <20150221180424.3865.41447@psf.io> Message-ID: On 21.02.15 20:04, steve.dower wrote: > https://hg.python.org/cpython/rev/307713759a62 > changeset: 94720:307713759a62 > parent: 94718:4f6f4aa0d80f > user: Steve Dower > date: Sat Feb 21 10:04:10 2015 -0800 > summary: > Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat What about time_t_to_FILE_TIME? From Steve.Dower at microsoft.com Sun Feb 22 00:28:51 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 21 Feb 2015 23:28:51 +0000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #23152: Implement _Py_fstat() to support files larger than 2 GB on In-Reply-To: References: , Message-ID: Agreed. I've made both of these changes. Thanks for the suggestions Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Brett Cannon Sent: ?2/?21/?2015 14:13 To: Ben Hoyt; Python-Dev Subject: Re: [Python-Dev] [Python-checkins] cpython: Issue #23152: Implement _Py_fstat() to support files larger than 2 GB on On Sat Feb 21 2015 at 4:23:16 PM Ben Hoyt > wrote: When merging some changes while working on scandir, I noticed a minor issue with this commit: https://hg.python.org/cpython/rev/4f6f4aa0d80f The definition of "struct win32_stat" has been moved to fileutils.h and renamed to "struct _Py_stat_struct", which is fine -- however, the old "struct win32_stat" definition is still present (but unused) in posixmodule.c. So I think the old "struct win32_stat { ... }" definition can simply be removed from posixmodule.c now. I don't think win32_stat is part of the stable ABI so as long as everything keeps working then I don't see why it needs to stick around. Also, unrelated to this commit, I notice the _Py_attribute_data_to_stat function (was attribute_data_to_stat) can't fail and always returns 0, and all callers ignore its return value anyway. Can it be changed to return void? Don't see why not since it's a private API. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sun Feb 22 00:26:06 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 21 Feb 2015 23:26:06 +0000 Subject: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat In-Reply-To: References: <20150221180424.3865.41447@psf.io>, Message-ID: Thanks! Fixed now. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Serhiy Storchaka Sent: ?2/?21/?2015 14:14 To: python-dev at python.org Subject: Re: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat On 21.02.15 20:04, steve.dower wrote: > https://hg.python.org/cpython/rev/307713759a62 > changeset: 94720:307713759a62 > parent: 94718:4f6f4aa0d80f > user: Steve Dower > date: Sat Feb 21 10:04:10 2015 -0800 > summary: > Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat What about time_t_to_FILE_TIME? _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Sun Feb 22 00:52:33 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 21 Feb 2015 23:52:33 +0000 Subject: [Python-Dev] TypeError messages In-Reply-To: <20150221181422.1062c547@fsol> References: <20150221181422.1062c547@fsol> Message-ID: <54E91A41.7010401@btinternet.com> On 21/02/2015 17:14, Antoine Pitrou wrote: > On Fri, 20 Feb 2015 14:05:11 +0000 > Brett Cannon wrote: >> On Thu Feb 19 2015 at 5:52:07 PM Serhiy Storchaka >> wrote: >> >>> Different patterns for TypeError messages are used in the stdlib: >>> >>> expected X, Y found >>> expected X, found Y >>> expected X, but Y found >>> expected X instance, Y found >>> X expected, not Y >>> expect X, not Y >>> need X, Y found >>> X is required, not Y >>> Z must be X, not Y >>> Z should be X, not Y >>> >>> and more. >>> >>> What the pattern is most preferable? >>> >> My preference is for "expected X, but found Y". > If we are busy nitpicking, why are we saying "found Y"? Nothing was > *found* by the callee, it just *got* an argument. > > So it should be "expected X, but got Y". > > Personally, I think the "but" is superfluous: the contradiction is > already implied, so "expected X, got Y" is terser and conveys the > meaning just as well. > > Regards > > Antoine. +1 Rob Cliffe From rob.cliffe at btinternet.com Sun Feb 22 00:51:48 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 21 Feb 2015 23:51:48 +0000 Subject: [Python-Dev] TypeError messages In-Reply-To: <20150221155750.49F58250F93@webabinitio.net> References: <20150221155750.49F58250F93@webabinitio.net> Message-ID: <54E91A14.5070609@btinternet.com> On 21/02/2015 15:57, R. David Murray wrote: > On Sun, 22 Feb 2015 00:26:23 +1000, Nick Coghlan wrote: >> On 21 February 2015 at 00:05, Brett Cannon wrote: >>> I agree that type names don't need to be quoted. >> As a user, I actually appreciate the quotes, because they make the >> error pattern easier for me to parse. Compare: >> >> int expected, but found str >> float expected, but found int >> >> 'int' expected, but found 'str' >> 'float' expected, but found 'int' > It's not a big deal to me either way, but I find the version with the > quotes makes me think it is looking for the literal string 'int', but > found the literal string 'str', whereas in the first case it seems > clear it is looking for objects of that type. Perhaps this is just > because I am used to the existing messages, but I think it goes > beyond that. > > +1. FWIW, I feel the same as David (and Antoine). Rob Cliffe From rob.cliffe at btinternet.com Sun Feb 22 01:02:43 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sun, 22 Feb 2015 00:02:43 +0000 Subject: [Python-Dev] TypeError messages In-Reply-To: <54E8FB14.5050600@mrabarnett.plus.com> References: <20150221181422.1062c547@fsol> <54E8FB14.5050600@mrabarnett.plus.com> Message-ID: <54E91CA3.7050306@btinternet.com> On 21/02/2015 21:39, MRAB wrote: > On 2015-02-21 17:14, Antoine Pitrou wrote: >> On Fri, 20 Feb 2015 14:05:11 +0000 >> Brett Cannon wrote: >>> On Thu Feb 19 2015 at 5:52:07 PM Serhiy Storchaka >>> wrote: >>> >>> > Different patterns for TypeError messages are used in the stdlib: >>> > >>> > expected X, Y found >>> > expected X, found Y >>> > expected X, but Y found >>> > expected X instance, Y found >>> > X expected, not Y >>> > expect X, not Y >>> > need X, Y found >>> > X is required, not Y >>> > Z must be X, not Y >>> > Z should be X, not Y >>> > >>> > and more. >>> > >>> > What the pattern is most preferable? >>> > >>> >>> My preference is for "expected X, but found Y". >> >> If we are busy nitpicking, why are we saying "found Y"? Nothing was >> *found* by the callee, it just *got* an argument. >> > Well, it depends on the reason for the message. > > If you're passing an argument, then 'found' is the wrong word, but if > you're parsing, say, a regex, then 'got' is the wrong word. > >> So it should be "expected X, but got Y". >> >> Personally, I think the "but" is superfluous: the contradiction is >> already implied, so "expected X, got Y" is terser and conveys the >> meaning just as well. >> > If you wanted a message to cover both argument-passing and parsing, > then "expected Y, not Y" would do. Assuming you meant "expected Y, not X": +1 Perhaps better than all other suggestions so far. From ncoghlan at gmail.com Sun Feb 22 03:56:29 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Feb 2015 12:56:29 +1000 Subject: [Python-Dev] Enum is really serializable using json? In-Reply-To: References: <1424544641.3310894.230580221.32DB84BC@webmail.messagingengine.com> Message-ID: On 22 February 2015 at 05:10, Facundo Batista wrote: > On Sat, Feb 21, 2015 at 3:50 PM, Benjamin Peterson wrote: > >>> Am I failing to understand the bug tracker saying the patch was >>> applied for Py3.4? >> >> As the issue title suggests, IntEnum but not Enum supports JSON >> serialization. > > Arghhh.. stupid of me. Sorry for the noise, and thanks! I don't know if we actually clearly explain anywhere why that's the case, though. As I see it, the core problem is that when mapping an Enum to JSON, you may either do it by name (so it becomes a string on the JSON side) or by value (so you lose the fact that it's an enum, but can work with protocols that use integers rather than strings). There's no "right" answer in general as to which of those is correct - the one you want will depend on exactly what protocols you're working with. By contrast, for IntEnum, we want that to be an as-close-to-drop-in-as-is-feasible replacement for existing ints, so there's a clear correct answer: we need to serialise it to JSON as an integer, and lose the additional enum details. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 22 04:03:43 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Feb 2015 13:03:43 +1000 Subject: [Python-Dev] Generate all Argument Clinic code into separate files In-Reply-To: References: Message-ID: On 22 February 2015 at 06:58, Brett Cannon wrote: > > > On Sat Feb 21 2015 at 1:50:32 PM Serhiy Storchaka > wrote: >> >> Currently for some files converted to use Argument Clinic the generated >> code is written into a separate file, for other files it is inlined. >> >> I'm going to make a number of small enhancement to Argument Clinic to >> generate faster code for parsing arguments (see for example issue23492). >> Every such step will produce large diffs for >> generated code and will create code churn if it is inlined and mixed up >> with handwritten code. It would be better when only generated files will >> be changed. So I suggest to move all inlined generated code in separate >> file. What are you think? > > > +1 to moving to a separate file for all .c files. Might be painful now but > the long-terms benefits are worth it. Yeah, agreed. Originally I was a fan of having everything inline so I could see them while I was working on the code, but I eventually changed my mind to favour making it a clearer build step with a separate generated file. I suspect it was a matter of starting to trust AC to do the right thing, so having it implicitly asking me to check its work all the time ultimately become annoying rather than reassuring :) I assume some parts, like the header for the implementation function, will always need to be generated inline in the hand-edited implementation file, though. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sun Feb 22 07:56:40 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 22 Feb 2015 08:56:40 +0200 Subject: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat In-Reply-To: References: <20150221180424.3865.41447@psf.io>, Message-ID: On 22.02.15 01:26, Steve Dower wrote: > Thanks! Fixed now. Clinic removes the declaration of _Py_time_t_to_FILE_TIME. From storchaka at gmail.com Sun Feb 22 12:10:36 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 22 Feb 2015 13:10:36 +0200 Subject: [Python-Dev] Generate all Argument Clinic code into separate files In-Reply-To: References: Message-ID: On 22.02.15 05:03, Nick Coghlan wrote: > On 22 February 2015 at 06:58, Brett Cannon wrote: >> +1 to moving to a separate file for all .c files. Might be painful now but >> the long-terms benefits are worth it. > > Yeah, agreed. Originally I was a fan of having everything inline so I > could see them while I was working on the code, but I eventually > changed my mind to favour making it a clearer build step with a > separate generated file. I suspect it was a matter of starting to > trust AC to do the right thing, so having it implicitly asking me to > check its work all the time ultimately become annoying rather than > reassuring :) OK. Opened an issue: https://bugs.python.org/issue23501 From Steve.Dower at microsoft.com Sun Feb 22 15:12:54 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 22 Feb 2015 14:12:54 +0000 Subject: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat In-Reply-To: References: <20150221180424.3865.41447@psf.io>, , Message-ID: Why does it do that? Top-posted from my Windows Phone ________________________________ From: Serhiy Storchaka Sent: ?2/?21/?2015 22:57 To: python-dev at python.org Subject: Re: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat On 22.02.15 01:26, Steve Dower wrote: > Thanks! Fixed now. Clinic removes the declaration of _Py_time_t_to_FILE_TIME. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Feb 22 16:28:34 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 22 Feb 2015 17:28:34 +0200 Subject: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat In-Reply-To: References: <20150221180424.3865.41447@psf.io>, , Message-ID: On 22.02.15 16:12, Steve Dower wrote: > Why does it do that? Because it is in the section of generated code. From Steve.Dower at microsoft.com Sun Feb 22 16:45:28 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 22 Feb 2015 15:45:28 +0000 Subject: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat In-Reply-To: References: <20150221180424.3865.41447@psf.io>, , , Message-ID: Ah, that makes sense. Feel free to move it up a few lines (or into a header if that's preferred) if it's bothering you. Otherwise I can do it in about 6-8 hours. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Serhiy Storchaka Sent: ?2/?22/?2015 7:29 To: python-dev at python.org Subject: Re: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat On 22.02.15 16:12, Steve Dower wrote: > Why does it do that? Because it is in the section of generated code. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Feb 23 10:57:42 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 23 Feb 2015 11:57:42 +0200 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? Message-ID: See topic "Unrecognized backslash escapes in string literals" in Python list [1]. I agree that this is a problem, especially for novices (but even experience users can make a typo). May be emit SyntaxWarning on unrecognized backslash escapes? An exception is already raised on invalid octal or hexadecimal escapes. '\x' is syntax error, not two characters '\\' and 'x'. [1] http://comments.gmane.org/gmane.comp.python.general/772455 From guido at python.org Mon Feb 23 16:44:10 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Feb 2015 07:44:10 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: Message-ID: I think that's a bit too strong. This has been unquestionably valid, correct Python -- it was an intentional feature from the start. It may not have turned out great, but I think that before warning loudly about every instance of this we should have a silent deprecation (which you can turn into a visible warning with a command-line flag or a warnings filter). And we should have agreement that we're eventually going to make it a syntax error. On Mon, Feb 23, 2015 at 1:57 AM, Serhiy Storchaka wrote: > See topic "Unrecognized backslash escapes in string literals" in Python > list [1]. I agree that this is a problem, especially for novices (but even > experience users can make a typo). May be emit SyntaxWarning on > unrecognized backslash escapes? An exception is already raised on invalid > octal or hexadecimal escapes. '\x' is syntax error, not two characters '\\' > and 'x'. > > [1] http://comments.gmane.org/gmane.comp.python.general/772455 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Feb 23 16:54:57 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Feb 2015 02:54:57 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: Message-ID: On Tue, Feb 24, 2015 at 2:44 AM, Guido van Rossum wrote: > I think that's a bit too strong. This has been unquestionably valid, correct > Python -- it was an intentional feature from the start. It may not have > turned out great, but I think that before warning loudly about every > instance of this we should have a silent deprecation (which you can turn > into a visible warning with a command-line flag or a warnings filter). And > we should have agreement that we're eventually going to make it a syntax > error. Is it at all possible for this to be introduced in the 2.x line, or is the entire concept of a deprecation period one that has to start with a minor version? If it's never going to happen in 2.x, I'll raise this as yet another reason to get the course and all our students migrated to 3.x, but on the flip side, it means that we absolutely can't get the benefit until that jump is made. ChrisA From brett at python.org Mon Feb 23 17:12:48 2015 From: brett at python.org (Brett Cannon) Date: Mon, 23 Feb 2015 16:12:48 +0000 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? References: Message-ID: On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote: > On Tue, Feb 24, 2015 at 2:44 AM, Guido van Rossum > wrote: > > I think that's a bit too strong. This has been unquestionably valid, > correct > > Python -- it was an intentional feature from the start. It may not have > > turned out great, but I think that before warning loudly about every > > instance of this we should have a silent deprecation (which you can turn > > into a visible warning with a command-line flag or a warnings filter). > And > > we should have agreement that we're eventually going to make it a syntax > > error. > > Is it at all possible for this to be introduced in the 2.x line, or is > the entire concept of a deprecation period one that has to start with > a minor version? > Starts with a minor version. > > If it's never going to happen in 2.x, I'll raise this as yet another reason to get the course and all our students migrated to 3.x, but on > the flip side, it means that we absolutely can't get the benefit until > that jump is made. > Never going to happen in 2.x.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Feb 23 17:15:23 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Feb 2015 03:15:23 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: Message-ID: On Tue, Feb 24, 2015 at 3:12 AM, Brett Cannon wrote: > > > On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote: >> >> On Tue, Feb 24, 2015 at 2:44 AM, Guido van Rossum >> wrote: >> > I think that's a bit too strong. This has been unquestionably valid, >> > correct >> > Python -- it was an intentional feature from the start. It may not have >> > turned out great, but I think that before warning loudly about every >> > instance of this we should have a silent deprecation (which you can turn >> > into a visible warning with a command-line flag or a warnings filter). >> > And >> > we should have agreement that we're eventually going to make it a syntax >> > error. >> >> Is it at all possible for this to be introduced in the 2.x line, or is >> the entire concept of a deprecation period one that has to start with >> a minor version? > > > Starts with a minor version. > >> >> >> If it's never going to happen in 2.x, I'll raise this as yet another >> >> reason to get the course and all our students migrated to 3.x, but on >> the flip side, it means that we absolutely can't get the benefit until >> that jump is made. > > > Never going to happen in 2.x.. Thanks, thought that'd be the case but figured I may as well ask. ChrisA From ethan at stoneleaf.us Mon Feb 23 18:01:31 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 23 Feb 2015 09:01:31 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: Message-ID: <54EB5CEB.90305@stoneleaf.us> On 02/23/2015 08:12 AM, Brett Cannon wrote: > On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote: >> Is it at all possible for this to be introduced in the 2.x line [...] > > Starts with a minor version. Isn't there a -3 switch or something like that in 2.7.x to trigger warnings/errors to help port to 3.x? Seems like this kind of warning could go there. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From blaxxton at yahoo.com Mon Feb 23 18:09:51 2015 From: blaxxton at yahoo.com (Blaxton) Date: Mon, 23 Feb 2015 17:09:51 +0000 (UTC) Subject: [Python-Dev] Removing Python Binaries with make command In-Reply-To: <1857446440.8109273.1424709756765.JavaMail.yahoo@mail.yahoo.com> References: <1424267868.788959.229212373.496F2381@webmail.messagingengine.com> <1857446440.8109273.1424709756765.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1942879138.8132646.1424711391492.JavaMail.yahoo@mail.yahoo.com> Hi? I have installed Python3 with running ./configure , make and then make installand now can't find an option in Makefile to remove the binaries with make command. Is there a way to cleanly remove Python3 Binaries after compiling on a system ? ThanksDB -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Feb 23 18:29:09 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Feb 2015 09:29:09 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54EB5CEB.90305@stoneleaf.us> References: <54EB5CEB.90305@stoneleaf.us> Message-ID: On Mon, Feb 23, 2015 at 9:01 AM, Ethan Furman wrote: > On 02/23/2015 08:12 AM, Brett Cannon wrote: > > On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote: > > >> Is it at all possible for this to be introduced in the 2.x line [...] > > > > Starts with a minor version. > > Isn't there a -3 switch or something like that in 2.7.x to trigger > warnings/errors to help port to 3.x? Seems like this > kind of warning could go there. > If we agree it will be a syntax error in 3.x eventually. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Feb 23 18:45:08 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Feb 2015 17:45:08 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support Message-ID: The discussion on PEP 441 (started in thread https://mail.python.org/pipermail/python-dev/2015-February/138277.html and on the issue tracker at http://bugs.python.org/issue23491) seems to have mostly settled down. I don't think there are any outstanding tasks, so I'm happy that the PEP is ready for pronouncement. I've included the latest copy of the PEP inline below, for reference. (Can I also add a gentle reminder that PEP 486 is ready for pronouncement? It's be nice to get these into 3.5 alpha 2, which is due at the end of next week...) Paul PEP: 441 Title: Improving Python ZIP Application Support Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth , Paul Moore Discussions-To: https://mail.python.org/pipermail/python-dev/2015-February/138277.html Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 March 2013 Post-History: 30 March 2013, 1 April 2013, 16 February 2015 Improving Python ZIP Application Support ======================================== Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6 [1]_. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the ``__main__`` module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules. This feature is not as popular as it should be mainly because it was not promoted as part of Python 2.6 [2]_, so that it is relatively unknown, but also because the Windows installer does not register a file extension (other than ``.py``) for this format of file, to associate with the launcher. This PEP proposes to fix these problems by re-publicising the feature, defining the ``.pyz`` and ``.pyzw`` extensions as "Python ZIP Applications" and "Windowed Python ZIP Applications", and providing some simple tooling to manage the format. A New Python ZIP Application Extension ====================================== The terminology "Python Zip Application" will be the formal term used for a zip-format archive that contains Python code in a form that can be directly executed by Python (specifically, it must have a ``__main__.py`` file in the root directory of the archive). The extension ``.pyz`` will be formally associated with such files. The Python 3.5 installer will associate ``.pyz`` and ``.pyzw`` "Python Zip Applications" with the platform launcher so they can be executed. A ``.pyz`` archive is a console application and a ``.pyzw`` archive is a windowed application, indicating whether the console should appear when running the app. On Unix, it would be ideal if the ``.pyz`` extension and the name "Python Zip Application" were registered (in the mime types database?). However, such an association is out of scope for this PEP. Python Zip applications can be prefixed with a ``#!`` line pointing to the correct Python interpreter and an optional explanation:: #!/usr/bin/env python3 # Python application packed with zipapp module (binary contents of archive) On Unix, this allows the OS to run the file with the correct interpreter, via the standard "shebang" support. On Windows, the Python launcher implements shebang support. However, it is always possible to execute a ``.pyz`` application by supplying the filename to the Python interpreter directly. As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work. Minimal Tooling: The zipapp Module ================================== This PEP also proposes including a module for working with these archives. The module will contain functions for working with Python zip application archives, and a command line interface (via ``python -m zipapp``) for their creation and manipulation. More complete tools for managing Python Zip Applications are encouraged as 3rd party applications on PyPI. Currently, pyyzer [5]_ and pex [6]_ are two tools known to exist. Module Interface ---------------- The zipapp module will provide the following functions: ``pack(target, directory, interpreter=None, main=None)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Writes an application archive called *target*, containing the contents of *directory*. If *interpreter* is specified, it will be written to the start of the archive as a shebang line and the file will be made executable (if no interpreter is specified, the shebang line will be omitted). If the directory contains no ``__main__.py`` file, the function will construct a ``__main__.py`` which calls the function specified in the *main* argument (which should be in the form ``'pkg.mod:fn'``). It is an error to specify *main* if the directory contains a ``__main__.py``, or to omit *main* when there is no ``__main__.py`` (as that will result in an archive which has no main function and so cannot be executed). ``get_interpreter(archive)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Returns the interpreter specified in the shebang line of the *archive*. If there is no shebang, the function returns ``None``. ``set_interpreter(archive, new_archive, interpreter=None)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Modifies the *archive*'s shebang line to contain the specified interpreter, and writes the updated archive to *new_archive*. If the *interpreter* is ``None``, removes the shebang line. Command Line Usage ------------------ The zipapp module can be run with the python ``-m`` flag. The command line interface is as follows:: python -m zipapp [options] directory Create an archive from the contents of the given directory. By default, an archive will be created with the same name as the source directory, with a .pyz extension. The following options can be specified: -o archive / --output archive The destination archive will have the specified name. -p interpreter / --python interpreter The given interpreter will be written to the shebang line of the archive. If this option is not given, the archive will have no shebang line. -m pkg.mod:fn / --main pkg.mod:fn The source directory must not have a __main__.py file. The archiver will write a __main__.py file into the target which calls fn from the module pkg.mod. The behaviour of the command line interface matches that of ``zipapp.pack()``. As noted, the archives are standard zip files, and so can be unpacked using any standard ZIP utility or Python's zipfile module. FAQ --- Are you sure a standard ZIP utility can handle ``#!`` at the beginning? Absolutely. The zipfile specification allows for arbitrary data to be prepended to a zipfile. This feature is commonly used by "self-extracting zip" programs. If your archive program can't handle this, it is a bug in your archive program. Isn't zipapp just a very thin wrapper over the zipfile module? Yes. If you prefer to build your own Python zip application archives using other tools, they will work just as well. The zipapp module is a convenience, nothing more. Why not use just use a ``.zip`` or ``.py`` extension? Users expect a ``.zip`` file to be opened with an archive tool, and expect a ``.py`` file to contain readable text. Both would be confusing for this use case. How does this compete with existing package formats? The sdist, bdist and wheel formats are designed for packaging of modules to be installed into an existing Python installation. They are not intended to be used without installing. The executable zip format is specifically designed for standalone use, without needing to be installed. They are in effect a multi-file version of a standalone Python script. Rejected Proposals ================== Convenience Values for Shebang Lines ------------------------------------ Is it worth having "convenience" forms for any of the common interpreter values? For example, ``-p 3`` meaning the same as ``-p "/usr/bin/env python3"``. It would save a lot of typing for the common cases, as well as giving cross-platform options for people who don't want or need to understand the intricacies of shebang handling on "other" platforms. Downsides are that it's not obvious how to translate the abbreviations. For example, should "3" mean "/usr/bin/env python3", "/usr/bin/python3", "python3", or something else? Also, there is no obvious short form for the key case of "/usr/bin/env python" (any available version of Python), which could easily result in scripts being written with overly-restrictive shebang lines. Overall, this seems like there are more problems than benefits, and as a result has been dropped from consideration. Registering ``.pyz`` as a Media Type -------------------------------- It was suggested [3]_ that the ``.pyz`` extension should be registered in the Unix database of extensions. While it makes sense to do this as an equivalent of the Windows installer registering the extension, the ``.py`` extension is not listed in the media types database [4]_. It doesn't seem reasonable to register ``.pyz`` without ``.py``, so this idea has been omitted from this PEP. An interested party could arrange for *both* ``.py`` and ``.pyz`` to be registered at a future date. Default Interpreter ------------------- The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter. Unix users have problems with this behaviour, as the default for the python command on many distributions is Python 2, and it is felt that this PEP should prefer Python 3 by default. However, using a command of ``python3`` can result in unexpected behaviour for Windows users, where the default behaviour of the launcher for the command ``python`` is commonly customised by users, but the behaviour of ``python3`` may not be modified to match. As a result, the principle "in the face of ambiguity, refuse to guess" has been invoked, and archives have no shebang line unless explicitly requested. On Windows, the archives will still be run (with the default Python) by the launcher, and on Unix, the archives can be run by explicitly invoking the desired Python interpreter. Command Line Tool to Manage Shebang Lines ----------------------------------------- It is conceivable that users would want to modify the shebang line for an existing archive, or even just display the current shebang line. This is tricky to do so with existing tools (zip programs typically ignore prepended data totally, and text editors can have trouble editing files containing binary data). The zipapp module provides functions to handle the shebang line, but does not include a command line interface to that functionality. This is because it is not clear how to provide one without the resulting interface being over-complex and potentially confusing. Changing the shebang line is expected to be an uncommon requirement. Reference Implementation ======================== A reference implementation is at http://bugs.python.org/issue23491. References ========== .. [1] Allow interpreter to execute a zip file (http://bugs.python.org/issue1739468) .. [2] Feature is not documented (http://bugs.python.org/issue17359) .. [3] Discussion of adding a .pyz mime type on python-dev (https://mail.python.org/pipermail/python-dev/2015-February/138338.html) .. [4] Register of media types (http://www.iana.org/assignments/media-types/media-types.xhtml) .. [5] pyzzer - A tool for creating Python-executable archives (https://pypi.python.org/pypi/pyzzer) .. [6] pex - The PEX packaging toolchain (https://pypi.python.org/pypi/pex) The discussion of this PEP took place on the python-dev mailing list, in the thread starting at https://mail.python.org/pipermail/python-dev/2015-February/138277.html Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From brett at python.org Mon Feb 23 18:54:50 2015 From: brett at python.org (Brett Cannon) Date: Mon, 23 Feb 2015 17:54:50 +0000 Subject: [Python-Dev] Removing Python Binaries with make command References: <1424267868.788959.229212373.496F2381@webmail.messagingengine.com> <1857446440.8109273.1424709756765.JavaMail.yahoo@mail.yahoo.com> <1942879138.8132646.1424711391492.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Mon Feb 23 2015 at 12:14:08 PM Blaxton wrote: > > > > Hi > > I have installed Python3 with running ./configure , make and then make > install > and now can't find an option in Makefile to remove the binaries with make > command. > > Is there a way to cleanly remove Python3 Binaries after compiling on a > system ? > Sorry, there's is no explicit `make uninstall`. Since most people either use the installers for Windows or OS X or use their Linux distro's installer I don't know if anyone has tried to write a `make uninstall` command. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Mon Feb 23 18:48:01 2015 From: phd at phdru.name (Oleg Broytman) Date: Mon, 23 Feb 2015 18:48:01 +0100 Subject: [Python-Dev] Removing Python Binaries with make command In-Reply-To: <1942879138.8132646.1424711391492.JavaMail.yahoo@mail.yahoo.com> References: <1424267868.788959.229212373.496F2381@webmail.messagingengine.com> <1857446440.8109273.1424709756765.JavaMail.yahoo@mail.yahoo.com> <1942879138.8132646.1424711391492.JavaMail.yahoo@mail.yahoo.com> Message-ID: <20150223174801.GA12948@phdru.name> On Mon, Feb 23, 2015 at 05:09:51PM +0000, Blaxton wrote: > Hi? > I have installed Python3 with running ./configure , make and then make installand now can't find an option in Makefile to remove the binaries with make command. > Is there a way to cleanly remove Python3 Binaries after compiling on a system ? > ThanksDB There is no way AFAIK. That what CheckInstall[1] is for but you should have used it before make install. 1. http://asic-linux.com.mx/~izto/checkinstall/index.php PS. And such questions belong to python-list, I think, rather than python-dev which is about developing Python, not using it. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From brett at python.org Mon Feb 23 19:16:47 2015 From: brett at python.org (Brett Cannon) Date: Mon, 23 Feb 2015 18:16:47 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support References: Message-ID: Overall I like this and don't see any reason not to accept it, so +1. I do have a couple comments/questions on the module API, though. On Mon Feb 23 2015 at 12:45:28 PM Paul Moore wrote: > > ``set_interpreter(archive, new_archive, interpreter=None)`` > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > Modifies the *archive*'s shebang line to contain the specified > interpreter, and writes the updated archive to *new_archive*. If the > *interpreter* is ``None``, removes the shebang line. > Should new_archive default to None to allow for in-place editing? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Feb 23 19:17:25 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Feb 2015 10:17:25 -0800 Subject: [Python-Dev] Removing Python Binaries with make command In-Reply-To: <20150223174801.GA12948@phdru.name> References: <1424267868.788959.229212373.496F2381@webmail.messagingengine.com> <1857446440.8109273.1424709756765.JavaMail.yahoo@mail.yahoo.com> <1942879138.8132646.1424711391492.JavaMail.yahoo@mail.yahoo.com> <20150223174801.GA12948@phdru.name> Message-ID: It could arguably considered a bug report. On Mon, Feb 23, 2015 at 9:48 AM, Oleg Broytman wrote: > On Mon, Feb 23, 2015 at 05:09:51PM +0000, Blaxton > wrote: > > Hi > > I have installed Python3 with running ./configure , make and then make > installand now can't find an option in Makefile to remove the binaries with > make command. > > Is there a way to cleanly remove Python3 Binaries after compiling on a > system ? > > ThanksDB > > There is no way AFAIK. That what CheckInstall[1] is for but you > should have used it before make install. > > 1. http://asic-linux.com.mx/~izto/checkinstall/index.php > > PS. And such questions belong to python-list, I think, rather than > python-dev which is about developing Python, not using it. > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Feb 23 19:27:43 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 23 Feb 2015 13:27:43 -0500 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On Mon, Feb 23, 2015 at 1:16 PM, Brett Cannon wrote: > Overall I like this and don't see any reason not to accept it, so +1. I do > have a couple comments/questions on the module API, though. > > On Mon Feb 23 2015 at 12:45:28 PM Paul Moore wrote: >> >> >> >> >> ``set_interpreter(archive, new_archive, interpreter=None)`` >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> >> Modifies the *archive*'s shebang line to contain the specified >> interpreter, and writes the updated archive to *new_archive*. If the >> *interpreter* is ``None``, removes the shebang line. > > > Should new_archive default to None to allow for in-place editing? > > -Brett That would be cool but more work. Unless the length of the new shebang is <= the old one, the zip file contents have to be moved out of the way. From p.f.moore at gmail.com Mon Feb 23 19:49:12 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Feb 2015 18:49:12 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 23 February 2015 at 18:40, Brett Cannon wrote: > Couldn't you just keep it in memory as bytes and then write directly over > the file? I realize that's a bit wasteful memory-wise but it is possible. > The docs could mention the memory cost is something to watch out for when > doing an in-place replacement. Heck the code could even make it an > io.BytesIO instance so the rest of the code doesn't have to care about this > special case. I did consider this option, and I still quite like it. In fact, originally I wrote the API to *only* be in-place, until I realised that wouldn't work for things bigger than memory (but who has a Python app that's bigger than RAM?) I'm happy to modify the API along these lines (details to be thrashed out) if people think it's worthwhile. Paul From dholth at gmail.com Mon Feb 23 20:01:30 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 23 Feb 2015 14:01:30 -0500 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: Sounds reasonable. It could be done by just reading the entire file contents after the shebang and re-writing them with the necessary offset all in RAM, truncating the file if necessary, without involving the zipfile module very much; the shebang could have some amount of padding by default; the file could just be re-compressed in memory depending on your appetite for complexity. On Mon, Feb 23, 2015 at 1:49 PM, Paul Moore wrote: > On 23 February 2015 at 18:40, Brett Cannon wrote: >> Couldn't you just keep it in memory as bytes and then write directly over >> the file? I realize that's a bit wasteful memory-wise but it is possible. >> The docs could mention the memory cost is something to watch out for when >> doing an in-place replacement. Heck the code could even make it an >> io.BytesIO instance so the rest of the code doesn't have to care about this >> special case. > > I did consider this option, and I still quite like it. In fact, > originally I wrote the API to *only* be in-place, until I realised > that wouldn't work for things bigger than memory (but who has a Python > app that's bigger than RAM?) > > I'm happy to modify the API along these lines (details to be thrashed > out) if people think it's worthwhile. > Paul From ethan at stoneleaf.us Mon Feb 23 20:22:06 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 23 Feb 2015 11:22:06 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <54EB7DDE.9030405@stoneleaf.us> On 02/23/2015 11:01 AM, Daniel Holth wrote: > On Mon, Feb 23, 2015 at 1:49 PM, Paul Moore wrote: >> On 23 February 2015 at 18:40, Brett Cannon wrote: >>> >>> Couldn't you just keep it in memory as bytes and then write directly over >>> the file? I realize that's a bit wasteful memory-wise but it is possible. >>> The docs could mention the memory cost is something to watch out for when >>> doing an in-place replacement. Heck the code could even make it an >>> io.BytesIO instance so the rest of the code doesn't have to care about this >>> special case. >> >> I did consider this option, and I still quite like it. In fact, >> originally I wrote the API to *only* be in-place, until I realised >> that wouldn't work for things bigger than memory (but who has a Python >> app that's bigger than RAM?) >> >> I'm happy to modify the API along these lines (details to be thrashed >> out) if people think it's worthwhile. > > Sounds reasonable. It could be done by just reading the entire file > contents after the shebang and re-writing them with the necessary > offset all in RAM, truncating the file if necessary, without involving > the zipfile module very much; the shebang could have some amount of > padding by default; the file could just be re-compressed in memory > depending on your appetite for complexity. This could be a completely stupid question, but how does the zip file know where the individual files are? More to the point, does the index work via relative or absolute offset? If absolute, wouldn't the index have to be rewritten if the zip portion of the file moves? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From njs at pobox.com Mon Feb 23 20:25:29 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 23 Feb 2015 11:25:29 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> Message-ID: On Feb 23, 2015 9:30 AM, "Guido van Rossum" wrote: > > On Mon, Feb 23, 2015 at 9:01 AM, Ethan Furman wrote: >> >> On 02/23/2015 08:12 AM, Brett Cannon wrote: >> > On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote: >> >> >> Is it at all possible for this to be introduced in the 2.x line [...] >> > >> > Starts with a minor version. >> >> Isn't there a -3 switch or something like that in 2.7.x to trigger warnings/errors to help port to 3.x? Seems like this >> kind of warning could go there. > > > If we agree it will be a syntax error in 3.x eventually. +1 to that FWIW. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Feb 23 20:26:49 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Feb 2015 19:26:49 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54EB7DDE.9030405@stoneleaf.us> References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 23 February 2015 at 19:22, Ethan Furman wrote: > This could be a completely stupid question, but how does the zip file know where the individual files are? More to the > point, does the index work via relative or absolute offset? If absolute, wouldn't the index have to be rewritten if the > zip portion of the file moves? Essentially the index is stored at the *end* of the file, and contains relative offsets to all the files. Gory details at https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT Paul From p.f.moore at gmail.com Mon Feb 23 20:24:10 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Feb 2015 19:24:10 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 23 February 2015 at 19:01, Daniel Holth wrote: > Sounds reasonable. It could be done by just reading the entire file > contents after the shebang and re-writing them with the necessary > offset all in RAM, truncating the file if necessary, without involving > the zipfile module very much; the shebang could have some amount of > padding by default; the file could just be re-compressed in memory > depending on your appetite for complexity. The biggest problem with that is finding the end of the prefix data. Frankly it's easier just to write a new prefix then use the zipfile module to rewrite all of the content. That's what the current code does writing to a new file. Paul From thomas at python.org Mon Feb 23 20:41:35 2015 From: thomas at python.org (Thomas Wouters) Date: Mon, 23 Feb 2015 20:41:35 +0100 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54EB7DDE.9030405@stoneleaf.us> References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On Mon, Feb 23, 2015 at 8:22 PM, Ethan Furman wrote: > On 02/23/2015 11:01 AM, Daniel Holth wrote: > > On Mon, Feb 23, 2015 at 1:49 PM, Paul Moore wrote: > >> On 23 February 2015 at 18:40, Brett Cannon wrote: > >>> > >>> Couldn't you just keep it in memory as bytes and then write directly > over > >>> the file? I realize that's a bit wasteful memory-wise but it is > possible. > >>> The docs could mention the memory cost is something to watch out for > when > >>> doing an in-place replacement. Heck the code could even make it an > >>> io.BytesIO instance so the rest of the code doesn't have to care about > this > >>> special case. > >> > >> I did consider this option, and I still quite like it. In fact, > >> originally I wrote the API to *only* be in-place, until I realised > >> that wouldn't work for things bigger than memory (but who has a Python > >> app that's bigger than RAM?) > >> > >> I'm happy to modify the API along these lines (details to be thrashed > >> out) if people think it's worthwhile. > > > > Sounds reasonable. It could be done by just reading the entire file > > contents after the shebang and re-writing them with the necessary > > offset all in RAM, truncating the file if necessary, without involving > > the zipfile module very much; the shebang could have some amount of > > padding by default; the file could just be re-compressed in memory > > depending on your appetite for complexity. > > This could be a completely stupid question, but how does the zip file know > where the individual files are? More to the > point, does the index work via relative or absolute offset? If absolute, > wouldn't the index have to be rewritten if the > zip portion of the file moves? > Yes and no. The ZIP format uses a 'central directory' which is a record of each file in the archive. The offsets are relative (although the specification is a little vague on what they're relative *to* when using a .zip file. The wording talks about disk numbers, ZIP being from the era of floppy disks.) You find the central directory by searching from the end (or reading a specific spot at the end, if you don't support archive comments. zipimport, for example, doesn't support archive comments) and it turns out you can find the central directory from just that information (and as far as I know, all tools do.) However, there are still some offsets that would change if you add stuff to the front of the ZIP file (or remove it), and some zip tools will complain (usually just in verbose mode, though.) -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Feb 23 19:40:21 2015 From: brett at python.org (Brett Cannon) Date: Mon, 23 Feb 2015 18:40:21 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support References: Message-ID: On Mon Feb 23 2015 at 1:34:03 PM Daniel Holth wrote: > On Mon, Feb 23, 2015 at 1:16 PM, Brett Cannon wrote: > > Overall I like this and don't see any reason not to accept it, so +1. I > do > > have a couple comments/questions on the module API, though. > > > > On Mon Feb 23 2015 at 12:45:28 PM Paul Moore > wrote: > >> > >> > >> > >> > >> ``set_interpreter(archive, new_archive, interpreter=None)`` > >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > >> > >> Modifies the *archive*'s shebang line to contain the specified > >> interpreter, and writes the updated archive to *new_archive*. If the > >> *interpreter* is ``None``, removes the shebang line. > > > > > > Should new_archive default to None to allow for in-place editing? > > > > -Brett > > That would be cool but more work. Unless the length of the new shebang > is <= the old one, the zip file contents have to be moved out of the > way. > Couldn't you just keep it in memory as bytes and then write directly over the file? I realize that's a bit wasteful memory-wise but it is possible. The docs could mention the memory cost is something to watch out for when doing an in-place replacement. Heck the code could even make it an io.BytesIO instance so the rest of the code doesn't have to care about this special case. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Feb 23 20:47:28 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Feb 2015 11:47:28 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: So is the PEP ready for pronouncement or should there be more discussion? Also, do you have a BDFL-delegate or do you want me to review it? On Mon, Feb 23, 2015 at 11:41 AM, Thomas Wouters wrote: > > > On Mon, Feb 23, 2015 at 8:22 PM, Ethan Furman wrote: > >> On 02/23/2015 11:01 AM, Daniel Holth wrote: >> > On Mon, Feb 23, 2015 at 1:49 PM, Paul Moore wrote: >> >> On 23 February 2015 at 18:40, Brett Cannon wrote: >> >>> >> >>> Couldn't you just keep it in memory as bytes and then write directly >> over >> >>> the file? I realize that's a bit wasteful memory-wise but it is >> possible. >> >>> The docs could mention the memory cost is something to watch out for >> when >> >>> doing an in-place replacement. Heck the code could even make it an >> >>> io.BytesIO instance so the rest of the code doesn't have to care >> about this >> >>> special case. >> >> >> >> I did consider this option, and I still quite like it. In fact, >> >> originally I wrote the API to *only* be in-place, until I realised >> >> that wouldn't work for things bigger than memory (but who has a Python >> >> app that's bigger than RAM?) >> >> >> >> I'm happy to modify the API along these lines (details to be thrashed >> >> out) if people think it's worthwhile. >> > >> > Sounds reasonable. It could be done by just reading the entire file >> > contents after the shebang and re-writing them with the necessary >> > offset all in RAM, truncating the file if necessary, without involving >> > the zipfile module very much; the shebang could have some amount of >> > padding by default; the file could just be re-compressed in memory >> > depending on your appetite for complexity. >> >> This could be a completely stupid question, but how does the zip file >> know where the individual files are? More to the >> point, does the index work via relative or absolute offset? If absolute, >> wouldn't the index have to be rewritten if the >> zip portion of the file moves? >> > > Yes and no. The ZIP format uses a 'central directory' which is a record of > each file in the archive. The offsets are relative (although the > specification is a little vague on what they're relative *to* when using a > .zip file. The wording talks about disk numbers, ZIP being from the era of > floppy disks.) You find the central directory by searching from the end (or > reading a specific spot at the end, if you don't support archive comments. > zipimport, for example, doesn't support archive comments) and it turns out > you can find the central directory from just that information (and as far > as I know, all tools do.) However, there are still some offsets that would > change if you add stuff to the front of the ZIP file (or remove it), and > some zip tools will complain (usually just in verbose mode, though.) > > -- > Thomas Wouters > > Hi! I'm an email virus! Think twice before sending your email to help me > spread! > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 23 20:47:51 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Feb 2015 20:47:51 +0100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? References: <54EB5CEB.90305@stoneleaf.us> Message-ID: <20150223204751.21c0cb1c@fsol> On Mon, 23 Feb 2015 09:29:09 -0800 Guido van Rossum wrote: > On Mon, Feb 23, 2015 at 9:01 AM, Ethan Furman wrote: > > > On 02/23/2015 08:12 AM, Brett Cannon wrote: > > > On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote: > > > > >> Is it at all possible for this to be introduced in the 2.x line [...] > > > > > > Starts with a minor version. > > > > Isn't there a -3 switch or something like that in 2.7.x to trigger > > warnings/errors to help port to 3.x? Seems like this > > kind of warning could go there. > > > > If we agree it will be a syntax error in 3.x eventually. That sounds frankly like a pedantic change. Regards Antoine. From ethan at stoneleaf.us Mon Feb 23 20:56:05 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 23 Feb 2015 11:56:05 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <54EB85D5.3030609@stoneleaf.us> On 02/23/2015 10:49 AM, Paul Moore wrote: > I did consider this option, and I still quite like it. In fact, > originally I wrote the API to *only* be in-place, until I realised > that wouldn't work for things bigger than memory (but who has a Python > app that's bigger than RAM?) Depends -- how many image files are that app? ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From jsbueno at python.org.br Mon Feb 23 20:58:29 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Mon, 23 Feb 2015 16:58:29 -0300 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <20150223204751.21c0cb1c@fsol> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: On 23 February 2015 at 16:47, Antoine Pitrou wrote: > On Mon, 23 Feb 2015 09:29:09 -0800 > Guido van Rossum wrote: >> On Mon, Feb 23, 2015 at 9:01 AM, Ethan Furman wrote: >> >> > On 02/23/2015 08:12 AM, Brett Cannon wrote: >> > > On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote: >> > >> > >> Is it at all possible for this to be introduced in the 2.x line [...] >> > > >> > > Starts with a minor version. >> > >> > Isn't there a -3 switch or something like that in 2.7.x to trigger >> > warnings/errors to help port to 3.x? Seems like this >> > kind of warning could go there. >> > >> >> If we agree it will be a syntax error in 3.x eventually. > > That sounds frankly like a pedantic change. The real problem there, even motivating the request, seems to be new users in Windows platforms using "\" as file path separator. That happens all the time, and is this use case that should possibly be addressed here - maybe something as simple as adding a couple of paragraphs to different places in the documentation could mitigate the issue. (in contrast to make a tons of otherwise valid code to become deprecated in a couple releases). > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br From thomas at python.org Mon Feb 23 21:00:38 2015 From: thomas at python.org (Thomas Wouters) Date: Mon, 23 Feb 2015 21:00:38 +0100 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On Mon, Feb 23, 2015 at 8:24 PM, Paul Moore wrote: > On 23 February 2015 at 19:01, Daniel Holth wrote: > > Sounds reasonable. It could be done by just reading the entire file > > contents after the shebang and re-writing them with the necessary > > offset all in RAM, truncating the file if necessary, without involving > > the zipfile module very much; the shebang could have some amount of > > padding by default; the file could just be re-compressed in memory > > depending on your appetite for complexity. > > The biggest problem with that is finding the end of the prefix data. > Frankly it's easier just to write a new prefix then use the zipfile > module to rewrite all of the content. That's what the current code > does writing to a new file. I don't think you need to rewrite all of the contents, if you don't mind poking into zipfile internals: endrec = zipfile._EndRecData(f) prefix_length = endrec[zipfile._ECD_LOCATION] - endrec[zipfile._ECD_SIZE] - endrec[zipfile._ECD_OFFSET] I do something similar to get at the prefix, although I need the zipfile opened anyway, so I use: endrec = zipfile._EndRecData(f) # pylint: disable=protected-access zf = zipfile.ZipFile(f) # endrec is None if reading it failed, but then ZipFile should have # raised an exception... assert endrec prefix_len = zf.start_dir - endrec[zipfile._ECD_OFFSET] # pylint: disable=protected-access Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/thomas%40python.org > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Feb 23 21:23:51 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 23 Feb 2015 22:23:51 +0200 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54EB7DDE.9030405@stoneleaf.us> References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 23.02.15 21:22, Ethan Furman wrote: > This could be a completely stupid question, but how does the zip file know where the individual files are? More to the > point, does the index work via relative or absolute offset? If absolute, wouldn't the index have to be rewritten if the > zip portion of the file moves? Absolute. From p.f.moore at gmail.com Mon Feb 23 21:32:39 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Feb 2015 20:32:39 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 23 February 2015 at 19:47, Guido van Rossum wrote: > So is the PEP ready for pronouncement or should there be more discussion? I think Brett's idea is worth incorporating, so let's thrash that out first. > Also, do you have a BDFL-delegate or do you want me to review it? No-one has stepped up as BDFL-delegate, and there's no obvious candidate, so I think you're it, sorry :-) Paul From storchaka at gmail.com Mon Feb 23 21:32:41 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 23 Feb 2015 22:32:41 +0200 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: Message-ID: On 23.02.15 17:54, Chris Angelico wrote: > If it's never going to happen in 2.x, I'll raise this as yet another > reason to get the course and all our students migrated to 3.x, but on > the flip side, it means that we absolutely can't get the benefit until > that jump is made. You can use third-party checkers. And if you use an editor with syntax highlighting, it is likely that unrecognized backslash escapes will be marked as suspicious. From solipsis at pitrou.net Mon Feb 23 21:38:45 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Feb 2015 21:38:45 +0100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: <20150223213845.0e08372a@fsol> On Mon, 23 Feb 2015 16:58:29 -0300 "Joao S. O. Bueno" wrote: > On 23 February 2015 at 16:47, Antoine Pitrou wrote: > > On Mon, 23 Feb 2015 09:29:09 -0800 > > Guido van Rossum wrote: > >> On Mon, Feb 23, 2015 at 9:01 AM, Ethan Furman wrote: > >> > >> > On 02/23/2015 08:12 AM, Brett Cannon wrote: > >> > > On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote: > >> > > >> > >> Is it at all possible for this to be introduced in the 2.x line [...] > >> > > > >> > > Starts with a minor version. > >> > > >> > Isn't there a -3 switch or something like that in 2.7.x to trigger > >> > warnings/errors to help port to 3.x? Seems like this > >> > kind of warning could go there. > >> > > >> > >> If we agree it will be a syntax error in 3.x eventually. > > > > That sounds frankly like a pedantic change. > > The real problem there, even motivating the request, seems to be new > users in Windows platforms > using "\" as file path separator. Thanks. That sounds like a good reason indeed. Regards Antoine. From p.f.moore at gmail.com Mon Feb 23 21:51:09 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Feb 2015 20:51:09 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 23 February 2015 at 18:40, Brett Cannon wrote: > Couldn't you just keep it in memory as bytes and then write directly over > the file? I realize that's a bit wasteful memory-wise but it is possible. > The docs could mention the memory cost is something to watch out for when > doing an in-place replacement. Heck the code could even make it an > io.BytesIO instance so the rest of the code doesn't have to care about this > special case. The real problem with overwriting is if there's a failure during the overwrite you lose the original file. My original API had overwrite as the default, but I think the risk makes that a bad idea. One option would be to allow outputs (TARGET in pack() and NEW_ARCHIVE in set_interpreter()) to be open files (open for write in bytes mode) as well as filenames[1]. Then the caller has the choice of how to manage the output. The docs could include an example of overwriting via a BytesIO object, and point out the risk. BTW, while I was looking at the API, I realised I don't like the order of arguments in pack(). I'm tempted to make it pack(directory, target=None, interpreter=None, main=None) where a target of None means "use the name of the source directory with .pyz tacked on", exactly as for the command line API. What do you think? The change would be no more than a few minutes' work if it's acceptable. Paul [1] What's the standard practice for such dual-mode arguments? ZipFile tests if the argument is a str instance and assumes a file if not. I'd be inclined to follow that practice here. From brett at python.org Mon Feb 23 22:02:09 2015 From: brett at python.org (Brett Cannon) Date: Mon, 23 Feb 2015 21:02:09 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support References: Message-ID: On Mon Feb 23 2015 at 3:51:18 PM Paul Moore wrote: > On 23 February 2015 at 18:40, Brett Cannon wrote: > > Couldn't you just keep it in memory as bytes and then write directly over > > the file? I realize that's a bit wasteful memory-wise but it is possible. > > The docs could mention the memory cost is something to watch out for when > > doing an in-place replacement. Heck the code could even make it an > > io.BytesIO instance so the rest of the code doesn't have to care about > this > > special case. > > The real problem with overwriting is if there's a failure during the > overwrite you lose the original file. My original API had overwrite as > the default, but I think the risk makes that a bad idea. > Couldn't you catch the exception, write the original file back out, and then re-raise the exception? > > One option would be to allow outputs (TARGET in pack() and NEW_ARCHIVE > in set_interpreter()) to be open files (open for write in bytes mode) > as well as filenames[1]. Then the caller has the choice of how to > manage the output. The docs could include an example of overwriting > via a BytesIO object, and point out the risk. > That sounds like a good idea. No reason to do the file opening on someone's behalf when opening files is so easy and keeps layering abstractions at a good level. Would this extend also to the archive being read to be consistent? I should mention I originally thought of extending this to pack() for 'main', but realized that passing in the function to set would require tools to import the code they are simply trying to pack and that was the wrong thing to do. > > BTW, while I was looking at the API, I realised I don't like the order > of arguments in pack(). I'm tempted to make it pack(directory, > target=None, interpreter=None, main=None) where a target of None means > "use the name of the source directory with .pyz tacked on", exactly as > for the command line API. > > What do you think? The change would be no more than a few minutes' > work if it's acceptable. > +1 from me. -Brett > Paul > > [1] What's the standard practice for such dual-mode arguments? ZipFile > tests if the argument is a str instance and assumes a file if not. I'd > be inclined to follow that practice here. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Feb 23 22:08:46 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 23 Feb 2015 23:08:46 +0200 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 23.02.15 22:23, Serhiy Storchaka wrote: > On 23.02.15 21:22, Ethan Furman wrote: >> This could be a completely stupid question, but how does the zip file >> know where the individual files are? More to the >> point, does the index work via relative or absolute offset? If >> absolute, wouldn't the index have to be rewritten if the >> zip portion of the file moves? > > Absolute. Oh, I were wrong. The specification and the are not very clear about this. From ethan at stoneleaf.us Mon Feb 23 22:16:25 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 23 Feb 2015 13:16:25 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: <54EB98A9.7020007@stoneleaf.us> On 02/23/2015 01:02 PM, Brett Cannon wrote: > On Mon Feb 23 2015 at 3:51:18 PM Paul Moore wrote: >> The real problem with overwriting is if there's a failure during the >> overwrite you lose the original file. My original API had overwrite as >> the default, but I think the risk makes that a bad idea. > > Couldn't you catch the exception, write the original file back out, and then re-raise the exception? This seems to be getting pretty complex for a nice-to-have. >> One option would be to allow outputs (TARGET in pack() and NEW_ARCHIVE >> in set_interpreter()) to be open files (open for write in bytes mode) >> as well as filenames[1]. +1 for this. >> BTW, while I was looking at the API, I realised I don't like the order >> of arguments in pack(). I'm tempted to make it pack(directory, >> target=None, interpreter=None, main=None) where a target of None means >> "use the name of the source directory with .pyz tacked on", exactly as >> for the command line API. >> >> What do you think? The change would be no more than a few minutes' >> work if it's acceptable. > > +1 from me. +1 from me as well. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From storchaka at gmail.com Mon Feb 23 22:18:07 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 23 Feb 2015 23:18:07 +0200 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 23.02.15 22:51, Paul Moore wrote: > BTW, while I was looking at the API, I realised I don't like the order > of arguments in pack(). I'm tempted to make it pack(directory, > target=None, interpreter=None, main=None) where a target of None means > "use the name of the source directory with .pyz tacked on", exactly as > for the command line API. If the order of arguments is not obvious, make them keyword-only. From storchaka at gmail.com Mon Feb 23 22:27:43 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 23 Feb 2015 23:27:43 +0200 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: On 23.02.15 21:58, Joao S. O. Bueno wrote: > That happens all the time, and is this use case that should possibly > be addressed here - maybe > something as simple as adding a couple of paragraphs to different places > in the documentation could mitigate the issue. (in contrast to make a > tons of otherwise valid code > to become deprecated in a couple releases). The problem is that the user don't know that he should read the documentation. It just find that his script works with "C:\sample.txt", but doesn't work with "D:\test.txt". He has no ideas what happen. From p.f.moore at gmail.com Mon Feb 23 22:38:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Feb 2015 21:38:25 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 23 February 2015 at 21:02, Brett Cannon wrote: >> The real problem with overwriting is if there's a failure during the >> overwrite you lose the original file. My original API had overwrite as >> the default, but I think the risk makes that a bad idea. > > > Couldn't you catch the exception, write the original file back out, and then > re-raise the exception? But you don't *have* the original file. You read the source archive entry-by-entry, not all at once. Apart from the implementation difficulty, this is getting too complex, and I think it's better to just give the user the tools to add whatever robustness or exception handling they want on top. Paul From breamoreboy at yahoo.co.uk Mon Feb 23 22:39:27 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 23 Feb 2015 21:39:27 +0000 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: On 23/02/2015 21:27, Serhiy Storchaka wrote: > On 23.02.15 21:58, Joao S. O. Bueno wrote: >> That happens all the time, and is this use case that should possibly >> be addressed here - maybe >> something as simple as adding a couple of paragraphs to different places >> in the documentation could mitigate the issue. (in contrast to make a >> tons of otherwise valid code >> to become deprecated in a couple releases). > > The problem is that the user don't know that he should read the > documentation. It just find that his script works with "C:\sample.txt", > but doesn't work with "D:\test.txt". He has no ideas what happen. > Isn't this why users have help desks? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From p.f.moore at gmail.com Mon Feb 23 22:42:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Feb 2015 21:42:25 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: Message-ID: On 23 February 2015 at 21:18, Serhiy Storchaka wrote: > On 23.02.15 22:51, Paul Moore wrote: >> >> BTW, while I was looking at the API, I realised I don't like the order >> of arguments in pack(). I'm tempted to make it pack(directory, >> target=None, interpreter=None, main=None) where a target of None means >> "use the name of the source directory with .pyz tacked on", exactly as >> for the command line API. > > > If the order of arguments is not obvious, make them keyword-only. To be honest, I don't think it *is* that non-obvious. I just think I got it wrong initially. With the new API, you have pack('myappdir') pack('myappdir', 'named_target.pyz') Seems obvious enough to me. It matches the "source, destination" order of similar functions like shutil.copy. And you can use a named argument if you're not sure. But I don't think it's worth forcing it. Paul From ncoghlan at gmail.com Mon Feb 23 23:40:01 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Feb 2015 08:40:01 +1000 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: On 24 February 2015 at 07:39, Mark Lawrence wrote: > On 23/02/2015 21:27, Serhiy Storchaka wrote: >> >> On 23.02.15 21:58, Joao S. O. Bueno wrote: >>> >>> That happens all the time, and is this use case that should possibly >>> be addressed here - maybe >>> something as simple as adding a couple of paragraphs to different places >>> in the documentation could mitigate the issue. (in contrast to make a >>> tons of otherwise valid code >>> to become deprecated in a couple releases). >> >> >> The problem is that the user don't know that he should read the >> documentation. It just find that his script works with "C:\sample.txt", >> but doesn't work with "D:\test.txt". He has no ideas what happen. > > Isn't this why users have help desks? Most don't, and cases like "\n" or "\t" in a Windows path name being converted to whitespace are utterly impossible to look up in an internet search when they fail, so a user learning on their own gets left with a broken program and no particularly effective ways to ask for help figuring it out. Like Unicode encoding errors they may appear a long way from the source of the offending data value (in this case, likely to be a file name copy and pasted from elsewhere on their system), and they don't give a particularly helpful error message (especially when the escape sequences are for whitespace). While I originally disliked the idea, I think this is a genuine usability issue on Windows that would be worth addressing. However, it's a significant enough change that I believe it needs a PEP and a reasonably long transition period before anything actually breaks. For example: - pep8 and pylint warnings as soon as a patch can be accepted - Py3kWarning in Python 2.7.x - DeprecationWarning in Python 3.5 - SyntaxWarning in Python 3.6 - SyntaxError in Python 3.7 Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Feb 23 23:50:41 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Feb 2015 08:50:41 +1000 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: On 24 February 2015 at 08:40, Nick Coghlan wrote: > On 24 February 2015 at 07:39, Mark Lawrence wrote: >> On 23/02/2015 21:27, Serhiy Storchaka wrote: >>> >>> On 23.02.15 21:58, Joao S. O. Bueno wrote: >>>> >>>> That happens all the time, and is this use case that should possibly >>>> be addressed here - maybe >>>> something as simple as adding a couple of paragraphs to different places >>>> in the documentation could mitigate the issue. (in contrast to make a >>>> tons of otherwise valid code >>>> to become deprecated in a couple releases). >>> >>> >>> The problem is that the user don't know that he should read the >>> documentation. It just find that his script works with "C:\sample.txt", >>> but doesn't work with "D:\test.txt". He has no ideas what happen. >> >> Isn't this why users have help desks? > > Most don't, and cases like "\n" or "\t" in a Windows path name being > converted to whitespace are utterly impossible to look up in an > internet search when they fail, so a user learning on their own gets > left with a broken program and no particularly effective ways to ask > for help figuring it out. > > Like Unicode encoding errors they may appear a long way from the > source of the offending data value (in this case, likely to be a file > name copy and pasted from elsewhere on their system), and they don't > give a particularly helpful error message (especially when the escape > sequences are for whitespace). > > While I originally disliked the idea, I think this is a genuine > usability issue on Windows that would be worth addressing. However, > it's a significant enough change that I believe it needs a PEP and a > reasonably long transition period before anything actually breaks. For > example: > > - pep8 and pylint warnings as soon as a patch can be accepted > - Py3kWarning in Python 2.7.x > - DeprecationWarning in Python 3.5 > - SyntaxWarning in Python 3.6 > - SyntaxError in Python 3.7 Another suggestion: we may want to turn this particular deprecation warning on by default in the interactive interpreter, and recommend that other interactive interpreter developers (such as the IPython folks) consider doing the same. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Mon Feb 23 23:52:51 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 23 Feb 2015 14:52:51 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: <54EBAF43.5060204@stoneleaf.us> On 02/23/2015 02:40 PM, Nick Coghlan wrote: > - pep8 and pylint warnings as soon as a patch can be accepted > - Py3kWarning in Python 2.7.x > - DeprecationWarning in Python 3.5 > - SyntaxWarning in Python 3.6 > - SyntaxError in Python 3.7 +1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From storchaka at gmail.com Tue Feb 24 00:32:58 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 24 Feb 2015 01:32:58 +0200 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: On 24.02.15 00:50, Nick Coghlan wrote: >> - SyntaxError in Python 3.7 I think that SyntaxError could wait to 4.0. > Another suggestion: we may want to turn this particular deprecation > warning on by default in the interactive interpreter, and recommend > that other interactive interpreter developers (such as the IPython > folks) consider doing the same. Why not turn all deprecation warning on by default in the interactive interpreter? From greg.ewing at canterbury.ac.nz Mon Feb 23 22:39:25 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 24 Feb 2015 10:39:25 +1300 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: <54EB9E0D.4070000@canterbury.ac.nz> Serhiy Storchaka wrote: > The problem is that the user don't know that he should read the > documentation. It just find that his script works with "C:\sample.txt", > but doesn't work with "D:\test.txt". He has no ideas what happen. Even with the syntax error, there are filenames that will mysteriously fail, e.g. "C:\ninjamoves.txt". -- Greg From python at mrabarnett.plus.com Tue Feb 24 00:40:55 2015 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 23 Feb 2015 23:40:55 +0000 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: <54EBBA87.90507@mrabarnett.plus.com> On 2015-02-23 22:50, Nick Coghlan wrote: > On 24 February 2015 at 08:40, Nick Coghlan wrote: >> On 24 February 2015 at 07:39, Mark Lawrence wrote: >>> On 23/02/2015 21:27, Serhiy Storchaka wrote: >>>> >>>> On 23.02.15 21:58, Joao S. O. Bueno wrote: >>>>> >>>>> That happens all the time, and is this use case that should possibly >>>>> be addressed here - maybe >>>>> something as simple as adding a couple of paragraphs to different places >>>>> in the documentation could mitigate the issue. (in contrast to make a >>>>> tons of otherwise valid code >>>>> to become deprecated in a couple releases). >>>> >>>> >>>> The problem is that the user don't know that he should read the >>>> documentation. It just find that his script works with "C:\sample.txt", >>>> but doesn't work with "D:\test.txt". He has no ideas what happen. >>> >>> Isn't this why users have help desks? >> >> Most don't, and cases like "\n" or "\t" in a Windows path name being >> converted to whitespace are utterly impossible to look up in an >> internet search when they fail, so a user learning on their own gets >> left with a broken program and no particularly effective ways to ask >> for help figuring it out. >> >> Like Unicode encoding errors they may appear a long way from the >> source of the offending data value (in this case, likely to be a file >> name copy and pasted from elsewhere on their system), and they don't >> give a particularly helpful error message (especially when the escape >> sequences are for whitespace). >> >> While I originally disliked the idea, I think this is a genuine >> usability issue on Windows that would be worth addressing. However, >> it's a significant enough change that I believe it needs a PEP and a >> reasonably long transition period before anything actually breaks. For >> example: >> >> - pep8 and pylint warnings as soon as a patch can be accepted >> - Py3kWarning in Python 2.7.x >> - DeprecationWarning in Python 3.5 >> - SyntaxWarning in Python 3.6 >> - SyntaxError in Python 3.7 > > Another suggestion: we may want to turn this particular deprecation > warning on by default in the interactive interpreter, and recommend > that other interactive interpreter developers (such as the IPython > folks) consider doing the same. > While we're at it, could something similar also be done to re? In that instance, unknown escapes with ASCII-range letters _drop_ the backslash: >>> re.match(r'\q', 'q') <_sre.SRE_Match object; span=(0, 1), match='q'> From rosuav at gmail.com Tue Feb 24 00:53:11 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Feb 2015 10:53:11 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: On Tue, Feb 24, 2015 at 9:40 AM, Nick Coghlan wrote: > Most don't, and cases like "\n" or "\t" in a Windows path name being > converted to whitespace are utterly impossible to look up in an > internet search when they fail, so a user learning on their own gets > left with a broken program and no particularly effective ways to ask > for help figuring it out. > > Like Unicode encoding errors they may appear a long way from the > source of the offending data value (in this case, likely to be a file > name copy and pasted from elsewhere on their system), and they don't > give a particularly helpful error message (especially when the escape > sequences are for whitespace). And on the subject of Unicode errors, imagine a path name of "C:\Users\YourName\Desktop\FileName.txt". Under Python 2, that'll work fine; under Python 3, it'll fail with an error that, in all probability, will be interpreted as "Unicode is hard :(" rather than "loose backslash escapes are a problem". ChrisA From rosuav at gmail.com Tue Feb 24 00:58:15 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Feb 2015 10:58:15 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: On Tue, Feb 24, 2015 at 9:40 AM, Nick Coghlan wrote: > While I originally disliked the idea, I think this is a genuine > usability issue on Windows that would be worth addressing. However, > it's a significant enough change that I believe it needs a PEP and a > reasonably long transition period before anything actually breaks. I'd be happy to write and champion such a PEP if that's the way forward. ChrisA From ezio.melotti at gmail.com Tue Feb 24 07:23:21 2015 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Tue, 24 Feb 2015 08:23:21 +0200 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: Hi, On Mon, Feb 23, 2015 at 9:58 PM, Joao S. O. Bueno wrote: > On 23 February 2015 at 16:47, Antoine Pitrou wrote: >> On Mon, 23 Feb 2015 09:29:09 -0800 >> Guido van Rossum wrote: >>> On Mon, Feb 23, 2015 at 9:01 AM, Ethan Furman wrote: >>> >>> > On 02/23/2015 08:12 AM, Brett Cannon wrote: >>> > > On Mon Feb 23 2015 at 10:55:23 AM Chris Angelico wrote: >>> > >>> > >> Is it at all possible for this to be introduced in the 2.x line [...] >>> > > >>> > > Starts with a minor version. >>> > >>> > Isn't there a -3 switch or something like that in 2.7.x to trigger >>> > warnings/errors to help port to 3.x? Seems like this >>> > kind of warning could go there. >>> > >>> >>> If we agree it will be a syntax error in 3.x eventually. >> >> That sounds frankly like a pedantic change. > > The real problem there, even motivating the request, seems to be new > users in Windows platforms > using "\" as file path separator. > > That happens all the time, and is this use case that should possibly > be addressed here - maybe > something as simple as adding a couple of paragraphs to different places > in the documentation could mitigate the issue. (in contrast to make a > tons of otherwise valid code > to become deprecated in a couple releases). > Note that in the introduction page of the tutorial[0], there is already the following snippet: """ If you don?t want characters prefaced by \ to be interpreted as special characters, you can use raw strings by adding an r before the first quote: >>> >>> print('C:\some\name') # here \n means newline! C:\some ame >>> print(r'C:\some\name') # note the r before the quote C:\some\name """" Best Regards, Ezio Melotti [0]: https://docs.python.org/3/tutorial/introduction.html >> >> Regards >> >> Antoine. >> From rosuav at gmail.com Tue Feb 24 08:02:03 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Feb 2015 18:02:03 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54EB9E0D.4070000@canterbury.ac.nz> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> Message-ID: On Tue, Feb 24, 2015 at 8:39 AM, Greg Ewing wrote: > Serhiy Storchaka wrote: >> >> The problem is that the user don't know that he should read the >> documentation. It just find that his script works with "C:\sample.txt", but >> doesn't work with "D:\test.txt". He has no ideas what happen. > > > Even with the syntax error, there are filenames that > will mysteriously fail, e.g. "C:\ninjamoves.txt". Yes, there are; but they will fail, rather than randomly happening to work. With the proposed syntax warning, there will be no cases which silently succeed; everything will either be interpreted as an escape sequence (as in your example), fail with a hard error ("C:\Users" in a Unicode string, or "C:\x-do-not-track"), or produce a warning (everything else). The most tricky escape sequence that I can envision is \134, so you might find that an all-numeric path gets beheaded: >>> print("C:\134689") C:\689 The chances that someone will accidentally get something like this are very low, and the main thing is, "\X" for any X will never be interpreted the same way as "\\X", so the backslash will _always_ be seen as special. ChrisA From mal at egenix.com Tue Feb 24 09:40:10 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 24 Feb 2015 09:40:10 +0100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: <54EC38EA.2060306@egenix.com> On 23.02.2015 23:50, Nick Coghlan wrote: > On 24 February 2015 at 08:40, Nick Coghlan wrote: >> On 24 February 2015 at 07:39, Mark Lawrence wrote: >>> On 23/02/2015 21:27, Serhiy Storchaka wrote: >>>> >>>> On 23.02.15 21:58, Joao S. O. Bueno wrote: >>>>> >>>>> That happens all the time, and is this use case that should possibly >>>>> be addressed here - maybe >>>>> something as simple as adding a couple of paragraphs to different places >>>>> in the documentation could mitigate the issue. (in contrast to make a >>>>> tons of otherwise valid code >>>>> to become deprecated in a couple releases). >>>> >>>> >>>> The problem is that the user don't know that he should read the >>>> documentation. It just find that his script works with "C:\sample.txt", >>>> but doesn't work with "D:\test.txt". He has no ideas what happen. >>> >>> Isn't this why users have help desks? >> >> Most don't, and cases like "\n" or "\t" in a Windows path name being >> converted to whitespace are utterly impossible to look up in an >> internet search when they fail, so a user learning on their own gets >> left with a broken program and no particularly effective ways to ask >> for help figuring it out. I think the easiest way would be to tweak the error message output to indicate the real problem. At the moment, you get: >>> open('c:\test.txt') Traceback (most recent call last): File "", line 1, in FileNotFoundError: [Errno 2] No such file or directory: 'c:\test.txt' which isn't helpful. It would be better to display the escapes as what they are: FileNotFoundError: [Errno 2] No such file or directory: 'c: est.txt' or use a special representation function to highlight the possible problem: FileNotFoundError: [Errno 2] No such file or directory: 'c:[\t]est.txt' - warning: embedded escapes with the representation being: "'c:[\t]est.txt' - warning: embedded escapes" Such a change doesn't require a deprecation phase or a PEP. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 24 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Tue Feb 24 10:03:15 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Feb 2015 19:03:15 +1000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 24 February 2015 at 06:32, Paul Moore wrote: > On 23 February 2015 at 19:47, Guido van Rossum wrote: >> So is the PEP ready for pronouncement or should there be more discussion? > > I think Brett's idea is worth incorporating, so let's thrash that out first. > >> Also, do you have a BDFL-delegate or do you want me to review it? > > No-one has stepped up as BDFL-delegate, and there's no obvious > candidate, so I think you're it, sorry :-) If Guido isn't keen, I'd be willing to cover it as the current runpy module maintainer and the one that updated this part of the interpreter startup sequence to handle the switch to importlib in 3.3. The PEP itself doesn't actually touch any of that internal machinery though, it just makes the capability a bit more discoverable and user-friendly. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Feb 24 10:09:09 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Feb 2015 19:09:09 +1000 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54EC38EA.2060306@egenix.com> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> Message-ID: On 24 February 2015 at 18:40, M.-A. Lemburg wrote: > On 23.02.2015 23:50, Nick Coghlan wrote: >> On 24 February 2015 at 08:40, Nick Coghlan wrote: >>> On 24 February 2015 at 07:39, Mark Lawrence wrote: >>>> On 23/02/2015 21:27, Serhiy Storchaka wrote: >>>>> >>>>> On 23.02.15 21:58, Joao S. O. Bueno wrote: >>>>>> >>>>>> That happens all the time, and is this use case that should possibly >>>>>> be addressed here - maybe >>>>>> something as simple as adding a couple of paragraphs to different places >>>>>> in the documentation could mitigate the issue. (in contrast to make a >>>>>> tons of otherwise valid code >>>>>> to become deprecated in a couple releases). >>>>> >>>>> >>>>> The problem is that the user don't know that he should read the >>>>> documentation. It just find that his script works with "C:\sample.txt", >>>>> but doesn't work with "D:\test.txt". He has no ideas what happen. >>>> >>>> Isn't this why users have help desks? >>> >>> Most don't, and cases like "\n" or "\t" in a Windows path name being >>> converted to whitespace are utterly impossible to look up in an >>> internet search when they fail, so a user learning on their own gets >>> left with a broken program and no particularly effective ways to ask >>> for help figuring it out. > > I think the easiest way would be to tweak the error message > output to indicate the real problem. > > At the moment, you get: > >>>> open('c:\test.txt') > Traceback (most recent call last): > File "", line 1, in > FileNotFoundError: [Errno 2] No such file or directory: 'c:\test.txt' > > which isn't helpful. > > It would be better to display the escapes as what they are: > > FileNotFoundError: [Errno 2] No such file or directory: 'c: est.txt' > > or use a special representation function to highlight the > possible problem: > > FileNotFoundError: [Errno 2] No such file or directory: 'c:[\t]est.txt' - warning: embedded escapes > > with the representation being: > > "'c:[\t]est.txt' - warning: embedded escapes" > > Such a change doesn't require a deprecation phase or a PEP. Oh, I like that idea, especially the variant that includes a search friendly phrase like "embedded escapes" in the error message. And unlike changing the handling of string literals generally, the impact would be isolated to a few exception __repr__ implementations. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Tue Feb 24 10:21:41 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 24 Feb 2015 11:21:41 +0200 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> Message-ID: On 24.02.15 11:09, Nick Coghlan wrote: > Oh, I like that idea, especially the variant that includes a search > friendly phrase like "embedded escapes" in the error message. And > unlike changing the handling of string literals generally, the impact > would be isolated to a few exception __repr__ implementations. It can require changing code in tens of places like ``PyErr_Format(FooError, "File %R is not valid FOO", filename)`` or ``raise FooError("File %r is not valid FOO" % (filename,))``. And of course 'c:\test.txt' is valid name on Unix. From rosuav at gmail.com Tue Feb 24 11:58:55 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Feb 2015 21:58:55 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54EC38EA.2060306@egenix.com> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> Message-ID: On Tue, Feb 24, 2015 at 7:40 PM, M.-A. Lemburg wrote: > I think the easiest way would be to tweak the error message > output to indicate the real problem. > > At the moment, you get: > >>>> open('c:\test.txt') > Traceback (most recent call last): > File "", line 1, in > FileNotFoundError: [Errno 2] No such file or directory: 'c:\test.txt' > > which isn't helpful. The problem isn't the cases where the file can't be found. Those can be dealt with fairly easily, one way or another. The problem is much, much earlier, when the student was using "c:\sample.txt" and everything was working fine. Then he changed the code over to use his own file instead of the provided sample, and at the same time, switched from using open() to using csv.reader(open()), and moved all the code into a function, and fixed three other bugs, and now it isn't working. And he can't figure out why. That's why I'd like "c:\sample.txt" to raise a warning. Should I start writing up a PEP? Is that the way forward with this? ChrisA From breamoreboy at yahoo.co.uk Tue Feb 24 12:11:23 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 24 Feb 2015 11:11:23 +0000 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> Message-ID: On 23/02/2015 23:32, Serhiy Storchaka wrote: > On 24.02.15 00:50, Nick Coghlan wrote: >>> - SyntaxError in Python 3.7 > > I think that SyntaxError could wait to 4.0. > >> Another suggestion: we may want to turn this particular deprecation >> warning on by default in the interactive interpreter, and recommend >> that other interactive interpreter developers (such as the IPython >> folks) consider doing the same. > > Why not turn all deprecation warning on by default in the interactive > interpreter? > In any given release how many of these would you see? I wouldn't mind seeing a few lines whenever I ran the II, but if the number started getting into tens or hundreds, at best I'd find it extremely irritating. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From jsbueno at python.org.br Tue Feb 24 13:00:53 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 24 Feb 2015 09:00:53 -0300 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54EB9E0D.4070000@canterbury.ac.nz> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> Message-ID: On 23 February 2015 at 18:39, Greg Ewing wrote: > Serhiy Storchaka wrote: >> >> The problem is that the user don't know that he should read the >> documentation. It just find that his script works with "C:\sample.txt", but >> doesn't work with "D:\test.txt". He has no ideas what happen. > > > Even with the syntax error, there are filenames that > will mysteriously fail, e.g. "C:\ninjamoves.txt". That is the reason I urge folks to think about the root cause, and check if this proposal is the best way to tackle it: The failing cases are the ones that won't be affected by this change at all - as they are valid escaped strings! There will be no error for c:\tmp\myfile.txt - it will ever be just "file not found" - unless the warning comes to using "\" as file path separator (and no, I don think just doing that would properly address the issue as well). Could we just use Guido's time machine and go back to the point in time where some MS head decided to go with "\" instead of "/", and deliver a well placed punch? :-) Or maybe have IOError for "file not found" to get some fuzzy logic and display a more verbose error message if there are "\n"s and "\t"s on the filename and platform is Windows? I think that could go along the proposal for deprecating non-escaping "\" sequences . By the way, although I am writing to get the root issue covered, I think deprecating them would be a good thing, yes. Despite my previous comment on deprecating code that works, on a deeper though it _is_ worth the issue. js -><- > > -- > Greg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br From thomas at python.org Tue Feb 24 14:00:47 2015 From: thomas at python.org (Thomas Wouters) Date: Tue, 24 Feb 2015 14:00:47 +0100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> Message-ID: On Tue, Feb 24, 2015 at 1:00 PM, Joao S. O. Bueno wrote: > On 23 February 2015 at 18:39, Greg Ewing > wrote: > > Serhiy Storchaka wrote: > >> > >> The problem is that the user don't know that he should read the > >> documentation. It just find that his script works with "C:\sample.txt", > but > >> doesn't work with "D:\test.txt". He has no ideas what happen. > > > > > > Even with the syntax error, there are filenames that > > will mysteriously fail, e.g. "C:\ninjamoves.txt". > > > That is the reason I urge folks to think about the root cause, > and check if this proposal is the best way to tackle it: The failing cases are the ones that won't be affected by this change at all > - > as they are valid escaped strings! > I would argue that some cases working is the root cause of the confusion. If *none* of the cases work, it would be a lot less confusing, even if the error message is different. Trying to make the error messages more similar, or more identifying, may be a good idea (as long as they aren't misleading when people *meant* to use escape sequences in a string) but fortunately that's separate from making the cases that current silently and misleadingly work, emit a warning (or error.) Emitting a warning does not address the question 'why does "C:\test" not exist when I see it in Explorer.' It does address the question 'why does my program break when I change "C:\Test" to "C:\test". It also addresses 'why does my regexp not work when I change "\d+" to "\b\d+", and unlike changing the error message for opening files it is much closer, in time and space, to the actual problem. It's a helpful thing regardless of what other changes are made. I do expect there to need to be a much longer deprecation period than one or even two releases, though: there is a lot of code that uses the misfeature, mostly by accident (not realizing they should be using r'', etc.) > There will be no error for > c:\tmp\myfile.txt - it will ever be just "file not found" - unless the > warning comes to > using "\" as file path separator (and no, I don think just doing that > would properly > address the issue as well). > > Could we just use Guido's time machine and go back to the point in time > where > some MS head decided to go with "\" instead of "/", and deliver a well > placed punch? :-) > > Or maybe have IOError for "file not found" to get some fuzzy logic and > display a more verbose error message if there are "\n"s and "\t"s on > the filename > and platform is Windows? I think that could go along the proposal for > deprecating non-escaping "\" sequences . > > By the way, although I am writing to get the root issue covered, I think > deprecating them would be a good thing, yes. Despite my previous comment > on deprecating code that works, on a deeper though it _is_ worth the issue. > > js > -><- > > > > > > > -- > > Greg > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > > https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/thomas%40python.org > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Feb 24 14:15:55 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Feb 2015 00:15:55 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> Message-ID: On Wed, Feb 25, 2015 at 12:00 AM, Thomas Wouters wrote: > I do expect there to need to be a much longer deprecation period than one or > even two releases, though: there is a lot of code that uses the misfeature, > mostly by accident (not realizing they should be using r'', etc.) I don't mind how long the deprecation period is, as long as there can be an option to Python that makes a noisy warning. The mixing of tabs and spaces needed Python 3.0 to become an error, but "python -tt" is my go-to solution for anyone using Python 2, tabs set to four spaces, and an editor that doesn't highlight indentation mismatches. Now, maybe we can't get this option backported to 2.7 (I'd like to, but I don't think this is on par with hash randomization!), but we could at least have Python 3.5 or 3.6 accept an option "python3 -e" to warn on unrecognized escapes, and (paralleling the others) "-ee" to raise SyntaxErrors. Eventually, even if the deprecation lasts until Python 4.0 or beyond, we could at least depend on being able to call up the warnings. ChrisA From stephen at xemacs.org Tue Feb 24 16:05:29 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 25 Feb 2015 00:05:29 +0900 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> Message-ID: <87a903e4ee.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Angelico writes: > I don't mind how long the deprecation period is, as long as there can > be an option to Python that makes a noisy warning. If that's OK with you and for the use case you explicitly described, though, a .bat file that runs a linter first might be the better option since (a) you don't have to wait and (b) you can put any bugaboo you like in there and (c) it can warn about syntacticly correct strings that "just fail" and may be hard to find. I think the Zen is right on, here: Now is better than never. Although never is often better than *right* now. I don't have a good sense for which it is, though, partly because (a) I don't program on Windows, but more importantly, (b) many of the dangerous strings actually used won't generate warnings or errors ever. From lac at openend.se Tue Feb 24 16:53:38 2015 From: lac at openend.se (Laura Creighton) Date: Tue, 24 Feb 2015 16:53:38 +0100 Subject: [Python-Dev] easy_install ? Message-ID: <201502241553.t1OFrcht028700@fido.openend.se> Hello all, I wonder what the status of easy_install is. I keep finding people who needed to install something 'path.py' is the latest, who needed to use pip, and couldn't get easy_install to work. Should we tell people that easy_install is deprecated, or ask them to file bugs when they could not get it to work, or ... Laura From brett at python.org Tue Feb 24 17:30:21 2015 From: brett at python.org (Brett Cannon) Date: Tue, 24 Feb 2015 16:30:21 +0000 Subject: [Python-Dev] easy_install ? References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: On Tue Feb 24 2015 at 10:54:14 AM Laura Creighton wrote: > Hello all, > I wonder what the status of easy_install is. I keep finding people > who needed to install something 'path.py' is the latest, who needed to > use pip, and couldn't get easy_install to work. Should we tell people > that easy_install is deprecated, or ask them to file bugs when > they could not get it to work, or ... > Tell people to use pip. Having ensurepip in Python 2.7 and 3.4 makes it as official as anything will be as the recommended tool to install projects. Otherwise easy_install has nothing to do directly with python-dev so I don't think we can comment on a group as to what people should do in terms of bugs, etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Feb 24 17:37:56 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Feb 2015 08:37:56 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> Message-ID: <54ECA8E4.1010306@stoneleaf.us> On 02/24/2015 02:58 AM, Chris Angelico wrote: > On Tue, Feb 24, 2015 at 7:40 PM, M.-A. Lemburg wrote: >> I think the easiest way would be to tweak the error message >> output to indicate the real problem. >> >> At the moment, you get: >> >>>>> open('c:\test.txt') >> Traceback (most recent call last): >> File "", line 1, in >> FileNotFoundError: [Errno 2] No such file or directory: 'c:\test.txt' >> >> which isn't helpful. > > The problem isn't the cases where the file can't be found. Those can > be dealt with fairly easily, one way or another. The problem is much, > much earlier, when the student was using "c:\sample.txt" and > everything was working fine. Then he changed the code over to use his > own file instead of the provided sample, and at the same time, > switched from using open() to using csv.reader(open()), and moved all > the code into a function, and fixed three other bugs, and now it isn't > working. And he can't figure out why. > > That's why I'd like "c:\sample.txt" to raise a warning. > > Should I start writing up a PEP? Is that the way forward with this? Go for it. I think the solution will be two pieces: - deprecating unknown backslash-escapes, with the goal of eventually being a SyntaxError (this removes a wart from the language, as well as allowing a warning backport to 2.7) - M.A.Lemburg's idea of changing the exception message in key places to make a successful backslash replace obvious (FileNotFoundError: [Errno 2] No such file or directory: 'c:[\t]est.txt' - warning: embedded escapes) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Tue Feb 24 17:44:20 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Feb 2015 16:44:20 +0000 Subject: [Python-Dev] easy_install ? In-Reply-To: References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: On 24 February 2015 at 16:30, Brett Cannon wrote: > Tell people to use pip. Having ensurepip in Python 2.7 and 3.4 makes it as > official as anything will be as the recommended tool to install projects. > Otherwise easy_install has nothing to do directly with python-dev so I don't > think we can comment on a group as to what people should do in terms of > bugs, etc. Bug reports would go to the setuptools project, but I agree, "use pip" is a better suggestion. And if pip won't work, it would be good to know why. Paul From ethan at stoneleaf.us Tue Feb 24 17:48:43 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Feb 2015 08:48:43 -0800 Subject: [Python-Dev] easy_install ? In-Reply-To: References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: <54ECAB6B.20402@stoneleaf.us> On 02/24/2015 08:44 AM, Paul Moore wrote: > On 24 February 2015 at 16:30, Brett Cannon wrote: >> >> Tell people to use pip. Having ensurepip in Python 2.7 and 3.4 makes it as >> official as anything will be as the recommended tool to install projects. >> Otherwise easy_install has nothing to do directly with python-dev so I don't >> think we can comment on a group as to what people should do in terms of >> bugs, etc. > > Bug reports would go to the setuptools project, but I agree, "use pip" > is a better suggestion. And if pip won't work, it would be good to > know why. And if/when pip doesn't work, a bug may need to be filed the project trying to be installed. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From lac at openend.se Tue Feb 24 17:52:13 2015 From: lac at openend.se (Laura Creighton) Date: Tue, 24 Feb 2015 17:52:13 +0100 Subject: [Python-Dev] easy_install ? In-Reply-To: Message from Paul Moore of "Tue, 24 Feb 2015 16:44:20 +0000." References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: <201502241652.t1OGqDoX009095@fido.openend.se> In a message of Tue, 24 Feb 2015 16:44:20 +0000, Paul Moore writes: >On 24 February 2015 at 16:30, Brett Cannon wrote: >> Tell people to use pip. Having ensurepip in Python 2.7 and 3.4 makes it as >> official as anything will be as the recommended tool to install projects. >> Otherwise easy_install has nothing to do directly with python-dev so I don't >> think we can comment on a group as to what people should do in terms of >> bugs, etc. > >Bug reports would go to the setuptools project, but I agree, "use pip" >is a better suggestion. And if pip won't work, it would be good to >know why. > >Paul I want to recommend that people not use easy_install. This has been my personal preferred solution for many years now. But I did not want to say such a thing if there was a known use case, one that I never have, that made easy_install better. Ok. will go back to my personal integrity which matches what python-dev recommends. Thank you for answering. Laura (answering questions in python-list and as webmaster) From thomas at python.org Tue Feb 24 18:18:38 2015 From: thomas at python.org (Thomas Wouters) Date: Tue, 24 Feb 2015 18:18:38 +0100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54ECA8E4.1010306@stoneleaf.us> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> Message-ID: On Tue, Feb 24, 2015 at 5:37 PM, Ethan Furman wrote: > On 02/24/2015 02:58 AM, Chris Angelico wrote: > > On Tue, Feb 24, 2015 at 7:40 PM, M.-A. Lemburg wrote: > >> I think the easiest way would be to tweak the error message > >> output to indicate the real problem. > >> > >> At the moment, you get: > >> > >>>>> open('c:\test.txt') > >> Traceback (most recent call last): > >> File "", line 1, in > >> FileNotFoundError: [Errno 2] No such file or directory: 'c:\test.txt' > >> > >> which isn't helpful. > > > > The problem isn't the cases where the file can't be found. Those can > > be dealt with fairly easily, one way or another. The problem is much, > > much earlier, when the student was using "c:\sample.txt" and > > everything was working fine. Then he changed the code over to use his > > own file instead of the provided sample, and at the same time, > > switched from using open() to using csv.reader(open()), and moved all > > the code into a function, and fixed three other bugs, and now it isn't > > working. And he can't figure out why. > > > > That's why I'd like "c:\sample.txt" to raise a warning. > > > > Should I start writing up a PEP? Is that the way forward with this? > > Go for it. > > I think the solution will be two pieces: > > - deprecating unknown backslash-escapes, with the goal of eventually > being a SyntaxError > (this removes a wart from the language, as well as allowing a warning > backport to 2.7) > > - M.A.Lemburg's idea of changing the exception message in key places to > make a successful > backslash replace obvious > (FileNotFoundError: [Errno 2] No such file or directory: > 'c:[\t]est.txt' - warning: embedded escapes) > Any ideas on how this check could be implemented? How would it distinguish \t from an actual tab in the string literal, or "C:\x64-python" from "C:\d-python"...? > > -- > ~Ethan~ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/thomas%40python.org > > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 24 18:44:21 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 09:44:21 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> Message-ID: TBH I think this will be a tough sell. I worry that an meme will emerge to make all string literals use raw mode, or worse, to deprecate non-raw string literals. I'm not sure how we'll get out of the whole conundrum, and I'm not in a hurry to see this in 3.5. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 24 18:46:27 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 09:46:27 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: I can do it but I don't want to be reviewing and accepting a PEP that's still under discussion, and I don't have the bandwidth to follow the discussion here -- I can only read the PEP. I will start that now. On Tue, Feb 24, 2015 at 1:03 AM, Nick Coghlan wrote: > On 24 February 2015 at 06:32, Paul Moore wrote: > > On 23 February 2015 at 19:47, Guido van Rossum wrote: > >> So is the PEP ready for pronouncement or should there be more > discussion? > > > > I think Brett's idea is worth incorporating, so let's thrash that out > first. > > > >> Also, do you have a BDFL-delegate or do you want me to review it? > > > > No-one has stepped up as BDFL-delegate, and there's no obvious > > candidate, so I think you're it, sorry :-) > > If Guido isn't keen, I'd be willing to cover it as the current runpy > module maintainer and the one that updated this part of the > interpreter startup sequence to handle the switch to importlib in 3.3. > > The PEP itself doesn't actually touch any of that internal machinery > though, it just makes the capability a bit more discoverable and > user-friendly. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Feb 24 18:46:50 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Feb 2015 09:46:50 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> Message-ID: <54ECB90A.1050606@stoneleaf.us> On 02/24/2015 09:18 AM, Thomas Wouters wrote: > On Tue, Feb 24, 2015 at 5:37 PM, Ethan Furman wrote: >> >> - M.A.Lemburg's idea of changing the exception message in key places to make a successful >> backslash replace obvious >> (FileNotFoundError: [Errno 2] No such file or directory: 'c:[\t]est.txt' - warning: embedded escapes) > > Any ideas on how this check could be implemented? How would it distinguish \t > from an actual tab in the string literal, or "C:\x64-python" from "C:\d-python"...? --> 'a tab' 'a\ttab' --> 'a\ttab' 'a\ttab' So the phrase "embedded escapes" may not always be accurate, but will be something to search on, and will help explain what is being seen. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Tue Feb 24 18:52:59 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Feb 2015 17:52:59 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 24 February 2015 at 17:46, Guido van Rossum wrote: > I can do it but I don't want to be reviewing and accepting a PEP that's > still under discussion, and I don't have the bandwidth to follow the > discussion here -- I can only read the PEP. I will start that now. I'm just about to push an update based on Brett's comments, then it should be done. Can you hold off for a short while? Thanks. Paul. (Sorry, always the way - final comments appear as soon as you say "it's done" :-)) From ethan at stoneleaf.us Tue Feb 24 19:00:07 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Feb 2015 10:00:07 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> Message-ID: <54ECBC27.7050702@stoneleaf.us> On 02/24/2015 09:44 AM, Guido van Rossum wrote: > > TBH I think this will be a tough sell. I worry that an meme will emerge to make all > string literals use raw mode, or worse, to deprecate non-raw string literals. I'm > not sure how we'll get out of the whole conundrum, and I'm not in a hurry to see > this in 3.5. So should we move forward with just the better error messages? It's a good first step, and even if it ends up being the only step it would be a huge improvement. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From guido at python.org Tue Feb 24 19:14:48 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 10:14:48 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54ECBC27.7050702@stoneleaf.us> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> <54ECBC27.7050702@stoneleaf.us> Message-ID: On Tue, Feb 24, 2015 at 10:00 AM, Ethan Furman wrote: > On 02/24/2015 09:44 AM, Guido van Rossum wrote: > > > > TBH I think this will be a tough sell. I worry that an meme will emerge > to make all > > string literals use raw mode, or worse, to deprecate non-raw string > literals. I'm > > not sure how we'll get out of the whole conundrum, and I'm not in a > hurry to see > > this in 3.5. > > So should we move forward with just the better error messages? It's a > good first step, and even if it ends up being the > only step it would be a huge improvement. > This is about messages from failing file-open operations if the filename contains an escaped character? I'd go slow there too: here are a lot of places where files are opened and messages are printed, both in the C code and in the stdlib. I'm not sure I buy the argument that just echoing the repr() of the name back doesn't help -- the escapes in there are actually useful in case a filename containing garbage chars (or even a trailing space) was read from some other source. And I'd weigh the needs of users who know what they are doing somewhat higher than educating newbies through error messages. While newbies are most likely to try out open() with a string literal, in "real" programs that is rare, and filenames are typically taken from the command line or from some other source where the peculiarities of Python string literals are irrelevant. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Tue Feb 24 19:31:35 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 24 Feb 2015 13:31:35 -0500 Subject: [Python-Dev] easy_install ? In-Reply-To: References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: On Tue, Feb 24, 2015 at 11:44 AM, Paul Moore wrote: > And if pip won't work, it would be good to > know why. > Is there a recommended way to invoke pip from setup.py? When I specify "tests_require=" and run "python setup.py test", the requirements get installed using setuptools' easy_install function. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Feb 24 19:47:10 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Feb 2015 10:47:10 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> <54ECBC27.7050702@stoneleaf.us> Message-ID: <54ECC72E.1070705@stoneleaf.us> On 02/24/2015 10:14 AM, Guido van Rossum wrote: > This is about messages from failing file-open operations if the filename contains an escaped character? I'd go slow > there too: here are a lot of places where files are opened and messages are printed, both in the C code and in the > stdlib. I'm not sure I buy the argument that just echoing the repr() of the name back doesn't help -- the escapes in > there are actually useful in case a filename containing garbage chars (or even a trailing space) was read from some > other source. I can attest from my impoverished Windows programming days that looking at --> os.listdir('c:\temp') SomeErrorMessage about syntax 'c:\temp' is not very helpful. There is zero visual indication that the \ and the t are one character, not two. Changing that error message to: SomeErrorMessage about syntax 'c:[\t]emp' or SomeErrorMessage about syntax 'c:\x07emp' or something that shouts out, "hey! one character in this location!" would be a good thing. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From guido at python.org Tue Feb 24 19:49:56 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 10:49:56 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54ECC72E.1070705@stoneleaf.us> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> <54ECBC27.7050702@stoneleaf.us> <54ECC72E.1070705@stoneleaf.us> Message-ID: I like the \x07 solution. On Tue, Feb 24, 2015 at 10:47 AM, Ethan Furman wrote: > On 02/24/2015 10:14 AM, Guido van Rossum wrote: > > > This is about messages from failing file-open operations if the filename > contains an escaped character? I'd go slow > > there too: here are a lot of places where files are opened and messages > are printed, both in the C code and in the > > stdlib. I'm not sure I buy the argument that just echoing the repr() of > the name back doesn't help -- the escapes in > > there are actually useful in case a filename containing garbage chars > (or even a trailing space) was read from some > > other source. > > I can attest from my impoverished Windows programming days that looking at > > --> os.listdir('c:\temp') > SomeErrorMessage about syntax 'c:\temp' > > is not very helpful. There is zero visual indication that the \ and the t > are one character, not two. Changing that > error message to: > > SomeErrorMessage about syntax 'c:[\t]emp' > > or > > SomeErrorMessage about syntax 'c:\x07emp' > > or something that shouts out, "hey! one character in this location!" > would be a good thing. > > -- > ~Ethan~ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 24 19:56:51 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 10:56:51 -0800 Subject: [Python-Dev] str(IntEnum) In-Reply-To: <93B631C3-DC57-4DB7-9DCA-8AFCD6D23FBE@gmail.com> References: <5D84B808-5AB3-4FD4-94F1-A997548D7353@gmail.com> <5C3F4038-92F6-47DB-A9E8-6A811AB04924@gmail.com> <20150220183602.GA429@tonks> <93B631C3-DC57-4DB7-9DCA-8AFCD6D23FBE@gmail.com> Message-ID: On Sat, Feb 21, 2015 at 8:39 AM, Demian Brecht wrote: > > > > On Feb 20, 2015, at 7:03 PM, Ian Cordasco > wrote: > > I hope this helps. > > It does, as do the other replies, thanks all. To be clear, my first gripe > has stemmed into two related (but very minor) problems: > > 1. IntEnum.__str__. I understand the reasoning behind the current > implementation. Personally I?d still prefer it to be consistent with int > (and other types as shown above) and if I wanted to know where it came > from, I could use repr(), but I understand that this /was/ previously > thought out and is contrary to the decision made. I?m fine with being in > the minority (or on my own as it seems in this case) and leaving it alone > (with the only caveat being the next point). > You are indeed very much out of sync here. There is no requirement or even convention that int subclasses have the same str() as plain ints -- in fact a custom str() is a common reason. For example, it's one of the reasons why I eventually caved in and added bool to the language. People want the custom representation used when they say print(x), they don't want to have to say print(repr(x)). > 2. Consistency of __str__ semantics across the standard library and > builtins. I believe that the value of __str__ is something that I, as a > user, should be able to infer when using disparate types. Unfortunately, > some represent the values while other represent the object themselves, > nearly the same problem that __repr__ solves minus the requirement of being > a valid Python expression where possible. In my mind, a clear separation > between __str__ representing the value of an instance and __repr__ > representing the object, or where the value came from (and perhaps /not/ > having the auto-fallback) makes sense, but would only be one potential > solution to promoting consistency. > Again, this is an area where you are on your own looking for consistency. str() is what print() uses (by mutual definition) and print() should print whatever is the most useful as the default for a given use case. There is *no* rule that says a subclass cannot change the str(). > In any event, there are many larger problems to be solved (that is, if > anyone else /does/ consider this a problem) and I very well may be on my > own with this thinking. > It sure seems that way. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 24 20:01:47 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 11:01:47 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On Tue, Feb 24, 2015 at 10:50 AM, Paul Moore wrote: > On 24 February 2015 at 18:24, Guido van Rossum wrote: > > Here's my review. I really like where this is going but I have a few > > questions and suggestions (I can't help myself :-). > > Thanks. > > > [I sneaked a peek at the update you sent to peps at python.org.] > > > > "Currently, pyyzer [5] and pex [6] are two tools known to exist." -> "... > > are two such tools." > > > > It's not stated whether the archive names include the .pyz[w] extension > or > > not (though I'm guessing it's not -- this should be stated). > > No, I assumed that automatically adding extensions is not the norm. A > lot of Windows tools do it, but I don't get the impression Unix tools > do. I'll clarify the wording. > > > The naming of the functions feels inconsistent -- maybe pack(directory, > > target) -> create_archive(directory, archive), and set_interpreter() -> > > copy_archive(archive, new_archive)? > > I'm OK with create_archive. But not so sure about copy_archive - the > purpose of that function is solely to change the shebang line on an > existing archive. It does so by copying, sure, but that's incidental. > I see get_interpreter/set_interpreter as complementary "helper" APIs, > and create_archive as the main API. > To me, the name set_() so strongly evokes the idea of in-place modification that I cannot live with it. (I also assume that your code won't do the right thing if the output is the same as the input, right?) > > Why no command-line equivalent for the other two methods? I propose the > > following interface: if there's only one positional argument, we're > asking > > to print its shebang line; if there are two and the input position is an > > archive instead of a directory, we're copying. (In the future people > will > > want an option to print more stuff, e.g. the main function or even a full > > listing.) > > Mainly because I couldn't think of an API that didn't look muddled, or > make the simple case of "I want to pack up a directory" look messy. > But I'll think about your proposal - it sounds nice, I'll see how it > fares when I try to implement it :-) > > > I've not seen the pkg.mod:fn notation before. Where is this taken from? > > Why can't it be pkg.mod.fn? > > It's common usage in setuptools and things like tat - it's the "entry > point" syntax used in packaging. I don't know of any reason why > all-dotted notation wouldn't work, though. I'll ask around if there's > any reason I'm not aware of. > OK, no need to change this then. > > I'd specify that when the output argument is a file open for writing, it > is > > the caller's responsibility to close the file. Also, can the file be a > > pipe? (I.e. are we using seek()/tell() or not?) And what about the > input > > archive? Can that be a file open for reading? > > I'll clarify all of these points. They are mostly "it can be whatever > the zipfile module accepts", though, which isn't explicitly stated > itself :-( > > The input can't be a file object for pack/create_archive because it's > a directory, not an archive, there. But it could be for > set_interpreter. I just wasn't sure if it was worth adding. It's not > hard to do, though. I'm not sure what use case this is addressing anyway, but assuming there is one, you might as well go all the way (except for directories, obviously). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 24 20:03:47 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 24 Feb 2015 14:03:47 -0500 Subject: [Python-Dev] easy_install ? In-Reply-To: References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: On Tue, Feb 24, 2015 at 1:31 PM, Alexander Belopolsky wrote: > > On Tue, Feb 24, 2015 at 11:44 AM, Paul Moore wrote: >> >> And if pip won't work, it would be good to >> know why. > > > Is there a recommended way to invoke pip from setup.py? When I specify > "tests_require=" and run "python setup.py test", the requirements get > installed using setuptools' easy_install function. The solution is to not do that. A substitute is to specify your test requirements in a [test] extra and install them with pip or to run tests with tox. This gives control of the installer back to the user instead of the setup.py author. https://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their- own-dependencies From guido at python.org Tue Feb 24 19:58:37 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 10:58:37 -0800 Subject: [Python-Dev] Fwd: Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: [Sorry, accidentally dropped the list from this message.] Here's my review. I really like where this is going but I have a few questions and suggestions (I can't help myself :-). [I sneaked a peek at the update you sent to peps at python.org.] "Currently, pyyzer [5] and pex [6] are two tools known to exist." -> "... are two such tools." It's not stated whether the archive names include the .pyz[w] extension or not (though I'm guessing it's not -- this should be stated). The naming of the functions feels inconsistent -- maybe pack(directory, target) -> create_archive(directory, archive), and set_interpreter() -> copy_archive(archive, new_archive)? Why no command-line equivalent for the other two methods? I propose the following interface: if there's only one positional argument, we're asking to print its shebang line; if there are two and the input position is an archive instead of a directory, we're copying. (In the future people will want an option to print more stuff, e.g. the main function or even a full listing.) I've not seen the pkg.mod:fn notation before. Where is this taken from? Why can't it be pkg.mod.fn? I'd specify that when the output argument is a file open for writing, it is the caller's responsibility to close the file. Also, can the file be a pipe? (I.e. are we using seek()/tell() or not?) And what about the input archive? Can that be a file open for reading? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Feb 24 20:07:36 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Feb 2015 11:07:36 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> <54ECBC27.7050702@stoneleaf.us> <54ECC72E.1070705@stoneleaf.us> Message-ID: <54ECCBF8.9010106@stoneleaf.us> On 02/24/2015 10:49 AM, Guido van Rossum wrote: > On Tue, Feb 24, 2015 at 10:47 AM, Ethan Furman wrote: >> >> I can attest from my impoverished Windows programming days that looking at >> >> --> os.listdir('c:\temp') >> SomeErrorMessage about syntax 'c:\temp' >> >> is not very helpful. There is zero visual indication that the \ and the t >> are one character, not two. Changing that error message to: >> >> SomeErrorMessage about syntax 'c:[\t]emp' >> >> or >> >> SomeErrorMessage about syntax 'c:\x07emp' >> >> or something that shouts out, "hey! one character in this location!" would be a good thing. > > I like the \x07 solution. So final question is do we take the easy road and change the repr for str to always use hex or unicode escapes instead of simple backslash-escapes (and then run for our lives), or do we just hunt down and change the appropriate error messages for files (and possibly re) ? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From demianbrecht at gmail.com Tue Feb 24 20:12:06 2015 From: demianbrecht at gmail.com (Demian Brecht) Date: Tue, 24 Feb 2015 11:12:06 -0800 Subject: [Python-Dev] str(IntEnum) In-Reply-To: References: <5D84B808-5AB3-4FD4-94F1-A997548D7353@gmail.com> <5C3F4038-92F6-47DB-A9E8-6A811AB04924@gmail.com> <20150220183602.GA429@tonks> <93B631C3-DC57-4DB7-9DCA-8AFCD6D23FBE@gmail.com> Message-ID: <837B0A73-4CD5-4B49-8D65-EC6F6D03A45B@gmail.com> > On Feb 24, 2015, at 10:56 AM, Guido van Rossum wrote: > It sure seems that way. Thanks for the additional feedback Guido. I?d spent some further time thinking about what it was that I was looking for and determined it was bollocks. The proposal was a poor solution to a specific immediate problem without taking more general use cases into account. My apologies for the noise. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dholth at gmail.com Tue Feb 24 20:23:05 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 24 Feb 2015 14:23:05 -0500 Subject: [Python-Dev] Fwd: Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On Tue, Feb 24, 2015 at 1:58 PM, Guido van Rossum wrote: > [Sorry, accidentally dropped the list from this message.] > > Here's my review. I really like where this is going but I have a few > questions and suggestions (I can't help myself :-). > > [I sneaked a peek at the update you sent to peps at python.org.] > > "Currently, pyyzer [5] and pex [6] are two tools known to exist." -> "... > are two such tools." > > It's not stated whether the archive names include the .pyz[w] extension or > not (though I'm guessing it's not -- this should be stated). > > The naming of the functions feels inconsistent -- maybe pack(directory, > target) -> create_archive(directory, archive), and set_interpreter() -> > copy_archive(archive, new_archive)? > > Why no command-line equivalent for the other two methods? I propose the > following interface: if there's only one positional argument, we're asking > to print its shebang line; if there are two and the input position is an > archive instead of a directory, we're copying. (In the future people will > want an option to print more stuff, e.g. the main function or even a full > listing.) > > I've not seen the pkg.mod:fn notation before. Where is this taken from? > Why can't it be pkg.mod.fn? Translates to import pkg.mod; pkg.mod.fn() with no exception handling to figure out which part is importable. pkg.mod:ob.prop.fn would turn into import pkg.mod; pkg.mod.ob.prop.fn() > I'd specify that when the output argument is a file open for writing, it is > the caller's responsibility to close the file. Also, can the file be a > pipe? (I.e. are we using seek()/tell() or not?) And what about the input > archive? Can that be a file open for reading? It seems like the very next thing I would want after trying pack() would be to pass a callback that returns True iff a file should be included in the archive. After that I might just want a ZipFile subclass or a regular ZipFile to which I could add my own files? "return ZipFile with shebang already filled in". It's hard for me to say where the boundary between the convenience API and re-implementing this simple thing yourself if you have complex needs should be. From greg at krypto.org Tue Feb 24 20:33:14 2015 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 24 Feb 2015 19:33:14 +0000 Subject: [Python-Dev] getstatusoutput()'s behavior changed in 3.3.4 and 3.4 Message-ID: While porting some code from 2.7 to 3.4 I discovered that command.getstatusoutput() (renamed to subprocess.getstatusoutput() in 3.x) had changed. Surprise! The code was working under an earlier version of 3.3 but broke when I ran it on 3.4. Nowhere was this documented that I could find. Tracking down what changed, I discovered it was unintentional due to the meaning of the returned status int never being tested. http://bugs.python.org/issue23508 filed... Given it has shipped in several stable 3.x releases and as part of some major Linux distros, reverting the behavior change seems wrong. The new behavior is nicer and more consistent with the rest of the subprocess module. I suggest just documenting it and moving on. It seems too late to fix this mistake without causing additional headaches. Anyone disagree? -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 24 20:33:57 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Feb 2015 19:33:57 +0000 Subject: [Python-Dev] Fwd: Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 24 February 2015 at 19:23, Daniel Holth wrote: >> I'd specify that when the output argument is a file open for writing, it is >> the caller's responsibility to close the file. Also, can the file be a >> pipe? (I.e. are we using seek()/tell() or not?) And what about the input >> archive? Can that be a file open for reading? > > It seems like the very next thing I would want after trying pack() > would be to pass a callback that returns True iff a file should be > included in the archive. After that I might just want a ZipFile > subclass or a regular ZipFile to which I could add my own files? > "return ZipFile with shebang already filled in". It's hard for me to > say where the boundary between the convenience API and re-implementing > this simple thing yourself if you have complex needs should be. Yes, it's a slippery slope. The whole API is a pretty thin wrapper over a ZipFile object. I'd rather keep it to the most basic requirements, and defer anything even slightly complicated to the ZipFile API. The one exception is the set_interpreter/get_interpreter APIs, which are messy to do with ZipFile, and a pain to do "by hand" (because working with part-text, part-binary files is just naturally messy). It would be possible to write up how easy it is to create a pyz file by hand using the zipfile module, but doing so would (IMO) lose the nice simple message of this PEP - "use zipapp to bundle your code into an app that's supported all the way back to Python 2.6". Paul From greg.ewing at canterbury.ac.nz Tue Feb 24 20:54:08 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 Feb 2015 08:54:08 +1300 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> Message-ID: <54ECD6E0.1010708@canterbury.ac.nz> Chris Angelico wrote: > Then he changed the code over to use his > own file instead of the provided sample, and at the same time, > switched from using open() to using csv.reader(open()), and moved all > the code into a function, and fixed three other bugs, and now it isn't > working. And he can't figure out why. There's probably a useful meta-lesson in there: test after making every change! -- Greg From greg.ewing at canterbury.ac.nz Tue Feb 24 21:16:43 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 Feb 2015 09:16:43 +1300 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> Message-ID: <54ECDC2B.2050708@canterbury.ac.nz> Thomas Wouters wrote: > Trying to make the > error messages more similar, or more identifying, may be a good idea (as > long as they aren't misleading when people *meant* to use escape > sequences in a string) It seems that Windows won't let you use control characters in filenames, so there is room for a more pointed error message from functions that take pathnames on Windows. -- Greg From p.f.moore at gmail.com Tue Feb 24 21:20:26 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Feb 2015 20:20:26 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 24 February 2015 at 18:58, Guido van Rossum wrote: > Why no command-line equivalent for the other two methods? I propose the > following interface: if there's only one positional argument, we're asking > to print its shebang line; if there are two and the input position is an > archive instead of a directory, we're copying. (In the future people will > want an option to print more stuff, e.g. the main function or even a full > listing.) Thinking about this, there are 3 main uses: 1. Create an archive 2. Print the shebang 3. Change the shebang Of these, (1) is the crucial one. Basic usage should be python -m zipapp mydir [-o anothername.pyz] [-p interpreter] [-m entry:point] This zips up mydir to create an archive mydir.pyz. Options to change the target name, set a shebang line (side note: --python/-p or --interpreter/-i?) and set the entry point, I see this as pretty non-negotiable, this is the key use case that needs to be as simple as possible. To print the shebang, we could use python -m zipapp myapp.pyz --show This allows for future expansion by adding options, although most other things you might want to do (list the files, display __main__.py) can be done with a standard zip utility. I'm not keen on the option name --show, but I can't think of anything substantially better. To modify an archive could be done using python -m zipapp old.pyz new.pyz [-p interpreter] Default is to strip the shebang (no -p option). There's no option to omit the target and do an inplace update because I feel the default action (strip the shebang from the existing file with no backup) is too dangerous. To be explicit, "python -m zipapp app.pyz" will fail with a message "In-place editing of python zip applications is not supported". That seems to work. Open questions: 1. To create an archive, use -o target for an explicit target name, or just "target". The former is more conventional, the latter consistent with modification. Or we could make modification use a (mandatory) -o option. 2. -p/--python or -i/--interpreter for the shebang setting option 3. What to call the "show the shebang line" option Paul From tjreedy at udel.edu Tue Feb 24 21:26:35 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 24 Feb 2015 15:26:35 -0500 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> <54ECBC27.7050702@stoneleaf.us> Message-ID: On 2/24/2015 1:14 PM, Guido van Rossum wrote: > And I'd weigh the needs of users who know what they are doing somewhat > higher than educating newbies through error messages. While newbies are > most likely to try out open() with a string literal, in "real" programs > that is rare, and filenames are typically taken from the command line or > from some other source where the peculiarities of Python string literals > are irrelevant. I have been thinking about proposing a new how-to: Understanding Error Messages, with a section on tracebacks followed by an alphabetical listing of Exceptions that gives people problems, with possible solutions for each. The following begins my first draft. FileNotFoundError: Possible cause [Windows]: You used a normal string literal to create a filename, you used '\' as the path separator, and you forgot that Python (like most languages) treats '\' as a case-sensitive escape character. For example: "C:\Test" is 7 chars and works as a file name, while 'C:\test' is 6 chars, one a literal tab character. The latter does not work as a file name and will raise FileNotFoundError. Possible solutions: 1. Use raw string literals for Windows path names (r'C:\test'). 2 (recommended). Use '/' as the path separator ('C:/test'), just as one does on other systems. This always works when passing file names from Python to Windows, even though it sometimes does not work in Windows Command Prompt console. -- Terry Jan Reedy From barry at python.org Tue Feb 24 21:32:46 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 24 Feb 2015 15:32:46 -0500 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: <20150224153246.0b42c6fe@anarchist.wooz.org> On Feb 24, 2015, at 08:20 PM, Paul Moore wrote: >(side note: --python/-p or --interpreter/-i?) and set the entry point, Both virtualenv and (I think) pex use --python/-p so that seems to be the overwhelming trend . >To modify an archive could be done using > > python -m zipapp old.pyz new.pyz [-p interpreter] > >Default is to strip the shebang (no -p option). There's no option to >omit the target and do an inplace update because I feel the default >action (strip the shebang from the existing file with no backup) is >too dangerous. You have to be careful about the case where old.pyz == new.pyz (e.g. either handling this case safely or complaining loudly) , but also you could handle it by using a .tmp file and renaming. E.g. old.pyz -> old.pyz.bak and old.pyz.tmp -> old.pyz. >3. What to call the "show the shebang line" option I don't know how useful this is, given that (on *nix at least) you can effectively do the same with head(1). Cheers, -Barry From brett at python.org Tue Feb 24 21:41:27 2015 From: brett at python.org (Brett Cannon) Date: Tue, 24 Feb 2015 20:41:27 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On Tue Feb 24 2015 at 3:21:30 PM Paul Moore wrote: > On 24 February 2015 at 18:58, Guido van Rossum wrote: > > Why no command-line equivalent for the other two methods? I propose the > > following interface: if there's only one positional argument, we're > asking > > to print its shebang line; if there are two and the input position is an > > archive instead of a directory, we're copying. (In the future people > will > > want an option to print more stuff, e.g. the main function or even a full > > listing.) > > Thinking about this, there are 3 main uses: > > 1. Create an archive > 2. Print the shebang > 3. Change the shebang > > Of these, (1) is the crucial one. > > Basic usage should be > > python -m zipapp mydir [-o anothername.pyz] [-p interpreter] [-m > entry:point] > > This zips up mydir to create an archive mydir.pyz. Options to change > the target name, set a shebang line (side note: --python/-p or > --interpreter/-i?) and set the entry point, > > I see this as pretty non-negotiable, this is the key use case that > needs to be as simple as possible. > > To print the shebang, we could use > > python -m zipapp myapp.pyz --show > > This allows for future expansion by adding options, although most > other things you might want to do (list the files, display > __main__.py) can be done with a standard zip utility. I'm not keen on > the option name --show, but I can't think of anything substantially > better. > > To modify an archive could be done using > > python -m zipapp old.pyz new.pyz [-p interpreter] > > Default is to strip the shebang (no -p option). There's no option to > omit the target and do an inplace update because I feel the default > action (strip the shebang from the existing file with no backup) is > too dangerous. > > To be explicit, "python -m zipapp app.pyz" will fail with a message > "In-place editing of python zip applications is not supported". > > That seems to work. > > Open questions: > > 1. To create an archive, use -o target for an explicit target name, or > just "target". The former is more conventional, the latter consistent > with modification. Or we could make modification use a (mandatory) -o > option. > EIBTI suggests requiring the -o. Pragmatic suggests just [in] [out] and use context based on what kind of thing [in] points at as well as whether -p is specified and whether it has an argument, which is the most minimal UX you can have. Question is whether you can screw up by specifying the wrong thing somehow (you might have to require that [out] doesn't already exist to make it work). > 2. -p/--python or -i/--interpreter for the shebang setting option > Since you are going to be using `python -m pyzip` then -i/--interpreter is less redundant-looking on the command-line. > 3. What to call the "show the shebang line" option As suggested above, -p w/o an argument could do it, otherwise --show or --info seems fine (I like --shebang, but that will probably be tough on non-English speakers). -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 24 21:45:16 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 12:45:16 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 486 - Make the Python Launcher aware of virtual environments In-Reply-To: References: Message-ID: Thanks for doing this, Paul! I think this is a fine PEP and I have no comments. (Note that I am not much of a Windows user so I am just taking most of the things you claim at face value. :-) I will accept the PEP in 1-3 days unless a storm of objections gets piled onto this thread. --Guido On Sat, Feb 21, 2015 at 1:31 AM, Paul Moore wrote: > The discussion on PEP 486 (started at > https://mail.python.org/pipermail/python-dev/2015-February/138171.html, > following from the thread at > https://mail.python.org/pipermail/python-dev/2015-February/138160.html) > seems to have died down. There's an implementation at > http://bugs.python.org/issue23465. > > So, I think this is ready for pronouncement. > > Thanks, > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 24 21:51:00 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Feb 2015 20:51:00 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <20150224153246.0b42c6fe@anarchist.wooz.org> References: <54EB7DDE.9030405@stoneleaf.us> <20150224153246.0b42c6fe@anarchist.wooz.org> Message-ID: On 24 February 2015 at 20:32, Barry Warsaw wrote: >>To modify an archive could be done using >> >> python -m zipapp old.pyz new.pyz [-p interpreter] >> >>Default is to strip the shebang (no -p option). There's no option to >>omit the target and do an inplace update because I feel the default >>action (strip the shebang from the existing file with no backup) is >>too dangerous. > > You have to be careful about the case where old.pyz == new.pyz (e.g. either > handling this case safely or complaining loudly) , but also you could handle > it by using a .tmp file and renaming. E.g. old.pyz -> old.pyz.bak and > old.pyz.tmp -> old.pyz. There are a *lot* of obscure failure modes here. What if old and new are symlinks (or hard links) to the same file? What if a .tmp file already exists? What if the user hits Ctrl-C at a bad moment? On the principle of keeping it simple, I prefer just requiring a target, giving an error if the source name and target name are the same (which still leaves loopholes for the determined fool on case insensitive filesystems :-)) and just documenting that inplace modification isn't supported. The PEP clearly states that it's *minimal* tooling, after all... > >>3. What to call the "show the shebang line" option > > I don't know how useful this is, given that (on *nix at least) you can > effectively do the same with head(1). I don't think it's that useful, TBH (although will head not print binary junk if there is no shebang line?) I quite like Brett's suggestion of --info, and maybe be a bit verbose: $ python -m zipapp foo.pyz --info Interpreter: /usr/bin/python $ python -m zipapp bar.pyz --info Interpreter: I can't see it being useful for scripting, and if it matters, there's always get_interpreter() then. It's mainly just as a diagnostic for people who are getting the wrong interpreter invoked. Paul From ethan at stoneleaf.us Tue Feb 24 22:00:43 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Feb 2015 13:00:43 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> <20150224153246.0b42c6fe@anarchist.wooz.org> Message-ID: <54ECE67B.7000203@stoneleaf.us> On 02/24/2015 12:51 PM, Paul Moore wrote: > I don't think it's that useful, TBH (although will head not print > binary junk if there is no shebang line?) I quite like Brett's > suggestion of --info, and maybe be a bit verbose: > > $ python -m zipapp foo.pyz --info > Interpreter: /usr/bin/python > $ python -m zipapp bar.pyz --info > Interpreter: I like that! +1 -- ~Ethan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ijmorlan at uwaterloo.ca Tue Feb 24 21:56:17 2015 From: ijmorlan at uwaterloo.ca (Isaac Morland) Date: Tue, 24 Feb 2015 15:56:17 -0500 (EST) Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54ECCBF8.9010106@stoneleaf.us> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> <54ECBC27.7050702@stoneleaf.us> <54ECC72E.1070705@stoneleaf.us> <54ECCBF8.9010106@stoneleaf.us> Message-ID: On Tue, 24 Feb 2015, Ethan Furman wrote: > On 02/24/2015 10:49 AM, Guido van Rossum wrote: >> On Tue, Feb 24, 2015 at 10:47 AM, Ethan Furman wrote: >>> >>> I can attest from my impoverished Windows programming days that looking at >>> >>> --> os.listdir('c:\temp') >>> SomeErrorMessage about syntax 'c:\temp' >>> >>> is not very helpful. There is zero visual indication that the \ and the t >>> are one character, not two. Changing that error message to: >>> >>> SomeErrorMessage about syntax 'c:[\t]emp' >>> >>> or >>> >>> SomeErrorMessage about syntax 'c:\x07emp' >>> >>> or something that shouts out, "hey! one character in this >>> location!" would be a good thing. >> >> I like the \x07 solution. > > So final question is do we take the easy road and change the repr for > str to always use hex or unicode escapes instead of simple > backslash-escapes (and then run for our lives), or do we just hunt down > and change the appropriate error messages for files (and possibly re) ? Just a data point from a random programmer: I like the \x07 solution for the error message as it draws attention to the character at issue, but I also like to see \n, \t etc. in the result of repr because it is more readable for the common cases. Isaac Morland CSCF Web Guru DC 2619, x36650 WWW Software Specialist From ethan at stoneleaf.us Tue Feb 24 22:09:01 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Feb 2015 13:09:01 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54ECE67B.7000203@stoneleaf.us> References: <54EB7DDE.9030405@stoneleaf.us> <20150224153246.0b42c6fe@anarchist.wooz.org> <54ECE67B.7000203@stoneleaf.us> Message-ID: <54ECE86D.7060308@stoneleaf.us> On 02/24/2015 01:00 PM, Ethan Furman wrote: > On 02/24/2015 12:51 PM, Paul Moore wrote: >> $ python -m zipapp foo.pyz --info >> Interpreter: /usr/bin/python >> $ python -m zipapp bar.pyz --info >> Interpreter: Another way to support this is with subcommands. Have the default [implicit] command be to create the zip app, and then add any subcommands we need: python -m zipapp [create] foo #creates a foo.pyz from the foo directory python -m zipapp info foo.pyz # prints out shebang for foo.pyz python -m zipapp info --all foo.pyz # prints out shebang and directory structure and files and .... This easily leaves the door open to add new commands in the future if any are still needed, and makes the current commands easy and simple to use. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From alexander.belopolsky at gmail.com Tue Feb 24 22:21:19 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 24 Feb 2015 16:21:19 -0500 Subject: [Python-Dev] easy_install ? In-Reply-To: References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: On Tue, Feb 24, 2015 at 2:03 PM, Daniel Holth wrote: > > > Is there a recommended way to invoke pip from setup.py? When I specify > > "tests_require=" and run "python setup.py test", the requirements get > > installed using setuptools' easy_install function. > > The solution is to not do that. A substitute is to specify your test > requirements in a [test] extra and install them with pip or to run > tests with tox. This gives control of the installer back to the user > instead of the setup.py author. Isn't this a chicken and egg problem? I currently have tests_require=['tox'], and this is exactly what tox recommends: https://testrun.org/tox/latest/example/basic.html#integration-with-setuptools-distribute-test-commands Note that my CI box is a CentOS 6.5 with Python 2.6.6, setuptools 0.6. This is still a very common server configuration. What is the recommended way to bootstrap tox in such environment? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Feb 24 22:24:24 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 24 Feb 2015 16:24:24 -0500 Subject: [Python-Dev] easy_install ? In-Reply-To: References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: That might mostly do what you want, since tox could install any additional test requirements based on its configuration. On Tue, Feb 24, 2015 at 4:21 PM, Alexander Belopolsky wrote: > > On Tue, Feb 24, 2015 at 2:03 PM, Daniel Holth wrote: >> >> > Is there a recommended way to invoke pip from setup.py? When I specify >> > "tests_require=" and run "python setup.py test", the requirements get >> > installed using setuptools' easy_install function. >> >> The solution is to not do that. A substitute is to specify your test >> requirements in a [test] extra and install them with pip or to run >> tests with tox. This gives control of the installer back to the user >> instead of the setup.py author. > > > Isn't this a chicken and egg problem? I currently have > > tests_require=['tox'], > > and this is exactly what tox recommends: > > https://testrun.org/tox/latest/example/basic.html#integration-with-setuptools-distribute-test-commands > > > Note that my CI box is a CentOS 6.5 with Python 2.6.6, setuptools 0.6. This > is still a very common server configuration. What is the recommended way to > bootstrap tox in such environment? > From alexander.belopolsky at gmail.com Tue Feb 24 22:32:46 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 24 Feb 2015 16:32:46 -0500 Subject: [Python-Dev] easy_install ? In-Reply-To: References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: On Tue, Feb 24, 2015 at 4:24 PM, Daniel Holth wrote: > > That might mostly do what you want, since tox could install any > additional test requirements based on its configuration. Does "that" refer to using tests_require=['tox'] as I described below? This means using easy_install implicitly and being exposed to easy_install bugs. Note that my attempts to avoid easy_install so far led to dependency hell starting from the need to install virtualenv. Unlike setuptools, pip does not seem to be able to install dependencies in a temporary directory. It wants a full-blown venv tree. > > > On Tue, Feb 24, 2015 at 4:21 PM, Alexander Belopolsky > wrote: > > > > On Tue, Feb 24, 2015 at 2:03 PM, Daniel Holth wrote: > >> > >> > Is there a recommended way to invoke pip from setup.py? When I specify > >> > "tests_require=" and run "python setup.py test", the requirements get > >> > installed using setuptools' easy_install function. > >> > >> The solution is to not do that. A substitute is to specify your test > >> requirements in a [test] extra and install them with pip or to run > >> tests with tox. This gives control of the installer back to the user > >> instead of the setup.py author. > > > > > > Isn't this a chicken and egg problem? I currently have > > > > tests_require=['tox'], > > > > and this is exactly what tox recommends: > > > > https://testrun.org/tox/latest/example/basic.html#integration-with-setuptools-distribute-test-commands > > > > > > Note that my CI box is a CentOS 6.5 with Python 2.6.6, setuptools 0.6. This > > is still a very common server configuration. What is the recommended way to > > bootstrap tox in such environment? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 24 22:35:44 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 13:35:44 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54ECEC3D.5040701@g.nevcal.com> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> <54ECBC27.7050702@stoneleaf.us> <54ECC72E.1070705@stoneleaf.us> <54ECEC3D.5040701@g.nevcal.com> Message-ID: [Adding back python-dev] Actually, I wasn't proposing to change repr() -- my sentiments are similar to Isaac Morland's. Only the error message for the most basic file open() call should be tweaked. No solution is perfect -- but it's probably common enough for someone to create a file or folder named C:\test and being stumped by "cannot open 'C:\test'." On Tue, Feb 24, 2015 at 1:25 PM, Glenn Linderman wrote: > On 2/24/2015 10:49 AM, Guido van Rossum wrote: > > I like the \x07 solution. > > > The more I think about it, the more I think this is a sufficient > solution. People that use repr to get a Python-compatible string syntax > for further program use get one. People that see them in an error message > are much less likely to be fooled into thinking it is two characters, > because they are thinking of it as two characters to start with. > > On the other hand, I have a directory full of "throw away" experimental > source files named x01, x02, x03, ... :) > > And I suppose extensive use of certain characters in repr in intermediate > or archived data could make such data grow in size. > > And while \t and \n are very commonly used escapes, maybe string repr > could have a special function called > .I_promise_not_to_report_issues_when_I_use_backslash_escapes_in_character_literals_and_get_confused() > which would switch back from \x07 to \t and \x0a to \n, etc. This would be > callable only from __main__ :) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 24 22:43:07 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Feb 2015 21:43:07 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54ECE86D.7060308@stoneleaf.us> References: <54EB7DDE.9030405@stoneleaf.us> <20150224153246.0b42c6fe@anarchist.wooz.org> <54ECE67B.7000203@stoneleaf.us> <54ECE86D.7060308@stoneleaf.us> Message-ID: On 24 February 2015 at 21:09, Ethan Furman wrote: > Another way to support this is with subcommands. Have the default [implicit] command be to create the zip app, and then > add any subcommands we need: > > python -m zipapp [create] foo #creates a foo.pyz from the foo directory > > python -m zipapp info foo.pyz # prints out shebang for foo.pyz > > python -m zipapp info --all foo.pyz # prints out shebang and directory structure and files and .... > > This easily leaves the door open to add new commands in the future if any are still needed, and makes the current > commands easy and simple to use. I didn't know an implicit subcommand was allowed. That would work, although it does mean that you can't build a pyz from a directory "info" without explicitly using the create subcommand. I think I'm going to go with "python -m zipapp foo.pyz --info" (And an -o option for the target, mandatory when the source is a pyz file, and --python for the interpreter). It's not set in stone yet, so if anyone objects, there's still time to change my mind. Paul From ethan at stoneleaf.us Tue Feb 24 23:21:14 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Feb 2015 14:21:14 -0800 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECA8E4.1010306@stoneleaf.us> <54ECBC27.7050702@stoneleaf.us> <54ECC72E.1070705@stoneleaf.us> <54ECEC3D.5040701@g.nevcal.com> Message-ID: <54ECF95A.2000609@stoneleaf.us> On 02/24/2015 01:35 PM, Guido van Rossum wrote: > Actually, I wasn't proposing to change repr() -- my sentiments are similar > to Isaac Morland's. Only the error message for the most basic file open() > call should be tweaked. As are mine -- I just like to be thorough when discussing possibilities. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ncoghlan at gmail.com Tue Feb 24 23:32:05 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Feb 2015 08:32:05 +1000 Subject: [Python-Dev] getstatusoutput()'s behavior changed in 3.3.4 and 3.4 In-Reply-To: References: Message-ID: Documentation (and adding the missing test) sounds right to me. We'd at least want a "versionchanged" note on the function itself, and an entry in the porting section of the (3.3? 3.4?) what's new guide. Is there anywhere else we might want to mention it? Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 24 23:44:09 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Feb 2015 08:44:09 +1000 Subject: [Python-Dev] easy_install ? In-Reply-To: References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: On 25 Feb 2015 07:23, "Alexander Belopolsky" wrote: > > > On Tue, Feb 24, 2015 at 2:03 PM, Daniel Holth wrote: > > > > > Is there a recommended way to invoke pip from setup.py? When I specify > > > "tests_require=" and run "python setup.py test", the requirements get > > > installed using setuptools' easy_install function. > > > > The solution is to not do that. A substitute is to specify your test > > requirements in a [test] extra and install them with pip or to run > > tests with tox. This gives control of the installer back to the user > > instead of the setup.py author. > > > Isn't this a chicken and egg problem? I currently have > > tests_require=['tox'], > > and this is exactly what tox recommends: > > https://testrun.org/tox/latest/example/basic.html#integration-with-setuptools-distribute-test-commands > > > Note that my CI box is a CentOS 6.5 with Python 2.6.6, setuptools 0.6. This is still a very common server configuration. What is the recommended way to bootstrap tox in such environment? If running in the system Python isn't absolutely essential, then the Python 2.7 collection from softwarecollections.org is the preferred way to get a newer Python 2 (including pip et al) on CentOS. You can also get access to Python 3 that way. Failing that, pip & virtualenv are also available from the EPEL 6 repos. Both of those approaches rely on the system package manager to do the bootstrapping of the Python specific tooling. If both softwarecollections.org and EPEL are considered unacceptable dependencies, then you're going to have to do your own bootstrapping for PyPI access on CentOS (which may include relying on easy_install to bootstrap pip and/or virtualenv) Regards, Nick. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Feb 24 23:42:16 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Feb 2015 09:42:16 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <54ECD6E0.1010708@canterbury.ac.nz> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EC38EA.2060306@egenix.com> <54ECD6E0.1010708@canterbury.ac.nz> Message-ID: On Wed, Feb 25, 2015 at 6:54 AM, Greg Ewing wrote: > Chris Angelico wrote: >> >> Then he changed the code over to use his >> own file instead of the provided sample, and at the same time, >> switched from using open() to using csv.reader(open()), and moved all >> the code into a function, and fixed three other bugs, and now it isn't >> working. And he can't figure out why. > > > There's probably a useful meta-lesson in there: > test after making every change! Of course! But when you're working with students remotely, you don't necessarily even know that that's what's happened. All you get is "It works with sample.txt, but it doesn't work with my file". You may or may not even get to see the code at first. Firm believer in "commit often" policy... ChrisA From ncoghlan at gmail.com Tue Feb 24 23:54:38 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Feb 2015 08:54:38 +1000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> <20150224153246.0b42c6fe@anarchist.wooz.org> Message-ID: On 25 Feb 2015 06:52, "Paul Moore" wrote: > > On 24 February 2015 at 20:32, Barry Warsaw wrote: > >>To modify an archive could be done using > >> > >> python -m zipapp old.pyz new.pyz [-p interpreter] > >> > >>Default is to strip the shebang (no -p option). There's no option to > >>omit the target and do an inplace update because I feel the default > >>action (strip the shebang from the existing file with no backup) is > >>too dangerous. > > > > You have to be careful about the case where old.pyz == new.pyz (e.g. either > > handling this case safely or complaining loudly) , but also you could handle > > it by using a .tmp file and renaming. E.g. old.pyz -> old.pyz.bak and > > old.pyz.tmp -> old.pyz. > > There are a *lot* of obscure failure modes here. What if old and new > are symlinks (or hard links) to the same file? What if a .tmp file > already exists? What if the user hits Ctrl-C at a bad moment? > > On the principle of keeping it simple, I prefer just requiring a > target, giving an error if the source name and target name are the > same (which still leaves loopholes for the determined fool on case > insensitive filesystems :-)) and just documenting that inplace > modification isn't supported. The PEP clearly states that it's > *minimal* tooling, after all... https://docs.python.org/3/library/os.path.html#os.path.samefile covers this check in a robust, cross-platform way. > >>3. What to call the "show the shebang line" option > > > > I don't know how useful this is, given that (on *nix at least) you can > > effectively do the same with head(1). > > I don't think it's that useful, TBH (although will head not print > binary junk if there is no shebang line?) I quite like Brett's > suggestion of --info, and maybe be a bit verbose: > > $ python -m zipapp foo.pyz --info > Interpreter: /usr/bin/python > $ python -m zipapp bar.pyz --info > Interpreter: > > I can't see it being useful for scripting, and if it matters, there's > always get_interpreter() then. It's mainly just as a diagnostic for > people who are getting the wrong interpreter invoked. The corresponding CLI option for the inspect module is "--details": https://docs.python.org/3/library/inspect.html#command-line-interface (By default "python -m inspect " prints the module source code) Cheers, Nick. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 24 23:54:22 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Feb 2015 14:54:22 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> <20150224153246.0b42c6fe@anarchist.wooz.org> Message-ID: Maybe just fail if the target name already exists? On Tue, Feb 24, 2015 at 12:51 PM, Paul Moore wrote: > On 24 February 2015 at 20:32, Barry Warsaw wrote: > >>To modify an archive could be done using > >> > >> python -m zipapp old.pyz new.pyz [-p interpreter] > >> > >>Default is to strip the shebang (no -p option). There's no option to > >>omit the target and do an inplace update because I feel the default > >>action (strip the shebang from the existing file with no backup) is > >>too dangerous. > > > > You have to be careful about the case where old.pyz == new.pyz (e.g. > either > > handling this case safely or complaining loudly) , but also you could > handle > > it by using a .tmp file and renaming. E.g. old.pyz -> old.pyz.bak and > > old.pyz.tmp -> old.pyz. > > There are a *lot* of obscure failure modes here. What if old and new > are symlinks (or hard links) to the same file? What if a .tmp file > already exists? What if the user hits Ctrl-C at a bad moment? > > On the principle of keeping it simple, I prefer just requiring a > target, giving an error if the source name and target name are the > same (which still leaves loopholes for the determined fool on case > insensitive filesystems :-)) and just documenting that inplace > modification isn't supported. The PEP clearly states that it's > *minimal* tooling, after all... > > > > >>3. What to call the "show the shebang line" option > > > > I don't know how useful this is, given that (on *nix at least) you can > > effectively do the same with head(1). > > I don't think it's that useful, TBH (although will head not print > binary junk if there is no shebang line?) I quite like Brett's > suggestion of --info, and maybe be a bit verbose: > > $ python -m zipapp foo.pyz --info > Interpreter: /usr/bin/python > $ python -m zipapp bar.pyz --info > Interpreter: > > I can't see it being useful for scripting, and if it matters, there's > always get_interpreter() then. It's mainly just as a diagnostic for > people who are getting the wrong interpreter invoked. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Feb 25 00:01:14 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Feb 2015 09:01:14 +1000 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <87a903e4ee.fsf@uwakimon.sk.tsukuba.ac.jp> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> <87a903e4ee.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 25 Feb 2015 01:07, "Stephen J. Turnbull" wrote: > > Chris Angelico writes: > > > I don't mind how long the deprecation period is, as long as there can > > be an option to Python that makes a noisy warning. > > If that's OK with you and for the use case you explicitly described, > though, a .bat file that runs a linter first might be the better > option since (a) you don't have to wait and (b) you can put any > bugaboo you like in there and (c) it can warn about syntacticly > correct strings that "just fail" and may be hard to find. The linter developers don't have a decision making process that lets them pursue things like this on their own. By contrast, if we went with a deprecation warning (and potentially a command line option) then the linter developers can backport the already defined check comparatively easily. Regards, Nick. > > I think the Zen is right on, here: > > Now is better than never. > Although never is often better than *right* now. > > I don't have a good sense for which it is, though, partly because (a) > I don't program on Windows, but more importantly, (b) many of the > dangerous strings actually used won't generate warnings or errors ever. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Feb 25 00:02:06 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 25 Feb 2015 00:02:06 +0100 Subject: [Python-Dev] getstatusoutput()'s behavior changed in 3.3.4 and 3.4 In-Reply-To: References: Message-ID: Hi, It looks like new tests are required to check that the behaviour will not change again. Victor Le mardi 24 f?vrier 2015, Gregory P. Smith a ?crit : > While porting some code from 2.7 to 3.4 I discovered that > command.getstatusoutput() (renamed to subprocess.getstatusoutput() in 3.x) > had changed. Surprise! > > The code was working under an earlier version of 3.3 but broke when I ran > it on 3.4. Nowhere was this documented that I could find. Tracking down > what changed, I discovered it was unintentional due to the meaning of the > returned status int never being tested. http://bugs.python.org/issue23508 > filed... > > Given it has shipped in several stable 3.x releases and as part of some > major Linux distros, reverting the behavior change seems wrong. The new > behavior is nicer and more consistent with the rest of the subprocess > module. I suggest just documenting it and moving on. It seems too late to > fix this mistake without causing additional headaches. Anyone disagree? > > -gps > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Feb 25 00:39:32 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 24 Feb 2015 18:39:32 -0500 Subject: [Python-Dev] easy_install ? In-Reply-To: References: <201502241553.t1OFrcht028700@fido.openend.se> Message-ID: The other option might be to use http://bitbucket.org/dholth/setup-requires It uses pip to install requirements into an isolated directory before setup.py runs, with pip, doing exactly what you requested. On Feb 24, 2015 5:44 PM, "Nick Coghlan" wrote: > > On 25 Feb 2015 07:23, "Alexander Belopolsky" < > alexander.belopolsky at gmail.com> wrote: > > > > > > On Tue, Feb 24, 2015 at 2:03 PM, Daniel Holth wrote: > > > > > > > Is there a recommended way to invoke pip from setup.py? When I > specify > > > > "tests_require=" and run "python setup.py test", the requirements get > > > > installed using setuptools' easy_install function. > > > > > > The solution is to not do that. A substitute is to specify your test > > > requirements in a [test] extra and install them with pip or to run > > > tests with tox. This gives control of the installer back to the user > > > instead of the setup.py author. > > > > > > Isn't this a chicken and egg problem? I currently have > > > > tests_require=['tox'], > > > > and this is exactly what tox recommends: > > > > > https://testrun.org/tox/latest/example/basic.html#integration-with-setuptools-distribute-test-commands > > > > > > Note that my CI box is a CentOS 6.5 with Python 2.6.6, setuptools 0.6. > This is still a very common server configuration. What is the recommended > way to bootstrap tox in such environment? > > If running in the system Python isn't absolutely essential, then the > Python 2.7 collection from softwarecollections.org is the preferred way > to get a newer Python 2 (including pip et al) on CentOS. You can also get > access to Python 3 that way. > > Failing that, pip & virtualenv are also available from the EPEL 6 repos. > > Both of those approaches rely on the system package manager to do the > bootstrapping of the Python specific tooling. > > If both softwarecollections.org and EPEL are considered unacceptable > dependencies, then you're going to have to do your own bootstrapping for > PyPI access on CentOS (which may include relying on easy_install to > bootstrap pip and/or virtualenv) > > Regards, > Nick. > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed Feb 25 00:36:11 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 24 Feb 2015 17:36:11 -0600 Subject: [Python-Dev] Request for Pronouncement: PEP 486 - Make the Python Launcher aware of virtual environments In-Reply-To: References: Message-ID: Neat, but there's a typo under rationale: # Execute an installed module (these could use python -m, # which is longer to type but is a little *mopre* similar to the # launcher approach) pip install pytest py.test s/mopre/more/ On Sat, Feb 21, 2015 at 3:31 AM, Paul Moore wrote: > The discussion on PEP 486 (started at > https://mail.python.org/pipermail/python-dev/2015-February/138171.html, > following from the thread at > https://mail.python.org/pipermail/python-dev/2015-February/138160.html) > seems to have died down. There's an implementation at > http://bugs.python.org/issue23465. > > So, I think this is ready for pronouncement. > > Thanks, > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gandkcrane at msn.com Wed Feb 25 02:15:42 2015 From: gandkcrane at msn.com (GARRY M CRANE) Date: Tue, 24 Feb 2015 17:15:42 -0800 Subject: [Python-Dev] ?s re documentation of Python features Message-ID: I am trying to learn Python for use in computational biology. I am using the interesting book: "Computing for Biologists; Python Programming and Principles" (by Ran Libeskind-Hadas and Eliot Bush). It has an interesting and useful set of programming exercises at www.cs.hmc.edu/CFB. I am actually enjoying solving (doing) the example problems. However, I find some of the instructions non-functional for me. For example the import function does not work, nor f=open("filename.txt"). I have saved files per instructions in the programming exercise inside the Python34 folder (I am using Python 3.4 in Windows 7). But use of the f=open() command produces an error message that the requested file does not exist. I assume I have chosen a wrong location for the saved file within that Python34 folder, but trial and error has not led to a successful use of these functions. import simply leaves a blank line .. no suggestion about the problem. Asking questions in Google and Ask about where to save Python-related files that can be used subsequently have not led to answers - just details about structuring or formatting things to be written/saved/use of the \n at end of each line, etc. Important details, but of no help. I am finding Python to be very handy at many biologic things such as working with DNA strings, etc. but I find the documentation and indexing for finding how to use many Python features exasperating. I am writing to you based on a READ ME file in my Python folder - generated when I installed Python. FYI, I asked a few questions of one of the authors of the interesting book - who politely replied he was too busy to answer right now - the book and problems were meant for a class ... though neither the book nor problems say so. The professor hopes to get around to issues of use by non-students sometime - but not now. Another feature I have come across so far that does not work is importation of matplotlib. I copy computed results (that otherwise would go to your plotting routine) then go to Excel and with manipulation produce a usable chart there - but at a cost of time and energy. Your Python tool has many intriguing features - but some of the most basic functions do not work for me (even though many features do, e.g., import random does work). The failure of these features - so far as I can tell - is because of lack of description (for the general non-expert public) about where/how to install various features. Perhaps I need to reinstall from the ground up??? If so, just what should I do? If there is a less drastic solution, can you tell me about it? Thank you for any help ... and if you could provide me a lead regarding WHERE to ask subsequent questions I would be most grateful. Sometimes, Google or Ask or a U Tube tutorial does a good job - but if one does not know the 'proper' name or term for something, it often is frustrating or impossible to get an answer. I have not heard about any comprehensive handbook for Python34 aimed at one who wants to use Python for creating programs (functions) that work - and is not an expert at back-room structure of files and folders.... have I simply missed it? So far, I have not found a local Python expert to ask for help. I am sure some are in the greater Seattle area where I live- but I don't know how to find even one at this time. Garry Crane gandkcrane at msn.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed Feb 25 02:43:52 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 24 Feb 2015 19:43:52 -0600 Subject: [Python-Dev] ?s re documentation of Python features In-Reply-To: References: Message-ID: Ask on python-list . Also check out the FAQ and the Help page . Not sure what your problem is; Python is EXTREMELY well documented. On Tue, Feb 24, 2015 at 7:15 PM, GARRY M CRANE wrote: > I am trying to learn Python for use in computational biology. I am using > the interesting book: "*Computing for Biologists; Python Programming and > Principles*" (by Ran Libeskind-Hadas and Eliot Bush). It has an > interesting and useful set of programming exercises at www.cs.hmc.edu/CFB. > I am actually enjoying solving (doing) the example problems. However, I > find some of the instructions non-functional for me. For example the *import > *function does not work, nor *f=open("filename.txt")*. I have saved > files per instructions in the programming exercise inside the Python34 > folder (I am using Python 3.4 in Windows 7). But use of the *f=open()* > command produces an error message that the requested file does not exist. I > assume I have chosen a wrong location for the saved file within that > Python34 folder, but trial and error has not led to a successful use of > these functions. *import* simply leaves a blank line .. no suggestion > about the problem. > > Asking questions in Google and Ask about where to save Python-related > files that can be used subsequently have not led to answers - just details > about structuring or formatting things to be written/saved/use of the \n at > end of each line, etc. Important details, but of no help. I am finding > Python to be very handy at many biologic things such as working with DNA > strings, etc. but I find the documentation and indexing for finding how to > use many Python features exasperating. I am writing to you based on a READ > ME file in my Python folder - generated when I installed Python. > > FYI, I asked a few questions of one of the authors of the interesting book > - who politely replied he was too busy to answer right now - the book and > problems were meant for a class ... though neither the book nor problems > say so. The professor hopes to get around to issues of use by non-students > sometime - but not now. > > Another feature I have come across so far that does not work is > importation of *matplotlib*. I copy computed results (that otherwise > would go to your plotting routine) then go to Excel and with manipulation > produce a usable chart there - but at a cost of time and energy. > > Your Python tool has many intriguing features - but some of the most > basic functions do not work for me (even though many features do, e.g., > import random does work). The failure of these features - so far as I can > tell - is because of lack of description (for the general non-expert > public) about where/how to install various features. Perhaps I need to > reinstall from the ground up??? If so, just what should I do? If there is > a less drastic solution, can you tell me about it? > > Thank you for any help ... and if you could provide me a lead regarding > WHERE to ask subsequent questions I would be most grateful. Sometimes, > Google or Ask or a U Tube tutorial does a good job - but if one does not > know the 'proper' name or term for something, it often is frustrating or > impossible to get an answer. I have not heard about any comprehensive > handbook for Python34 aimed at one who wants to use Python for creating > programs (functions) that work - and is not an expert at back-room > structure of files and folders.... have I simply missed it? So far, I have > not found a local Python expert to ask for help. I am sure some are in the > greater Seattle area where I live- but I don't know how to find even one at > this time. > > Garry Crane > gandkcrane at msn.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Wed Feb 25 02:56:21 2015 From: greg at krypto.org (Gregory P. Smith) Date: Wed, 25 Feb 2015 01:56:21 +0000 Subject: [Python-Dev] how to inspect if something includes a bound first param Message-ID: inspect.getargspec(method) and inspect.signature(method) both include the 'self' parameter but how are we to figure out from method itself that it is actually bound and that its first parameter is expected to be a bound instance? So far I've come up with: *inspect.ismethod(method) or inspect.ismethoddescriptor(method)* But that is still not quite right as it remains False in 3.4 for the simple case of: >>> class A: ... def x(self): ... pass ... >>> inspect.ismethod(A.x) False >>> inspect.ismethoddescriptor(A.x) False >>> inspect.signature(A.x).parameters mappingproxy(OrderedDict([('self', )])) >>> inspect.getargspec(A.x) ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None) The above works if use it on object.__init__, but not on a method of an uninstantiated class. (it also works on the instantiated A().x) Checking if *(inspect.isfunction(method) and method.__qualname__ != method.__name__)* perhaps but that seems questionably hacky. Is there a right way to do this via the inspect module that I've missed? It seems like that'd be nice, but I don't feel like I know enough to write up a feature request for it yet. (granted I hope nobody *wants* to write code that does this...) -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Wed Feb 25 03:02:38 2015 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 24 Feb 2015 21:02:38 -0500 Subject: [Python-Dev] how to inspect if something includes a bound first param In-Reply-To: References: Message-ID: <54ED2D3E.8020501@trueblade.com> On 2/24/2015 8:56 PM, Gregory P. Smith wrote: > inspect.getargspec(method) and inspect.signature(method) both include > the 'self' parameter but how are we to figure out from method itself > that it is actually bound and that its first parameter is expected to be > a bound instance? > > So far I've come up with: > /inspect.ismethod(method) or inspect.ismethoddescriptor(method)/ > > But that is still not quite right as it remains False in 3.4 for the > simple case of: > >>>> class A: > ... def x(self): > ... pass > ... >>>> inspect.ismethod(A.x) > False >>>> inspect.ismethoddescriptor(A.x) > False >>>> inspect.signature(A.x).parameters > mappingproxy(OrderedDict([('self', )])) >>>> inspect.getargspec(A.x) > ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None) > > The above works if use it on object.__init__, but not on a method of an > uninstantiated class. (it also works on the instantiated A().x) > > Checking if /(inspect.isfunction(method) and method.__qualname__ != > method.__name__)/ perhaps but that seems questionably hacky. Is there a > right way to do this via the inspect module that I've missed? > > It seems like that'd be nice, but I don't feel like I know enough to > write up a feature request for it yet. (granted I hope nobody /wants/ > to write code that does this...) I'm not sure if it's correct, but deep in a library of mine I have: elif type(fn) == types.MethodType: # bound method? if fn.im_self is None: # no 'self' nskip = 0 else: # need to supply 'self' nskip = 1 This is also a 2.x library. No idea if it works with 3.x. Eric. From tjreedy at udel.edu Wed Feb 25 04:21:29 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 24 Feb 2015 22:21:29 -0500 Subject: [Python-Dev] how to inspect if something includes a bound first param In-Reply-To: References: Message-ID: On 2/24/2015 8:56 PM, Gregory P. Smith wrote: > inspect.getargspec(method) and inspect.signature(method) both include > the 'self' parameter but how are we to figure out from method itself > that it is actually bound and that its first parameter is expected to be > a bound instance? This seems like a python-list question. > So far I've come up with: > /inspect.ismethod(method) or inspect.ismethoddescriptor(method)/ > > But that is still not quite right as it remains False in 3.4 for the > simple case of: > > >>> class A: > ... def x(self): > ... pass > ... > >>> inspect.ismethod(A.x) > False > >>> inspect.ismethoddescriptor(A.x) > False These are correct. A.x is the function resulting from execution of the def statement. >>> type(A.x) This is different from 2.x. If you call the function as a function, there is no requirement on its argument, and no binding. >>> A.x(3) # returns None >>> -- Terry Jan Reedy From tjreedy at udel.edu Wed Feb 25 04:25:29 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 24 Feb 2015 22:25:29 -0500 Subject: [Python-Dev] how to inspect if something includes a bound first param In-Reply-To: <54ED2D3E.8020501@trueblade.com> References: <54ED2D3E.8020501@trueblade.com> Message-ID: On 2/24/2015 9:02 PM, Eric V. Smith wrote: > I'm not sure if it's correct, but deep in a library of mine I have: > > elif type(fn) == types.MethodType: > # bound method? > if fn.im_self is None: > # no 'self' > nskip = 0 > else: > # need to supply 'self' > nskip = 1 > > This is also a 2.x library. No idea if it works with 3.x. It will not. 3.x does not have 'unbound methods' in the above sense, and for bound methods, fn.im_self is now fn.__self__. -- Terry Jan Reedy From stephen at xemacs.org Wed Feb 25 06:16:15 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 25 Feb 2015 14:16:15 +0900 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> <87a903e4ee.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <8761aqefkw.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > The linter developers don't have a decision making process that lets them > pursue things like this on their own. I'm not suggesting that the linter developers do any such thing. I'm suggesting that users (specifically Chris) customize their preferred linters (most permit that). What bothers me is that the hardest to understand failures cannot possibly be helped, because they are legal, meaningful syntax that conflicts with syntax common in external contexts. Replacing mnemonics like '\t' with '\x09' (or even '\u0009' in Python 3) in debugging output (including error messages) seems like a much better idea to me. From ben+python at benfinney.id.au Wed Feb 25 07:00:44 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 25 Feb 2015 17:00:44 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> <87a903e4ee.fsf@uwakimon.sk.tsukuba.ac.jp> <8761aqefkw.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <8561aq7coj.fsf@benfinney.id.au> "Stephen J. Turnbull" writes: > What bothers me is that the hardest to understand failures cannot > possibly be helped, because they are legal, meaningful syntax that > conflicts with syntax common in external contexts. There is a wider context here, too: semantics of the backslash escape commonly include ?backslash followed by a character not otherwise mentioned will produce that character, verbatim?. Proposals to change that for Python's string literals must account for the fact this will make Python's rules for backslash escape surprisingly different in this regard from many other usages of backslash escape. > Replacing mnemonics like '\t' with '\x09' (or even '\u0009' in Python > 3) in debugging output (including error messages) seems like a much > better idea to me. +1. This will not cause any valid behaviour today to fail, whereas it will make the common problems much more likely to be noticed by newcomers. -- \ ?I went to a fancy French restaurant called ?D?j? Vu?. The head | `\ waiter said, ?Don't I know you??? ?Steven Wright | _o__) | Ben Finney From rosuav at gmail.com Wed Feb 25 07:03:18 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Feb 2015 17:03:18 +1100 Subject: [Python-Dev] Emit SyntaxWarning on unrecognized backslash escapes? In-Reply-To: <8561aq7coj.fsf@benfinney.id.au> References: <54EB5CEB.90305@stoneleaf.us> <20150223204751.21c0cb1c@fsol> <54EB9E0D.4070000@canterbury.ac.nz> <87a903e4ee.fsf@uwakimon.sk.tsukuba.ac.jp> <8761aqefkw.fsf@uwakimon.sk.tsukuba.ac.jp> <8561aq7coj.fsf@benfinney.id.au> Message-ID: On Wed, Feb 25, 2015 at 5:00 PM, Ben Finney wrote: > There is a wider context here, too: semantics of the backslash escape > commonly include > ?backslash followed by a character not otherwise mentioned will produce > that character, verbatim?. > > Proposals to change that for Python's string literals must account for > the fact this will make Python's rules for backslash escape surprisingly > different in this regard from many other usages of backslash escape. That's different from Python's rule, which is that "backslash followed by a character not otherwise mentioned will produce *a backslash followed by* that character, verbatim". ChrisA From storchaka at gmail.com Wed Feb 25 10:11:28 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 25 Feb 2015 11:11:28 +0200 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 24.02.15 21:01, Guido van Rossum wrote: > On Tue, Feb 24, 2015 at 10:50 AM, Paul Moore > wrote: > > On 24 February 2015 at 18:24, Guido van Rossum > wrote: > > I'd specify that when the output argument is a file open for writing, it is > > the caller's responsibility to close the file. Also, can the file be a > > pipe? (I.e. are we using seek()/tell() or not?) And what about the input > > archive? Can that be a file open for reading? > > I'll clarify all of these points. They are mostly "it can be whatever > the zipfile module accepts", though, which isn't explicitly stated > itself :-( See issue23252. https://bugs.python.org/issue23252 From p.f.moore at gmail.com Wed Feb 25 10:55:45 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 Feb 2015 09:55:45 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> <20150224153246.0b42c6fe@anarchist.wooz.org> Message-ID: On 24 February 2015 at 22:54, Nick Coghlan wrote: >> On the principle of keeping it simple, I prefer just requiring a >> target, giving an error if the source name and target name are the >> same (which still leaves loopholes for the determined fool on case >> insensitive filesystems :-)) and just documenting that inplace >> modification isn't supported. The PEP clearly states that it's >> *minimal* tooling, after all... > > https://docs.python.org/3/library/os.path.html#os.path.samefile covers this > check in a robust, cross-platform way. Wow, I hadn't realised that samefile had become reliable on Windows (ino/dev always used to be unreliable years ago). It's the little things like this that sneak into new releases without much fanfare that make so much *actual* difference. Thanks to whoever implemented this, and to all the people putting in the little changes that make new versions so much better :-) Paul From p.f.moore at gmail.com Wed Feb 25 11:00:58 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 Feb 2015 10:00:58 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 24 February 2015 at 18:58, Guido van Rossum wrote: > The naming of the functions feels inconsistent -- maybe pack(directory, > target) -> create_archive(directory, archive), and set_interpreter() -> > copy_archive(archive, new_archive)? One possible source of confusion with copy_archive (and its command line equivalent "python -m zipapp old.pyz -o new.pyz") is that it isn't technically a copy, as it changes the shebang line (if you omit the interpreter argument it removes the existing shebang). We could change it to copy by default, but (a) that's redundant as a file copy works better, and (b) we'd need to add a method of specifying "remove the shebang" to replace omitting the interpreter arg. Is this a big enough issue to be worth changing the name of the function and the command line behaviour? I'm inclined to leave it, but mainly on the basis that I feel like I'm getting to the point of over-thinking things... Paul From ncoghlan at gmail.com Wed Feb 25 14:02:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Feb 2015 23:02:04 +1000 Subject: [Python-Dev] ?s re documentation of Python features In-Reply-To: References: Message-ID: On 25 February 2015 at 11:43, Ryan Gonzalez wrote: > Ask on python-list. Also check out the FAQ and the Help page. Not sure what > your problem is; Python is EXTREMELY well documented. The core Python docs aren't bad, but getting the full scientific Python stack up and running is still challenging for most folks. Garry, while Ryan is correct that python-list or python-tutor would be a better venue for this question (python-dev is for discussing the development *of* Python, rather than *with* Python), you may also be interested to know that getting the scientific Python stack up and running has its own dedicated section in the Python Packaging User's Guide: https://packaging.python.org/en/latest/science.html In particular, it is likely to be worth your time t to look at adopting one of the downstream Scientific Python distributions, rather than attempting to use upstream Python directly: http://www.scipy.org/install.html Regards, Nick. > > On Tue, Feb 24, 2015 at 7:15 PM, GARRY M CRANE wrote: >> >> I am trying to learn Python for use in computational biology. I am using >> the interesting book: "Computing for Biologists; Python Programming and >> Principles" (by Ran Libeskind-Hadas and Eliot Bush). It has an interesting >> and useful set of programming exercises at www.cs.hmc.edu/CFB. I am >> actually enjoying solving (doing) the example problems. However, I find >> some of the instructions non-functional for me. For example the import >> function does not work, nor f=open("filename.txt"). I have saved files per >> instructions in the programming exercise inside the Python34 folder (I am >> using Python 3.4 in Windows 7). But use of the f=open() command produces an >> error message that the requested file does not exist. I assume I have chosen >> a wrong location for the saved file within that Python34 folder, but trial >> and error has not led to a successful use of these functions. import simply >> leaves a blank line .. no suggestion about the problem. >> >> >> Asking questions in Google and Ask about where to save Python-related >> files that can be used subsequently have not led to answers - just details >> about structuring or formatting things to be written/saved/use of the \n at >> end of each line, etc. Important details, but of no help. I am finding >> Python to be very handy at many biologic things such as working with DNA >> strings, etc. but I find the documentation and indexing for finding how to >> use many Python features exasperating. I am writing to you based on a READ >> ME file in my Python folder - generated when I installed Python. >> >> FYI, I asked a few questions of one of the authors of the interesting book >> - who politely replied he was too busy to answer right now - the book and >> problems were meant for a class ... though neither the book nor problems say >> so. The professor hopes to get around to issues of use by non-students >> sometime - but not now. >> >> Another feature I have come across so far that does not work is >> importation of matplotlib. I copy computed results (that otherwise would go >> to your plotting routine) then go to Excel and with manipulation produce a >> usable chart there - but at a cost of time and energy. >> >> Your Python tool has many intriguing features - but some of the most basic >> functions do not work for me (even though many features do, e.g., import >> random does work). The failure of these features - so far as I can tell - >> is because of lack of description (for the general non-expert public) about >> where/how to install various features. Perhaps I need to reinstall from the >> ground up??? If so, just what should I do? If there is a less drastic >> solution, can you tell me about it? >> >> Thank you for any help ... and if you could provide me a lead regarding >> WHERE to ask subsequent questions I would be most grateful. Sometimes, >> Google or Ask or a U Tube tutorial does a good job - but if one does not >> know the 'proper' name or term for something, it often is frustrating or >> impossible to get an answer. I have not heard about any comprehensive >> handbook for Python34 aimed at one who wants to use Python for creating >> programs (functions) that work - and is not an expert at back-room structure >> of files and folders.... have I simply missed it? So far, I have not found >> a local Python expert to ask for help. I am sure some are in the greater >> Seattle area where I live- but I don't know how to find even one at this >> time. >> >> Garry Crane >> gandkcrane at msn.com >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com >> > > > > -- > Ryan > If anybody ever asks me why I prefer C++ to C, my answer will be simple: > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was > nul-terminated." > Personal reality distortion fields are immune to contradictory evidence. - > srean > Check out my website: http://kirbyfan64.github.io/ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From larry at hastings.org Wed Feb 25 15:33:59 2015 From: larry at hastings.org (Larry Hastings) Date: Wed, 25 Feb 2015 06:33:59 -0800 Subject: [Python-Dev] how to inspect if something includes a bound first param In-Reply-To: References: Message-ID: <54EDDD57.2040700@hastings.org> On 02/24/2015 05:56 PM, Gregory P. Smith wrote: > inspect.getargspec(method) and inspect.signature(method) both include > the 'self' parameter but how are we to figure out from method itself > that it is actually bound and that its first parameter is expected to > be a bound instance? Given the mechanisms involved, surely this question is a bit nonsensical? The function doesn't "expect" anything, it's just a function. (I remind you, Python 3 dropped the whole concept of an "unbound method".) If it happens to live inside a class, and it's accessed through an instance of the class, then the first parameter gets bound. Consider: >>> class A: ... def x(self, a): print(a) ... >>> a = A() inspect.signature(A.x).parameters has two parameters, "self" and "a". inspect.signature(a.x).parameters has only one parameter, "a". I claiim this is what you want. It's analagous to a functools.partial object. It would be awfully confusing if the signature of a functools.partial object include the parameters handled by the partial object. IMO inspect.getargspec and inspect.getfullargspec get this wrong; for a.x they include the "self" parameter. If you were constructing a call to this function dynamically you'd include one too many parameters. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Wed Feb 25 17:02:51 2015 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Wed, 25 Feb 2015 08:02:51 -0800 (PST) Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: Message-ID: <54edf22b.485c8c0a.558c.38b2@mx.google.com> On 24 February 2015 at 18:58, Guido van Rossum wrote: > The naming of the functions feels inconsistent -- maybe pack(directory, > target) -> create_archive(directory, archive), and set_interpreter() -> > copy_archive(archive, new_archive)? Paul Moore wrote: > One possible source of confusion with copy_archive (and its command > line equivalent "python -m zipapp old.pyz -o new.pyz") is that it > isn't technically a copy, as it changes the shebang line (if you omit > the interpreter argument it removes the existing shebang). Is the difference between create and copy important? e.g., is there anything wrong with create_archive(old_archive, output=new_archive) working as well as create_archive(directory, archive)? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From p.f.moore at gmail.com Wed Feb 25 18:06:07 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 Feb 2015 17:06:07 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54edf22b.485c8c0a.558c.38b2@mx.google.com> References: <54edf22b.485c8c0a.558c.38b2@mx.google.com> Message-ID: On 25 February 2015 at 16:02, Jim J. Jewett wrote: > On 24 February 2015 at 18:58, Guido van Rossum wrote: >> The naming of the functions feels inconsistent -- maybe pack(directory, >> target) -> create_archive(directory, archive), and set_interpreter() -> >> copy_archive(archive, new_archive)? > > > Paul Moore wrote: >> One possible source of confusion with copy_archive (and its command >> line equivalent "python -m zipapp old.pyz -o new.pyz") is that it >> isn't technically a copy, as it changes the shebang line (if you omit >> the interpreter argument it removes the existing shebang). > > Is the difference between create and copy important? e.g., is there > anything wrong with > > create_archive(old_archive, output=new_archive) working as well as > create_archive(directory, archive)? Probably not, now. The semantics have converged enough that this might be reasonable. It's how the command line interface works, after all. It would mean that the behaviour would be different depending on the value of the source argument (supplying the main argument and omitting the target are only valid for create), but again that's how the command line works. I'll have a go at implementing this change this evening and see how it plays out. Paul From rymg19 at gmail.com Wed Feb 25 19:17:52 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 25 Feb 2015 12:17:52 -0600 Subject: [Python-Dev] Working on issue 23496: should I use a macro test or an edit to configure.ac? Message-ID: So... There was a recent discussion here on porting Python to Android. Well, for those of you who saw too many unread messages and marked the whole thread as read like I usually do, I helped Cyd figure out some patches that make it work. Cyd then opened Issue 23496 . Now, I'm going to try to redo the patches against HEAD (or tip in Mercurial language). Which leads me to the question. See, of course, the patches should only be enabled if Python is being built targeting Android, but I'm not sure how that should be detected. I know that the Android target triple is arm-linux-androideabi. Should I test for the __ANDROID__ macro in the source file, or should I modify configure.ac to detect Android and define its own macro? Also, if a configure.ac edit is preferred, where should it be? It's slightly painful to scan through all 4000 lines of Python's configure script. ;) -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Feb 25 20:33:19 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 Feb 2015 19:33:19 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54edf22b.485c8c0a.558c.38b2@mx.google.com> Message-ID: On 25 February 2015 at 17:06, Paul Moore wrote: >> Is the difference between create and copy important? e.g., is there >> anything wrong with >> >> create_archive(old_archive, output=new_archive) working as well as >> create_archive(directory, archive)? > > Probably not, now. The semantics have converged enough that this might > be reasonable. It's how the command line interface works, after all. > It would mean that the behaviour would be different depending on the > value of the source argument (supplying the main argument and omitting > the target are only valid for create), but again that's how the > command line works. > > I'll have a go at implementing this change this evening and see how it > plays out. That worked out pretty well, IMO. The resulting API is a lot cleaner (internally, there's not much change, I still have a copy_archive function but it's now private). I've included the resulting API documentation below. It looks pretty good to me. Does anyone have any further suggestions or comments, or does this look ready to go back to Guido for a second review? Paul Python API ---------- The module defines two convenience functions: .. function:: create_archive(directory, target=None, interpreter=None, main=None) Create an application archive from *source*. The source can be any of the following: * The name of a directory, in which case a new application archive will be created from the content of that directory. * The name of an existing application archive file, in which case the file is copied to the target. The file name should include the ``.pyz`` extension, if required. * A file object open for reading in bytes mode. The content of the file should be an application archive, and the file object is assumed to be positioned at the start of the archive. The *target* argument determines where the resulting archive will be written: * If it is the name of a file, the archive will be written to that file. * If it is an open file object, the archive will be written to that file object, which must be open for writing in bytes mode. * If the target is omitted (or None), the source must be a directory and the target will be a file with the same name as the source, with a ``.pyz`` extension added. The *interpreter* argument specifies the name of the Python interpreter with which the archive will be executed. It is written as a "shebang" line at the start of the archive. On POSIX, this will be interpreted by the OS, and on Windows it will be handled by the Python launcher. Omitting the *interpreter* results in no shebang line being written. If an interpreter is specified, and the target is a filename, the executable bit of the target file will be set. The *main* argument specifies the name of a callable which will be used as the main program for the archive. It can only be specified if the source is a directory, and the source does not already contain a ``__main__.py`` file. The *main* argument should take the form "pkg.module:callable" and the archive will be run by importing "pkg.module" and executing the given callable with no arguments. It is an error to omit *main* if the source is a directory and does not contain a ``__main__.py`` file, as otherwise the resulting archive would not be executable. If a file object is specified for *source* or *target*, it is the caller's responsibility to close it after calling create_archive. When copying an existing archive, file objects supplied only need ``read`` and ``readline``, or ``write`` methods. When creating an archive from a directory, if the target is a file object it will be passed to the ``zipfile.ZipFile`` class, and must supply the methods needed by that class. .. function:: get_interpreter(archive) Return the interpreter specified in the ``#!`` line at the start of the archive. If there is no ``#!`` line, return :const:`None`. The *archive* argument can be a filename or a file-like object open for reading in bytes mode. It is assumed to be at the start of the archive. From brett at python.org Wed Feb 25 20:40:03 2015 From: brett at python.org (Brett Cannon) Date: Wed, 25 Feb 2015 19:40:03 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support References: <54edf22b.485c8c0a.558c.38b2@mx.google.com> Message-ID: On Wed, Feb 25, 2015 at 2:33 PM Paul Moore wrote: > On 25 February 2015 at 17:06, Paul Moore wrote: > >> Is the difference between create and copy important? e.g., is there > >> anything wrong with > >> > >> create_archive(old_archive, output=new_archive) working as well as > >> create_archive(directory, archive)? > > > > Probably not, now. The semantics have converged enough that this might > > be reasonable. It's how the command line interface works, after all. > > It would mean that the behaviour would be different depending on the > > value of the source argument (supplying the main argument and omitting > > the target are only valid for create), but again that's how the > > command line works. > > > > I'll have a go at implementing this change this evening and see how it > > plays out. > > That worked out pretty well, IMO. The resulting API is a lot cleaner > (internally, there's not much change, I still have a copy_archive > function but it's now private). I've included the resulting API > documentation below. It looks pretty good to me. > > Does anyone have any further suggestions or comments, or does this > look ready to go back to Guido for a second review? > +1 from me. -Brett > > Paul > > Python API > ---------- > > The module defines two convenience functions: > > > .. function:: create_archive(directory, target=None, interpreter=None, > main=None) > > Create an application archive from *source*. The source can be any > of the following: > > * The name of a directory, in which case a new application archive > will be created from the content of that directory. > * The name of an existing application archive file, in which case the > file is copied to the target. The file name should include the > ``.pyz`` extension, if required. > * A file object open for reading in bytes mode. The content of the > file should be an application archive, and the file object is > assumed to be positioned at the start of the archive. > > The *target* argument determines where the resulting archive will be > written: > > * If it is the name of a file, the archive will be written to that > file. > * If it is an open file object, the archive will be written to that > file object, which must be open for writing in bytes mode. > * If the target is omitted (or None), the source must be a directory > and the target will be a file with the same name as the source, with > a ``.pyz`` extension added. > > The *interpreter* argument specifies the name of the Python > interpreter with which the archive will be executed. It is written as > a "shebang" line at the start of the archive. On POSIX, this will be > interpreted by the OS, and on Windows it will be handled by the Python > launcher. Omitting the *interpreter* results in no shebang line being > written. If an interpreter is specified, and the target is a > filename, the executable bit of the target file will be set. > > The *main* argument specifies the name of a callable which will be > used as the main program for the archive. It can only be specified if > the source is a directory, and the source does not already contain a > ``__main__.py`` file. The *main* argument should take the form > "pkg.module:callable" and the archive will be run by importing > "pkg.module" and executing the given callable with no arguments. It > is an error to omit *main* if the source is a directory and does not > contain a ``__main__.py`` file, as otherwise the resulting archive > would not be executable. > > If a file object is specified for *source* or *target*, it is the > caller's responsibility to close it after calling create_archive. > > When copying an existing archive, file objects supplied only need > ``read`` and ``readline``, or ``write`` methods. When creating an > archive from a directory, if the target is a file object it will be > passed to the ``zipfile.ZipFile`` class, and must supply the methods > needed by that class. > > .. function:: get_interpreter(archive) > > Return the interpreter specified in the ``#!`` line at the start of the > archive. If there is no ``#!`` line, return :const:`None`. > The *archive* argument can be a filename or a file-like object open > for reading in bytes mode. It is assumed to be at the start of the > archive. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Feb 25 20:42:39 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 25 Feb 2015 11:42:39 -0800 Subject: [Python-Dev] PEP 448 review Message-ID: I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch , taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Feb 25 21:11:49 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Feb 2015 15:11:49 -0500 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54edf22b.485c8c0a.558c.38b2@mx.google.com> Message-ID: <20150225151149.6f828946@limelight.wooz.org> On Feb 25, 2015, at 07:33 PM, Paul Moore wrote: >The module defines two convenience functions: > > >.. function:: create_archive(directory, target=None, interpreter=None, >main=None) > > Create an application archive from *source*. The source can be any > of the following: I think you meant s/directory/source/ in the signature. Other than that, +1. Cheers, -Barry From jimjjewett at gmail.com Wed Feb 25 21:12:59 2015 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Wed, 25 Feb 2015 15:12:59 -0500 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54edf22b.485c8c0a.558c.38b2@mx.google.com> Message-ID: On Wed, Feb 25, 2015 at 2:33 PM, Paul Moore wrote: > On 25 February 2015 at 17:06, Paul Moore wrote: > I've included the resulting API > documentation below. It looks pretty good to me. Me too. I have a few nits anyhow. > .. function:: create_archive(directory, target=None, interpreter=None, > main=None) > Create an application archive from *source*. The source can be any > of the following: (1) *source* makes me think of source code, as opposed to binary. This is only a small objection, in part because I can't think of anything better. (2) If you do keep *source*, I think that the the "directory" parameter should be renamed to "source". (3) > * The name of an existing application archive file, in which case the > file is copied to the target. ==> > * The name of an existing application archive file, in which case the > file is copied (possibly with changes) to the target. My concern is that someone who does want just another copy will use this, see "copied", not read the other options, and be surprised when the shebang is dropped. > * A file object open for reading in bytes mode. The content of the > file should be an application archive, and the file object is > assumed to be positioned at the start of the archive. I like this way of ducking the "does it need to be seekable" question. > The *target* argument determines where the resulting archive will be > written: > > * If it is the name of a file, the archive will be written to that > file. (4) Note that the filename is not required to end with pyz, although that is good practice. Or maybe just be explicit that the function itself does not add a .pyz, and assumes that the caller will do so when appropriate. > The *interpreter* argument specifies the name of the Python > interpreter with which the archive will be executed. ... > ... Omitting the *interpreter* results in no shebang line being > written. (5) even if there was an explicit shebang line in the source archive. > If an interpreter is specified, and the target is a > filename, the executable bit of the target file will be set. (6) (target is a filename, or None) Or does that clarification just confuse the issue, and only benefit people so careful they'll verify it themselves anyway? (7) That is a good idea, but not quite as clear cut as it sounds. On unix, there are generally 3 different executable bits specifying *who* can run it. Setting the executable bit only for the owner is probably a conservative but sensible default. -jJ From p.f.moore at gmail.com Wed Feb 25 22:45:29 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 Feb 2015 21:45:29 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54edf22b.485c8c0a.558c.38b2@mx.google.com> Message-ID: On 25 February 2015 at 20:12, Jim J. Jewett wrote: > On Wed, Feb 25, 2015 at 2:33 PM, Paul Moore wrote: >> On 25 February 2015 at 17:06, Paul Moore wrote: > >> I've included the resulting API >> documentation below. It looks pretty good to me. > > Me too. I have a few nits anyhow. > >> .. function:: create_archive(directory, target=None, interpreter=None, >> main=None) > >> Create an application archive from *source*. The source can be any >> of the following: > > (1) *source* makes me think of source code, as opposed to binary. > This is only a small objection, in part because I can't think of > anything better. > > (2) If you do keep *source*, I think that the the "directory" > parameter should be renamed to "source". Yep, that's a typo. Think of it as source -> target as opposed to source code and it's fine :-) > (3) >> * The name of an existing application archive file, in which case the >> file is copied to the target. > > ==> > >> * The name of an existing application archive file, in which case the >> file is copied (possibly with changes) to the target. > > My concern is that someone who does want just another copy will use > this, see "copied", not read the other options, and be surprised when > the shebang is dropped. Hmm, how about "... the content of the archive is copied to the target with a replacement shebang line"? > >> * A file object open for reading in bytes mode. The content of the >> file should be an application archive, and the file object is >> assumed to be positioned at the start of the archive. > > I like this way of ducking the "does it need to be seekable" question. :-) >> The *target* argument determines where the resulting archive will be >> written: >> >> * If it is the name of a file, the archive will be written to that >> file. > > (4) Note that the filename is not required to end with pyz, although > that is good practice. Or maybe just be explicit that the function > itself does not add a .pyz, and assumes that the caller will do so > when appropriate. Hmm, I thought I'd added an explanation. Maybe I did that somewhere else and missed it here. I'll clarify. >> The *interpreter* argument specifies the name of the Python >> interpreter with which the archive will be executed. ... >> ... Omitting the *interpreter* results in no shebang line being >> written. > > (5) even if there was an explicit shebang line in the source archive. I'll clarify the wording. >> If an interpreter is specified, and the target is a >> filename, the executable bit of the target file will be set. > > (6) (target is a filename, or None) Or does that clarification just > confuse the issue, and only benefit people so careful they'll verify > it themselves anyway? Probably :-) How about "if the target is a real file" or "unless the target is a file-like object"? But in all honesty I think it's fine as is. > (7) That is a good idea, but not quite as clear cut as it sounds. On > unix, there are generally 3 different executable bits specifying *who* > can run it. Setting the executable bit only for the owner is probably > a conservative but sensible default. I know, but excuse the naivete of a Windows user. I'm inclined to leave it as it is and direct people to read the source if they care that much (I actually used I_EXEC, which is what I've seen other code use). The alternative is to not set the executable bit at all and make the user do it as a separate step. My instinct is that doing that would be less user friendly, but my instincts on what's good Unix behaviour aren't strong... Thanks for the comments. Paul From db3l.net at gmail.com Thu Feb 26 04:38:56 2015 From: db3l.net at gmail.com (David Bolen) Date: Wed, 25 Feb 2015 22:38:56 -0500 Subject: [Python-Dev] ubuntu buildbot In-Reply-To: <1416845241.1998368.194775069.7DEEE5BE@webmail.messagingengine.com> References: <1416799084.1792952.194543661.2FC7FCAA@webmail.messagingengine.com> <1416845241.1998368.194775069.7DEEE5BE@webmail.messagingengine.com> Message-ID: On Mon, Nov 24, 2014 at 11:07 AM, Benjamin Peterson wrote: > > On Mon, Nov 24, 2014, at 00:33, David Bolen wrote: >> Yeah, it definitely needs it. Historically it was intentional as my own >> servers were all on 8.04, but the last of those moved 12.04 last year. >> >> I think there's already a 12.04 buildbot, so perhaps 14.04 would be >> better? I do prefer sticking with an LTS. > > 14.04 would be good. > >> It'll need to move to 64-bit given the hosting environment, at least for >> the kernel. I could do 32-bit userspace via multiarch if keeping a 32-bit >> buildbot in the mix is attractive. > > It'd be nice to keep a 32-bit bot around. Took a bit longer than anticipated, but the slave upgrade is complete. The bolen-ubuntu slave is now a 32-bit Ubuntu 14.04.2 LTS system. I've re-run the most recent 2.7, 3.4 and 3.x builds which all pass (though with a few new compiler warnings in some cases). -- David From p.f.moore at gmail.com Thu Feb 26 18:05:24 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 Feb 2015 17:05:24 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 24 February 2015 at 18:24, Guido van Rossum wrote: > Here's my review. I really like where this is going but I have a few > questions and suggestions (I can't help myself :-). OK, I've updated both the PEP and the patch based on follow-up discussions. I think (again!) it is ready to go. I've included the updated PEP inline below, it'll be available at peps.python.org as soon as someone has a chance to upload it. Thanks to everyone for the various comments. If I've missed anything that someone thinks I'd said I would change, please let me know. Paul PEP: 441 Title: Improving Python ZIP Application Support Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth , Paul Moore Discussions-To: https://mail.python.org/pipermail/python-dev/2015-February/138277.html Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 March 2013 Post-History: 30 March 2013, 1 April 2013, 16 February 2015 Improving Python ZIP Application Support ======================================== Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6 [1]_. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the ``__main__`` module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules. This feature is not as popular as it should be mainly because it was not promoted as part of Python 2.6 [2]_, so that it is relatively unknown, but also because the Windows installer does not register a file extension (other than ``.py``) for this format of file, to associate with the launcher. This PEP proposes to fix these problems by re-publicising the feature, defining the ``.pyz`` and ``.pyzw`` extensions as "Python ZIP Applications" and "Windowed Python ZIP Applications", and providing some simple tooling to manage the format. A New Python ZIP Application Extension ====================================== The terminology "Python Zip Application" will be the formal term used for a zip-format archive that contains Python code in a form that can be directly executed by Python (specifically, it must have a ``__main__.py`` file in the root directory of the archive). The extension ``.pyz`` will be formally associated with such files. The Python 3.5 installer will associate ``.pyz`` and ``.pyzw`` "Python Zip Applications" with the platform launcher so they can be executed. A ``.pyz`` archive is a console application and a ``.pyzw`` archive is a windowed application, indicating whether the console should appear when running the app. On Unix, it would be ideal if the ``.pyz`` extension and the name "Python Zip Application" were registered (in the mime types database?). However, such an association is out of scope for this PEP. Python Zip applications can be prefixed with a ``#!`` line pointing to the correct Python interpreter and an optional explanation:: #!/usr/bin/env python3 # Python application packed with zipapp module (binary contents of archive) On Unix, this allows the OS to run the file with the correct interpreter, via the standard "shebang" support. On Windows, the Python launcher implements shebang support. However, it is always possible to execute a ``.pyz`` application by supplying the filename to the Python interpreter directly. As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work. Minimal Tooling: The zipapp Module ================================== This PEP also proposes including a module for working with these archives. The module will contain functions for working with Python zip application archives, and a command line interface (via ``python -m zipapp``) for their creation and manipulation. More complete tools for managing Python Zip Applications are encouraged as 3rd party applications on PyPI. Currently, pyzzer [5]_ and pex [6]_ are two such tools. Module Interface ---------------- The zipapp module will provide the following functions: ``create_archive(source, target=None, interpreter=None, main=None)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Create an application archive from *source*. The source can be any of the following: * The name of a directory, in which case a new application archive will be created from the content of that directory. * The name of an existing application archive file, in which case the file is copied to the target. The file name should include the ``.pyz`` extension, if required. * A file object open for reading in bytes mode. The content of the file should be an application archive, and the file object is assumed to be positioned at the start of the archive. The *target* argument determines where the resulting archive will be written: * If it is the name of a file, the archive will be written to that file. * If it is an open file object, the archive will be written to that file object, which must be open for writing in bytes mode. * If the target is omitted (or None), the source must be a directory and the target will be a file with the same name as the source, with a ``.pyz`` extension added. The *interpreter* argument specifies the name of the Python interpreter with which the archive will be executed. It is written as a "shebang" line at the start of the archive. On Unix, this will be interpreted by the OS, and on Windows it will be handled by the Python launcher. Omitting the *interpreter* results in no shebang line being written. If an interpreter is specified, and the target is a filename, the executable bit of the target file will be set. The *main* argument specifies the name of a callable which will be used as the main program for the archive. It can only be specified if the source is a directory, and the source does not already contain a ``__main__.py`` file. The *main* argument should take the form "pkg.module:callable" and the archive will be run by importing "pkg.module" and executing the given callable with no arguments. It is an error to omit *main* if the source is a directory and does not contain a ``__main__.py`` file, as otherwise the resulting archive would not be executable. If a file object is specified for *source* or *target*, it is the caller's responsibility to close it after calling create_archive. When copying an existing archive, file objects supplied only need ``read`` and ``readline``, or ``write`` methods. When creating an archive from a directory, if the target is a file object it will be passed to the ``zipfile.ZipFile`` class, and must supply the methods needed by that class. ``get_interpreter(archive)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Returns the interpreter specified in the shebang line of the *archive*. If there is no shebang, the function returns ``None``. The *archive* argument can be a filename or a file-like object open for reading in bytes mode. Command Line Usage ------------------ The zipapp module can be run with the python ``-m`` flag. The command line interface is as follows:: python -m zipapp directory [options] Create an archive from the given directory. An archive will be created from the contents of that directory. The archive will have the same name as the source directory with a .pyz extension. The following options can be specified: -o archive / --output archive The destination archive will have the specified name. The given name will be used as written, so should include the ".pyz" extension. -p interpreter / --python interpreter The given interpreter will be written to the shebang line of the archive. If this option is not given, the archive will have no shebang line. -m pkg.mod:fn / --main pkg.mod:fn The source directory must not have a __main__.py file. The archiver will write a __main__.py file into the target which calls fn from the module pkg.mod. The behaviour of the command line interface matches that of ``zipapp.create_archive()``. In addition, it is possible to use the command line interface to work with an existing archive:: python -m zipapp app.pyz --show Displays the shebang line of an archive. Output is of the form Interpreter: /usr/bin/env or Interpreter: and is intended for diagnostic use, not for scripts. python -m zipapp app.pyz -o newapp.pyz [-p interpreter] Copy app.pyz to newapp.pyz, modifying the shebang line based on the -p option (as for creating an archive, no -p option means remove the shebang line). Specifying a destination is mandatory. In-place modification of an archive is *not* supported, as the risk of damaging archives is too great for a simple tool. As noted, the archives are standard zip files, and so can be unpacked using any standard ZIP utility or Python's zipfile module. For this reason, no interfaces to list the contents of an archive, or unpack them, are provided or needed. FAQ --- Are you sure a standard ZIP utility can handle ``#!`` at the beginning? Absolutely. The zipfile specification allows for arbitrary data to be prepended to a zipfile. This feature is commonly used by "self-extracting zip" programs. If your archive program can't handle this, it is a bug in your archive program. Isn't zipapp just a very thin wrapper over the zipfile module? Yes. If you prefer to build your own Python zip application archives using other tools, they will work just as well. The zipapp module is a convenience, nothing more. Why not use just use a ``.zip`` or ``.py`` extension? Users expect a ``.zip`` file to be opened with an archive tool, and expect a ``.py`` file to contain readable text. Both would be confusing for this use case. How does this compete with existing package formats? The sdist, bdist and wheel formats are designed for packaging of modules to be installed into an existing Python installation. They are not intended to be used without installing. The executable zip format is specifically designed for standalone use, without needing to be installed. They are in effect a multi-file version of a standalone Python script. Rejected Proposals ================== Convenience Values for Shebang Lines ------------------------------------ Is it worth having "convenience" forms for any of the common interpreter values? For example, ``-p 3`` meaning the same as ``-p "/usr/bin/env python3"``. It would save a lot of typing for the common cases, as well as giving cross-platform options for people who don't want or need to understand the intricacies of shebang handling on "other" platforms. Downsides are that it's not obvious how to translate the abbreviations. For example, should "3" mean "/usr/bin/env python3", "/usr/bin/python3", "python3", or something else? Also, there is no obvious short form for the key case of "/usr/bin/env python" (any available version of Python), which could easily result in scripts being written with overly-restrictive shebang lines. Overall, this seems like there are more problems than benefits, and as a result has been dropped from consideration. Registering ``.pyz`` as a Media Type ------------------------------------ It was suggested [3]_ that the ``.pyz`` extension should be registered in the Unix database of extensions. While it makes sense to do this as an equivalent of the Windows installer registering the extension, the ``.py`` extension is not listed in the media types database [4]_. It doesn't seem reasonable to register ``.pyz`` without ``.py``, so this idea has been omitted from this PEP. An interested party could arrange for *both* ``.py`` and ``.pyz`` to be registered at a future date. Default Interpreter ------------------- The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter. Unix users have problems with this behaviour, as the default for the python command on many distributions is Python 2, and it is felt that this PEP should prefer Python 3 by default. However, using a command of ``python3`` can result in unexpected behaviour for Windows users, where the default behaviour of the launcher for the command ``python`` is commonly customised by users, but the behaviour of ``python3`` may not be modified to match. As a result, the principle "in the face of ambiguity, refuse to guess" has been invoked, and archives have no shebang line unless explicitly requested. On Windows, the archives will still be run (with the default Python) by the launcher, and on Unix, the archives can be run by explicitly invoking the desired Python interpreter. Command Line Tool to Manage Shebang Lines ----------------------------------------- It is conceivable that users would want to modify the shebang line for an existing archive, or even just display the current shebang line. This is tricky to do so with existing tools (zip programs typically ignore prepended data totally, and text editors can have trouble editing files containing binary data). The zipapp module provides functions to handle the shebang line, but does not include a command line interface to that functionality. This is because it is not clear how to provide one without the resulting interface being over-complex and potentially confusing. Changing the shebang line is expected to be an uncommon requirement. Reference Implementation ======================== A reference implementation is at http://bugs.python.org/issue23491. References ========== .. [1] Allow interpreter to execute a zip file (http://bugs.python.org/issue1739468) .. [2] Feature is not documented (http://bugs.python.org/issue17359) .. [3] Discussion of adding a .pyz mime type on python-dev (https://mail.python.org/pipermail/python-dev/2015-February/138338.html) .. [4] Register of media types (http://www.iana.org/assignments/media-types/media-types.xhtml) .. [5] pyzzer - A tool for creating Python-executable archives (https://pypi.python.org/pypi/pyzzer) .. [6] pex - The PEX packaging toolchain (https://pypi.python.org/pypi/pex) The discussion of this PEP took place on the python-dev mailing list, in the thread starting at https://mail.python.org/pipermail/python-dev/2015-February/138277.html Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From v+python at g.nevcal.com Thu Feb 26 18:28:52 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 26 Feb 2015 09:28:52 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: <54EF57D4.4020806@g.nevcal.com> On 2/26/2015 9:05 AM, Paul Moore wrote: > On 24 February 2015 at 18:24, Guido van Rossum wrote: >> Here's my review. I really like where this is going but I have a few >> questions and suggestions (I can't help myself :-). > OK, I've updated both the PEP and the patch based on follow-up > discussions. I think (again!) it is ready to go. > > I've included the updated PEP inline below, it'll be available at > peps.python.org as soon as someone has a chance to upload it. > > Thanks to everyone for the various comments. If I've missed anything > that someone thinks I'd said I would change, please let me know. Three very, very, very minor nits below. > > Paul > > PEP: 441 > Title: Improving Python ZIP Application Support > Version: $Revision$ > Last-Modified: $Date$ > Author: Daniel Holth , > Paul Moore > Discussions-To: > https://mail.python.org/pipermail/python-dev/2015-February/138277.html > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 30 March 2013 > Post-History: 30 March 2013, 1 April 2013, 16 February 2015 > > Improving Python ZIP Application Support > ======================================== > > Python has had the ability to execute directories or ZIP-format > archives as scripts since version 2.6 [1]_. When invoked with a zip > file or directory as its first argument the interpreter adds that > directory to sys.path and executes the ``__main__`` module. These > archives provide a great way to publish software that needs to be > distributed as a single file script but is complex enough to need to > be written as a collection of modules. > > This feature is not as popular as it should be mainly because it was > not promoted as part of Python 2.6 [2]_, so that it is relatively > unknown, but also because the Windows installer does not register a > file extension (other than ``.py``) for this format of file, to associate > with the launcher. > > This PEP proposes to fix these problems by re-publicising the feature, > defining the ``.pyz`` and ``.pyzw`` extensions as "Python ZIP Applications" > and "Windowed Python ZIP Applications", and providing some simple > tooling to manage the format. > > A New Python ZIP Application Extension > ====================================== > > The terminology "Python Zip Application" will be the formal term used > for a zip-format archive that contains Python code in a form that can > be directly executed by Python (specifically, it must have a > ``__main__.py`` file in the root directory of the archive). The > extension ``.pyz`` will be formally associated with such files. > > The Python 3.5 installer will associate ``.pyz`` and ``.pyzw`` "Python > Zip Applications" with the platform launcher so they can be executed. > A ``.pyz`` archive is a console application and a ``.pyzw`` archive is a > windowed application, indicating whether the console should appear > when running the app. > > On Unix, it would be ideal if the ``.pyz`` extension and the name > "Python Zip Application" were registered (in the mime types database?). > However, such an association is out of scope for this PEP. > > Python Zip applications can be prefixed with a ``#!`` line > pointing to the correct Python interpreter and an optional > explanation:: > > #!/usr/bin/env python3 > # Python application packed with zipapp module > (binary contents of archive) > > On Unix, this allows the OS to run the file with the correct > interpreter, via the standard "shebang" support. On Windows, the > Python launcher implements shebang support. > > However, it is always possible to execute a ``.pyz`` application by > supplying the filename to the Python interpreter directly. > > As background, ZIP archives are defined with a footer containing > relative offsets from the end of the file. They remain valid when > concatenated to the end of any other file. This feature is completely > standard and is how self-extracting ZIP archives and the bdist_wininst > installer format work. > > > Minimal Tooling: The zipapp Module > ================================== > > This PEP also proposes including a module for working with these > archives. The module will contain functions for working with Python > zip application archives, and a command line interface (via ``python > -m zipapp``) for their creation and manipulation. > > More complete tools for managing Python Zip Applications are > encouraged as 3rd party applications on PyPI. Currently, pyzzer [5]_ > and pex [6]_ are two such tools. > > Module Interface > ---------------- > > The zipapp module will provide the following functions: > > ``create_archive(source, target=None, interpreter=None, main=None)`` > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > Create an application archive from *source*. The source can be any > of the following: > > * The name of a directory, in which case a new application archive > will be created from the content of that directory. > * The name of an existing application archive file, in which case the > file is copied to the target. The file name should include the > ``.pyz`` extension, if required. Or ".pyzw", I presume. > * A file object open for reading in bytes mode. The content of the > file should be an application archive, and the file object is > assumed to be positioned at the start of the archive. > > The *target* argument determines where the resulting archive will be > written: > > * If it is the name of a file, the archive will be written to that > file. > * If it is an open file object, the archive will be written to that > file object, which must be open for writing in bytes mode. > * If the target is omitted (or None), the source must be a directory > and the target will be a file with the same name as the source, with > a ``.pyz`` extension added. Hmm. So one _must_ specify a target if one wants a .pyzw. > The *interpreter* argument specifies the name of the Python > interpreter with which the archive will be executed. It is written as > a "shebang" line at the start of the archive. On Unix, this will be > interpreted by the OS, and on Windows it will be handled by the Python > launcher. Omitting the *interpreter* results in no shebang line being > written. If an interpreter is specified, and the target is a > filename, the executable bit of the target file will be set. > > The *main* argument specifies the name of a callable which will be > used as the main program for the archive. It can only be specified if > the source is a directory, and the source does not already contain a > ``__main__.py`` file. The *main* argument should take the form > "pkg.module:callable" and the archive will be run by importing > "pkg.module" and executing the given callable with no arguments. It > is an error to omit *main* if the source is a directory and does not > contain a ``__main__.py`` file, as otherwise the resulting archive > would not be executable. > > If a file object is specified for *source* or *target*, it is the > caller's responsibility to close it after calling create_archive. > > When copying an existing archive, file objects supplied only need > ``read`` and ``readline``, or ``write`` methods. When creating an > archive from a directory, if the target is a file object it will be > passed to the ``zipfile.ZipFile`` class, and must supply the methods > needed by that class. > > ``get_interpreter(archive)`` > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > Returns the interpreter specified in the shebang line of the > *archive*. If there is no shebang, the function returns ``None``. > The *archive* argument can be a filename or a file-like object open > for reading in bytes mode. > > > Command Line Usage > ------------------ > > The zipapp module can be run with the python ``-m`` flag. The command > line interface is as follows:: > > python -m zipapp directory [options] > > Create an archive from the given directory. An archive will > be created from the contents of that directory. The archive > will have the same name as the source directory with a .pyz > extension. > > The following options can be specified: > > -o archive / --output archive > > The destination archive will have the specified name. The > given name will be used as written, so should include the > ".pyz" extension. Or ".pyzw", I presume. You used ``funny quotes`` around extensions earlier, but not here. > -p interpreter / --python interpreter > > The given interpreter will be written to the shebang line > of the archive. If this option is not given, the archive > will have no shebang line. > > -m pkg.mod:fn / --main pkg.mod:fn > > The source directory must not have a __main__.py file. The > archiver will write a __main__.py file into the target > which calls fn from the module pkg.mod. > > The behaviour of the command line interface matches that of > ``zipapp.create_archive()``. > > In addition, it is possible to use the command line interface to work > with an existing archive:: > > python -m zipapp app.pyz --show > > Displays the shebang line of an archive. Output is of the > form > > Interpreter: /usr/bin/env > or > Interpreter: > > and is intended for diagnostic use, not for scripts. > > python -m zipapp app.pyz -o newapp.pyz [-p interpreter] > > Copy app.pyz to newapp.pyz, modifying the shebang line based > on the -p option (as for creating an archive, no -p option > means remove the shebang line). Specifying a destination is > mandatory. > > In-place modification of an archive is *not* supported, as the > risk of damaging archives is too great for a simple tool. > > As noted, the archives are standard zip files, and so can be unpacked > using any standard ZIP utility or Python's zipfile module. For this > reason, no interfaces to list the contents of an archive, or unpack > them, are provided or needed. > > FAQ > --- > > Are you sure a standard ZIP utility can handle ``#!`` at the beginning? > Absolutely. The zipfile specification allows for arbitrary data to > be prepended to a zipfile. This feature is commonly used by > "self-extracting zip" programs. If your archive program can't > handle this, it is a bug in your archive program. > > Isn't zipapp just a very thin wrapper over the zipfile module? > Yes. If you prefer to build your own Python zip application > archives using other tools, they will work just as well. The > zipapp module is a convenience, nothing more. > > Why not use just use a ``.zip`` or ``.py`` extension? > Users expect a ``.zip`` file to be opened with an archive tool, and > expect a ``.py`` file to contain readable text. Both would be > confusing for this use case. > > How does this compete with existing package formats? > The sdist, bdist and wheel formats are designed for packaging of > modules to be installed into an existing Python installation. > They are not intended to be used without installing. The > executable zip format is specifically designed for standalone use, > without needing to be installed. They are in effect a multi-file > version of a standalone Python script. > > Rejected Proposals > ================== > > Convenience Values for Shebang Lines > ------------------------------------ > > Is it worth having "convenience" forms for any of the common > interpreter values? For example, ``-p 3`` meaning the same as ``-p > "/usr/bin/env python3"``. It would save a lot of typing for the > common cases, as well as giving cross-platform options for people who > don't want or need to understand the intricacies of shebang handling > on "other" platforms. > > Downsides are that it's not obvious how to translate the > abbreviations. For example, should "3" mean "/usr/bin/env python3", > "/usr/bin/python3", "python3", or something else? Also, there is no > obvious short form for the key case of "/usr/bin/env python" (any > available version of Python), which could easily result in scripts > being written with overly-restrictive shebang lines. > > Overall, this seems like there are more problems than benefits, and as > a result has been dropped from consideration. > > Registering ``.pyz`` as a Media Type > ------------------------------------ > > It was suggested [3]_ that the ``.pyz`` extension should be registered > in the Unix database of extensions. While it makes sense to do this > as an equivalent of the Windows installer registering the extension, > the ``.py`` extension is not listed in the media types database [4]_. > It doesn't seem reasonable to register ``.pyz`` without ``.py``, so > this idea has been omitted from this PEP. An interested party could > arrange for *both* ``.py`` and ``.pyz`` to be registered at a future > date. > > Default Interpreter > ------------------- > > The initial draft of this PEP proposed using ``/usr/bin/env python`` > as the default interpreter. Unix users have problems with this > behaviour, as the default for the python command on many distributions > is Python 2, and it is felt that this PEP should prefer Python 3 by > default. However, using a command of ``python3`` can result in > unexpected behaviour for Windows users, where the default behaviour of > the launcher for the command ``python`` is commonly customised by users, > but the behaviour of ``python3`` may not be modified to match. > > As a result, the principle "in the face of ambiguity, refuse to guess" > has been invoked, and archives have no shebang line unless explicitly > requested. On Windows, the archives will still be run (with the > default Python) by the launcher, and on Unix, the archives can be run > by explicitly invoking the desired Python interpreter. > > Command Line Tool to Manage Shebang Lines > ----------------------------------------- > > It is conceivable that users would want to modify the shebang line for > an existing archive, or even just display the current shebang line. > This is tricky to do so with existing tools (zip programs typically > ignore prepended data totally, and text editors can have trouble > editing files containing binary data). > > The zipapp module provides functions to handle the shebang line, but > does not include a command line interface to that functionality. This > is because it is not clear how to provide one without the resulting > interface being over-complex and potentially confusing. Changing the > shebang line is expected to be an uncommon requirement. > > Reference Implementation > ======================== > > A reference implementation is at http://bugs.python.org/issue23491. > > References > ========== > > .. [1] Allow interpreter to execute a zip file > (http://bugs.python.org/issue1739468) > > .. [2] Feature is not documented > (http://bugs.python.org/issue17359) > > .. [3] Discussion of adding a .pyz mime type on python-dev > (https://mail.python.org/pipermail/python-dev/2015-February/138338.html) > > .. [4] Register of media types > (http://www.iana.org/assignments/media-types/media-types.xhtml) > > .. [5] pyzzer - A tool for creating Python-executable archives > (https://pypi.python.org/pypi/pyzzer) > > .. [6] pex - The PEX packaging toolchain > (https://pypi.python.org/pypi/pex) > > The discussion of this PEP took place on the python-dev mailing list, > in the thread starting at > https://mail.python.org/pipermail/python-dev/2015-February/138277.html > > Copyright > ========= > > This document has been placed into the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Feb 26 19:23:24 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 Feb 2015 10:23:24 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54EF57D4.4020806@g.nevcal.com> References: <54EB7DDE.9030405@stoneleaf.us> <54EF57D4.4020806@g.nevcal.com> Message-ID: <54EF649C.4050204@stoneleaf.us> On 02/26/2015 09:28 AM, Glenn Linderman wrote: > On 2/26/2015 9:05 AM, Paul Moore wrote: >> ``create_archive(source, target=None, interpreter=None, main=None)`` >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> >> Create an application archive from *source*. The source can be any >> of the following: >> >> * The name of a directory, in which case a new application archive >> will be created from the content of that directory. >> * The name of an existing application archive file, in which case the >> file is copied to the target. The file name should include the >> ``.pyz`` extension, if required. > > Or ".pyzw", I presume. Hmm -- can the py launcher handle a `#!pythonw` line to properly launch a .pyz (or .py) file? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Thu Feb 26 20:02:40 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 Feb 2015 19:02:40 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54EF57D4.4020806@g.nevcal.com> References: <54EB7DDE.9030405@stoneleaf.us> <54EF57D4.4020806@g.nevcal.com> Message-ID: On 26 February 2015 at 17:28, Glenn Linderman wrote: > * The name of a directory, in which case a new application archive > will be created from the content of that directory. > * The name of an existing application archive file, in which case the > file is copied to the target. The file name should include the > ``.pyz`` extension, if required. > > > Or ".pyzw", I presume. Yes. > * If it is an open file object, the archive will be written to that > file object, which must be open for writing in bytes mode. > * If the target is omitted (or None), the source must be a directory > and the target will be a file with the same name as the source, with > a ``.pyz`` extension added. > > > Hmm. So one _must_ specify a target if one wants a .pyzw. Correct. Generally I don't think pyzw will be used often, so I don't see that as a major problem. I'll add a clarifying note if the PEP needs another round of edits, but I'm inclined not to bother the PEP editors if this is all that needs changing. > The destination archive will have the specified name. The > given name will be used as written, so should include the > ".pyz" extension. > > > Or ".pyzw", I presume. You used ``funny quotes`` around extensions earlier, > but not here. Again yes. Normal quotes because this chunk is already in a literal block, so ``...`` is redundant. Paul From p.f.moore at gmail.com Thu Feb 26 20:08:59 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 Feb 2015 19:08:59 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: <54EF649C.4050204@stoneleaf.us> References: <54EB7DDE.9030405@stoneleaf.us> <54EF57D4.4020806@g.nevcal.com> <54EF649C.4050204@stoneleaf.us> Message-ID: On 26 February 2015 at 18:23, Ethan Furman wrote: > On 02/26/2015 09:28 AM, Glenn Linderman wrote: >> On 2/26/2015 9:05 AM, Paul Moore wrote: > >>> ``create_archive(source, target=None, interpreter=None, main=None)`` >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> >>> Create an application archive from *source*. The source can be any >>> of the following: >>> >>> * The name of a directory, in which case a new application archive >>> will be created from the content of that directory. >>> * The name of an existing application archive file, in which case the >>> file is copied to the target. The file name should include the >>> ``.pyz`` extension, if required. >> >> Or ".pyzw", I presume. > > Hmm -- can the py launcher handle a `#!pythonw` line to properly launch a .pyz (or .py) file? No. The launcher doesn't handle pythonw, because it won't do what you expect. On Windows, the difference between a GUI and a console program is baked into the executable (the "subsystem" field in the exe file header). That's why we have python/pythonw and py/pyw executables. A .pyz with a #!pythonw shebang would be run by py.exe, which would launch pythonw.exe. So if you double click the file, a console window will open (for py.exe) but the script won't use it (because it's being run by pythonw). So you'll have a useless console window hanging round. Paul From guido at python.org Thu Feb 26 21:19:27 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 26 Feb 2015 12:19:27 -0800 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: As a follow-up, Joshua updated the PEP to remove *comprehensions, and it is now accepted. On Wed, Feb 25, 2015 at 11:42 AM, Guido van Rossum wrote: > I'm back, I've re-read the PEP, and I've re-read the long thread with "(no > subject)". > > I think Georg Brandl nailed it: > > """ > > > > > > > > > *I like the "sequence and dict flattening" part of the PEP, mostly because > itis consistent and should be easy to understand, but the comprehension > syntaxenhancements seem to be bad for readability and "comprehending" what > the codedoes.The call syntax part is a mixed bag on the one hand it is nice > to be consistent with the extended possibilities in literals (flattening), > but on the other hand there would be small but annoying inconsistencies > anyways (e.g. the duplicate kwarg case above).* > """ > > Greg Ewing followed up explaining that the inconsistency between dict > flattening and call syntax is inherent in the pre-existing different rules > for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, > a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, > **{'a': 1}) is a TypeError.) > > For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations > for function calls proposed by the PEP is an easy +1 -- it's a > straightforward extension of the existing pattern, and anybody who knows > what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, > **e) means shouldn't be hard either. Understanding the edge case for > duplicate keys with f(**d, **e) is a little harder, but the error messages > are pretty clear, and it is not a new edge case. > > The sequence and dict flattening syntax proposals are also clean and > logical -- we already have *-unpacking on the receiving side, so allowing > *x in tuple expressions reads pretty naturally (and the similarity with *a > in argument lists certainly helps). From here, having [a, *x, b, *y] is > also natural, and then the extension to other displays is natural: {a, *x, > b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. > > So that leaves comprehensions. IIRC, during the development of the patch > we realized that f(*x for x in xs) is sufficiently ambiguous that we > decided to disallow it -- note that f(x for x in xs) is already somewhat of > a special case because an argument can only be a "bare" generator > expression if it is the only argument. The same reasoning doesn't apply (in > that form) to list, set and dict comprehensions -- while f(x for x in xs) > is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the > same as [(x for x in xs)] (that's a list of one element, and the element is > a generator expression). > > The basic premise of this part of the proposal is that if you have a few > iterables, the new proposal (without comprehensions) lets you create a list > or generator expression that iterates over all of them, essentially > flattening them: > > >>> xs = [1, 2, 3] > >>> ys = ['abc', 'def'] > >>> zs = [99] > >>> [*xs, *ys, *zs] > [1, 2, 3, 'abc', 'def', 99] > >>> > > But now suppose you have a list of iterables: > > >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] > >>> [*xss[0], *xss[1], *xss[2]] > [1, 2, 3, 'abc', 'def', 99] > >>> > > Wouldn't it be nice if you could write the latter using a comprehension? > > >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] > >>> [*xs for xs in xss] > [1, 2, 3, 'abc', 'def', 99] > >>> > > This is somewhat seductive, and the following is even nicer: the *xs > position may be an expression, e.g.: > > >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] > >>> [*xs[:2] for xs in xss] > [1, 2, 'abc', 'def', 99] > >>> > > On the other hand, I had to explore the possibilities here by > experimenting in the interpreter, and I discovered some odd edge cases > (e.g. you can parenthesize the starred expression, but that seems a > syntactic accident). > > All in all I am personally +0 on the comprehension part of the PEP, and I > like that it provides a way to "flatten" a sequence of sequences, but I > think very few people in the thread have supported this part. Therefore I > would like to ask Neil to update the PEP and the patch to take out the > comprehension part, so that the two "easy wins" can make it into Python 3.5 > (basically, I am accepting two-thirds of the PEP :-). There is some time > yet until alpha 2. > > I would also like code reviewers (Benjamin?) to start reviewing the patch > , taking into account that the > comprehension part needs to be removed. > > -- > --Guido van Rossum (python.org/~guido) > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Feb 26 21:34:19 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 26 Feb 2015 12:34:19 -0800 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: One more thing. This PEP would never have been accepted without a working implementation. Thanks Neil and Joshua for that! (And for being flexible.) On Thu, Feb 26, 2015 at 12:19 PM, Guido van Rossum wrote: > As a follow-up, Joshua updated the PEP to remove *comprehensions, and it > is now accepted. > > On Wed, Feb 25, 2015 at 11:42 AM, Guido van Rossum > wrote: > >> I'm back, I've re-read the PEP, and I've re-read the long thread with >> "(no subject)". >> >> I think Georg Brandl nailed it: >> >> """ >> >> >> >> >> >> >> >> >> *I like the "sequence and dict flattening" part of the PEP, mostly >> because itis consistent and should be easy to understand, but the >> comprehension syntaxenhancements seem to be bad for readability and >> "comprehending" what the codedoes.The call syntax part is a mixed bag on >> the one hand it is nice to be consistent with the extended possibilities in >> literals (flattening), but on the other hand there would be small but >> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* >> """ >> >> Greg Ewing followed up explaining that the inconsistency between dict >> flattening and call syntax is inherent in the pre-existing different rules >> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, >> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, >> **{'a': 1}) is a TypeError.) >> >> For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations >> for function calls proposed by the PEP is an easy +1 -- it's a >> straightforward extension of the existing pattern, and anybody who knows >> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, >> **e) means shouldn't be hard either. Understanding the edge case for >> duplicate keys with f(**d, **e) is a little harder, but the error messages >> are pretty clear, and it is not a new edge case. >> >> The sequence and dict flattening syntax proposals are also clean and >> logical -- we already have *-unpacking on the receiving side, so allowing >> *x in tuple expressions reads pretty naturally (and the similarity with *a >> in argument lists certainly helps). From here, having [a, *x, b, *y] is >> also natural, and then the extension to other displays is natural: {a, *x, >> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. >> >> So that leaves comprehensions. IIRC, during the development of the patch >> we realized that f(*x for x in xs) is sufficiently ambiguous that we >> decided to disallow it -- note that f(x for x in xs) is already somewhat of >> a special case because an argument can only be a "bare" generator >> expression if it is the only argument. The same reasoning doesn't apply (in >> that form) to list, set and dict comprehensions -- while f(x for x in xs) >> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the >> same as [(x for x in xs)] (that's a list of one element, and the element is >> a generator expression). >> >> The basic premise of this part of the proposal is that if you have a few >> iterables, the new proposal (without comprehensions) lets you create a list >> or generator expression that iterates over all of them, essentially >> flattening them: >> >> >>> xs = [1, 2, 3] >> >>> ys = ['abc', 'def'] >> >>> zs = [99] >> >>> [*xs, *ys, *zs] >> [1, 2, 3, 'abc', 'def', 99] >> >>> >> >> But now suppose you have a list of iterables: >> >> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >> >>> [*xss[0], *xss[1], *xss[2]] >> [1, 2, 3, 'abc', 'def', 99] >> >>> >> >> Wouldn't it be nice if you could write the latter using a comprehension? >> >> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >> >>> [*xs for xs in xss] >> [1, 2, 3, 'abc', 'def', 99] >> >>> >> >> This is somewhat seductive, and the following is even nicer: the *xs >> position may be an expression, e.g.: >> >> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >> >>> [*xs[:2] for xs in xss] >> [1, 2, 'abc', 'def', 99] >> >>> >> >> On the other hand, I had to explore the possibilities here by >> experimenting in the interpreter, and I discovered some odd edge cases >> (e.g. you can parenthesize the starred expression, but that seems a >> syntactic accident). >> >> All in all I am personally +0 on the comprehension part of the PEP, and I >> like that it provides a way to "flatten" a sequence of sequences, but I >> think very few people in the thread have supported this part. Therefore I >> would like to ask Neil to update the PEP and the patch to take out the >> comprehension part, so that the two "easy wins" can make it into Python 3.5 >> (basically, I am accepting two-thirds of the PEP :-). There is some time >> yet until alpha 2. >> >> I would also like code reviewers (Benjamin?) to start reviewing the patch >> , taking into account that the >> comprehension part needs to be removed. >> >> -- >> --Guido van Rossum (python.org/~guido) >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Feb 26 21:37:29 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 Feb 2015 12:37:29 -0800 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: <54EF8409.80906@stoneleaf.us> On 02/26/2015 12:19 PM, Guido van Rossum wrote: > As a follow-up, Joshua updated the PEP to remove *comprehensions, and it is now accepted. Congratulations Thomas, Joshua, and Neil!! -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From brett at python.org Thu Feb 26 22:03:15 2015 From: brett at python.org (Brett Cannon) Date: Thu, 26 Feb 2015 21:03:15 +0000 Subject: [Python-Dev] PEP 448 review References: <54EF8409.80906@stoneleaf.us> Message-ID: On Thu, Feb 26, 2015 at 3:38 PM Ethan Furman wrote: > On 02/26/2015 12:19 PM, Guido van Rossum wrote: > > > As a follow-up, Joshua updated the PEP to remove *comprehensions, and it > is now accepted. > > Congratulations Thomas, Joshua, and Neil!! > I'll add a "thanks" to everyone involved with the PEP since it was an involved one implementation-wise and discussion-wise. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Feb 26 22:34:00 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 26 Feb 2015 13:34:00 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: Accepted! Thanks for your patience, Paul, and thanks everyone for their feedback. I know there are still a few small edits to the PEP, but those don't affect my acceptance. Congrats! --Guido On Thu, Feb 26, 2015 at 9:05 AM, Paul Moore wrote: > On 24 February 2015 at 18:24, Guido van Rossum wrote: > > Here's my review. I really like where this is going but I have a few > > questions and suggestions (I can't help myself :-). > > OK, I've updated both the PEP and the patch based on follow-up > discussions. I think (again!) it is ready to go. > > I've included the updated PEP inline below, it'll be available at > peps.python.org as soon as someone has a chance to upload it. > > Thanks to everyone for the various comments. If I've missed anything > that someone thinks I'd said I would change, please let me know. > > Paul > > PEP: 441 > Title: Improving Python ZIP Application Support > Version: $Revision$ > Last-Modified: $Date$ > Author: Daniel Holth , > Paul Moore > Discussions-To: > https://mail.python.org/pipermail/python-dev/2015-February/138277.html > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 30 March 2013 > Post-History: 30 March 2013, 1 April 2013, 16 February 2015 > > Improving Python ZIP Application Support > ======================================== > > Python has had the ability to execute directories or ZIP-format > archives as scripts since version 2.6 [1]_. When invoked with a zip > file or directory as its first argument the interpreter adds that > directory to sys.path and executes the ``__main__`` module. These > archives provide a great way to publish software that needs to be > distributed as a single file script but is complex enough to need to > be written as a collection of modules. > > This feature is not as popular as it should be mainly because it was > not promoted as part of Python 2.6 [2]_, so that it is relatively > unknown, but also because the Windows installer does not register a > file extension (other than ``.py``) for this format of file, to associate > with the launcher. > > This PEP proposes to fix these problems by re-publicising the feature, > defining the ``.pyz`` and ``.pyzw`` extensions as "Python ZIP Applications" > and "Windowed Python ZIP Applications", and providing some simple > tooling to manage the format. > > A New Python ZIP Application Extension > ====================================== > > The terminology "Python Zip Application" will be the formal term used > for a zip-format archive that contains Python code in a form that can > be directly executed by Python (specifically, it must have a > ``__main__.py`` file in the root directory of the archive). The > extension ``.pyz`` will be formally associated with such files. > > The Python 3.5 installer will associate ``.pyz`` and ``.pyzw`` "Python > Zip Applications" with the platform launcher so they can be executed. > A ``.pyz`` archive is a console application and a ``.pyzw`` archive is a > windowed application, indicating whether the console should appear > when running the app. > > On Unix, it would be ideal if the ``.pyz`` extension and the name > "Python Zip Application" were registered (in the mime types database?). > However, such an association is out of scope for this PEP. > > Python Zip applications can be prefixed with a ``#!`` line > pointing to the correct Python interpreter and an optional > explanation:: > > #!/usr/bin/env python3 > # Python application packed with zipapp module > (binary contents of archive) > > On Unix, this allows the OS to run the file with the correct > interpreter, via the standard "shebang" support. On Windows, the > Python launcher implements shebang support. > > However, it is always possible to execute a ``.pyz`` application by > supplying the filename to the Python interpreter directly. > > As background, ZIP archives are defined with a footer containing > relative offsets from the end of the file. They remain valid when > concatenated to the end of any other file. This feature is completely > standard and is how self-extracting ZIP archives and the bdist_wininst > installer format work. > > > Minimal Tooling: The zipapp Module > ================================== > > This PEP also proposes including a module for working with these > archives. The module will contain functions for working with Python > zip application archives, and a command line interface (via ``python > -m zipapp``) for their creation and manipulation. > > More complete tools for managing Python Zip Applications are > encouraged as 3rd party applications on PyPI. Currently, pyzzer [5]_ > and pex [6]_ are two such tools. > > Module Interface > ---------------- > > The zipapp module will provide the following functions: > > ``create_archive(source, target=None, interpreter=None, main=None)`` > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > Create an application archive from *source*. The source can be any > of the following: > > * The name of a directory, in which case a new application archive > will be created from the content of that directory. > * The name of an existing application archive file, in which case the > file is copied to the target. The file name should include the > ``.pyz`` extension, if required. > * A file object open for reading in bytes mode. The content of the > file should be an application archive, and the file object is > assumed to be positioned at the start of the archive. > > The *target* argument determines where the resulting archive will be > written: > > * If it is the name of a file, the archive will be written to that > file. > * If it is an open file object, the archive will be written to that > file object, which must be open for writing in bytes mode. > * If the target is omitted (or None), the source must be a directory > and the target will be a file with the same name as the source, with > a ``.pyz`` extension added. > > The *interpreter* argument specifies the name of the Python > interpreter with which the archive will be executed. It is written as > a "shebang" line at the start of the archive. On Unix, this will be > interpreted by the OS, and on Windows it will be handled by the Python > launcher. Omitting the *interpreter* results in no shebang line being > written. If an interpreter is specified, and the target is a > filename, the executable bit of the target file will be set. > > The *main* argument specifies the name of a callable which will be > used as the main program for the archive. It can only be specified if > the source is a directory, and the source does not already contain a > ``__main__.py`` file. The *main* argument should take the form > "pkg.module:callable" and the archive will be run by importing > "pkg.module" and executing the given callable with no arguments. It > is an error to omit *main* if the source is a directory and does not > contain a ``__main__.py`` file, as otherwise the resulting archive > would not be executable. > > If a file object is specified for *source* or *target*, it is the > caller's responsibility to close it after calling create_archive. > > When copying an existing archive, file objects supplied only need > ``read`` and ``readline``, or ``write`` methods. When creating an > archive from a directory, if the target is a file object it will be > passed to the ``zipfile.ZipFile`` class, and must supply the methods > needed by that class. > > ``get_interpreter(archive)`` > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > Returns the interpreter specified in the shebang line of the > *archive*. If there is no shebang, the function returns ``None``. > The *archive* argument can be a filename or a file-like object open > for reading in bytes mode. > > > Command Line Usage > ------------------ > > The zipapp module can be run with the python ``-m`` flag. The command > line interface is as follows:: > > python -m zipapp directory [options] > > Create an archive from the given directory. An archive will > be created from the contents of that directory. The archive > will have the same name as the source directory with a .pyz > extension. > > The following options can be specified: > > -o archive / --output archive > > The destination archive will have the specified name. The > given name will be used as written, so should include the > ".pyz" extension. > > -p interpreter / --python interpreter > > The given interpreter will be written to the shebang line > of the archive. If this option is not given, the archive > will have no shebang line. > > -m pkg.mod:fn / --main pkg.mod:fn > > The source directory must not have a __main__.py file. The > archiver will write a __main__.py file into the target > which calls fn from the module pkg.mod. > > The behaviour of the command line interface matches that of > ``zipapp.create_archive()``. > > In addition, it is possible to use the command line interface to work > with an existing archive:: > > python -m zipapp app.pyz --show > > Displays the shebang line of an archive. Output is of the > form > > Interpreter: /usr/bin/env > or > Interpreter: > > and is intended for diagnostic use, not for scripts. > > python -m zipapp app.pyz -o newapp.pyz [-p interpreter] > > Copy app.pyz to newapp.pyz, modifying the shebang line based > on the -p option (as for creating an archive, no -p option > means remove the shebang line). Specifying a destination is > mandatory. > > In-place modification of an archive is *not* supported, as the > risk of damaging archives is too great for a simple tool. > > As noted, the archives are standard zip files, and so can be unpacked > using any standard ZIP utility or Python's zipfile module. For this > reason, no interfaces to list the contents of an archive, or unpack > them, are provided or needed. > > FAQ > --- > > Are you sure a standard ZIP utility can handle ``#!`` at the beginning? > Absolutely. The zipfile specification allows for arbitrary data to > be prepended to a zipfile. This feature is commonly used by > "self-extracting zip" programs. If your archive program can't > handle this, it is a bug in your archive program. > > Isn't zipapp just a very thin wrapper over the zipfile module? > Yes. If you prefer to build your own Python zip application > archives using other tools, they will work just as well. The > zipapp module is a convenience, nothing more. > > Why not use just use a ``.zip`` or ``.py`` extension? > Users expect a ``.zip`` file to be opened with an archive tool, and > expect a ``.py`` file to contain readable text. Both would be > confusing for this use case. > > How does this compete with existing package formats? > The sdist, bdist and wheel formats are designed for packaging of > modules to be installed into an existing Python installation. > They are not intended to be used without installing. The > executable zip format is specifically designed for standalone use, > without needing to be installed. They are in effect a multi-file > version of a standalone Python script. > > Rejected Proposals > ================== > > Convenience Values for Shebang Lines > ------------------------------------ > > Is it worth having "convenience" forms for any of the common > interpreter values? For example, ``-p 3`` meaning the same as ``-p > "/usr/bin/env python3"``. It would save a lot of typing for the > common cases, as well as giving cross-platform options for people who > don't want or need to understand the intricacies of shebang handling > on "other" platforms. > > Downsides are that it's not obvious how to translate the > abbreviations. For example, should "3" mean "/usr/bin/env python3", > "/usr/bin/python3", "python3", or something else? Also, there is no > obvious short form for the key case of "/usr/bin/env python" (any > available version of Python), which could easily result in scripts > being written with overly-restrictive shebang lines. > > Overall, this seems like there are more problems than benefits, and as > a result has been dropped from consideration. > > Registering ``.pyz`` as a Media Type > ------------------------------------ > > It was suggested [3]_ that the ``.pyz`` extension should be registered > in the Unix database of extensions. While it makes sense to do this > as an equivalent of the Windows installer registering the extension, > the ``.py`` extension is not listed in the media types database [4]_. > It doesn't seem reasonable to register ``.pyz`` without ``.py``, so > this idea has been omitted from this PEP. An interested party could > arrange for *both* ``.py`` and ``.pyz`` to be registered at a future > date. > > Default Interpreter > ------------------- > > The initial draft of this PEP proposed using ``/usr/bin/env python`` > as the default interpreter. Unix users have problems with this > behaviour, as the default for the python command on many distributions > is Python 2, and it is felt that this PEP should prefer Python 3 by > default. However, using a command of ``python3`` can result in > unexpected behaviour for Windows users, where the default behaviour of > the launcher for the command ``python`` is commonly customised by users, > but the behaviour of ``python3`` may not be modified to match. > > As a result, the principle "in the face of ambiguity, refuse to guess" > has been invoked, and archives have no shebang line unless explicitly > requested. On Windows, the archives will still be run (with the > default Python) by the launcher, and on Unix, the archives can be run > by explicitly invoking the desired Python interpreter. > > Command Line Tool to Manage Shebang Lines > ----------------------------------------- > > It is conceivable that users would want to modify the shebang line for > an existing archive, or even just display the current shebang line. > This is tricky to do so with existing tools (zip programs typically > ignore prepended data totally, and text editors can have trouble > editing files containing binary data). > > The zipapp module provides functions to handle the shebang line, but > does not include a command line interface to that functionality. This > is because it is not clear how to provide one without the resulting > interface being over-complex and potentially confusing. Changing the > shebang line is expected to be an uncommon requirement. > > Reference Implementation > ======================== > > A reference implementation is at http://bugs.python.org/issue23491. > > References > ========== > > .. [1] Allow interpreter to execute a zip file > (http://bugs.python.org/issue1739468) > > .. [2] Feature is not documented > (http://bugs.python.org/issue17359) > > .. [3] Discussion of adding a .pyz mime type on python-dev > (https://mail.python.org/pipermail/python-dev/2015-February/138338.html > ) > > .. [4] Register of media types > (http://www.iana.org/assignments/media-types/media-types.xhtml) > > .. [5] pyzzer - A tool for creating Python-executable archives > (https://pypi.python.org/pypi/pyzzer) > > .. [6] pex - The PEX packaging toolchain > (https://pypi.python.org/pypi/pex) > > The discussion of this PEP took place on the python-dev mailing list, > in the thread starting at > https://mail.python.org/pipermail/python-dev/2015-February/138277.html > > Copyright > ========= > > This document has been placed into the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Feb 26 22:36:50 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 26 Feb 2015 13:36:50 -0800 Subject: [Python-Dev] Request for Pronouncement: PEP 486 - Make the Python Launcher aware of virtual environments In-Reply-To: References: Message-ID: PEP 486 is hereby accepted. You can fix the typo at the same time as marking it as accepted. Paul, thanks and congrats, everyone else, thanks for the feedback! --Guido On Tue, Feb 24, 2015 at 12:45 PM, Guido van Rossum wrote: > Thanks for doing this, Paul! > > I think this is a fine PEP and I have no comments. (Note that I am not > much of a Windows user so I am just taking most of the things you claim at > face value. :-) I will accept the PEP in 1-3 days unless a storm of > objections gets piled onto this thread. > > --Guido > > On Sat, Feb 21, 2015 at 1:31 AM, Paul Moore wrote: > >> The discussion on PEP 486 (started at >> https://mail.python.org/pipermail/python-dev/2015-February/138171.html, >> following from the thread at >> https://mail.python.org/pipermail/python-dev/2015-February/138160.html) >> seems to have died down. There's an implementation at >> http://bugs.python.org/issue23465. >> >> So, I think this is ready for pronouncement. >> >> Thanks, >> Paul >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Feb 26 22:48:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 Feb 2015 21:48:25 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 26 February 2015 at 21:34, Guido van Rossum wrote: > Accepted! > > Thanks for your patience, Paul, and thanks everyone for their feedback. > > I know there are still a few small edits to the PEP, but those don't affect > my acceptance. Congrats! Excellent, thanks to everyone for the helpful comments throughout. Paul From rymg19 at gmail.com Fri Feb 27 01:20:26 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 26 Feb 2015 18:20:26 -0600 Subject: [Python-Dev] Working on issue 23496: should I use a macro test or an edit to configure.ac? In-Reply-To: References: Message-ID: DOES NOBODY HAVE AN ANSWER TO THIS??? I'm REALLY relying on someone who works on Python to answer this. PLEASE?? On Wed, Feb 25, 2015 at 12:17 PM, Ryan Gonzalez wrote: > So... > > There was a recent discussion here on porting Python to Android. Well, for > those of you who saw too many unread messages and marked the whole thread > as read like I usually do, I helped Cyd figure out some patches that make > it work. Cyd then opened Issue 23496 . > Now, I'm going to try to redo the patches against HEAD (or tip in Mercurial > language). > > Which leads me to the question. See, of course, the patches should only be > enabled if Python is being built targeting Android, but I'm not sure how > that should be detected. > > I know that the Android target triple is arm-linux-androideabi. Should I > test for the __ANDROID__ macro in the source file, or should I modify > configure.ac to detect Android and define its own macro? Also, if a > configure.ac edit is preferred, where should it be? It's slightly painful > to scan through all 4000 lines of Python's configure script. ;) > > -- > Ryan > If anybody ever asks me why I prefer C++ to C, my answer will be simple: > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was > nul-terminated." > Personal reality distortion fields are immune to contradictory evidence. - > srean > Check out my website: http://kirbyfan64.github.io/ > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Feb 27 01:37:02 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 Feb 2015 16:37:02 -0800 Subject: [Python-Dev] Working on issue 23496: should I use a macro test or an edit to configure.ac? In-Reply-To: References: Message-ID: <54EFBC2E.9060908@stoneleaf.us> On 02/25/2015 10:17 AM, Ryan Gonzalez wrote: > Which leads me to the question. See, of course, the patches should only be enabled if Python is being built targeting > Android, but I'm not sure how that should be detected. > > I know that the Android target triple is arm-linux-androideabi. Should I test for the __ANDROID__ macro in the source > file, or should I modify configure.ac to detect Android and define its own macro? Also, if a > configure.ac edit is preferred, where should it be? It's slightly painful to scan through all 4000 > lines of Python's configure script. ;) Anybody have any words of wisdom on this? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rmsr at lab.net Fri Feb 27 02:13:20 2015 From: rmsr at lab.net (Ryan Smith-Roberts) Date: Thu, 26 Feb 2015 17:13:20 -0800 Subject: [Python-Dev] Working on issue 23496: should I use a macro test or an edit to configure.ac? In-Reply-To: References: Message-ID: I'm not an official cpython developer but ifdef __ANDROID__ is quite in line with other per-platform support (__FreeBSD__, __linux__, etc), as well as already being in use in Modules/_posixsubprocess.c. Is __ANDROID__ not being defined when it should be? On Thu, Feb 26, 2015 at 4:20 PM, Ryan Gonzalez wrote: > DOES NOBODY HAVE AN ANSWER TO THIS??? > > I'm REALLY relying on someone who works on Python to answer this. PLEASE?? > > On Wed, Feb 25, 2015 at 12:17 PM, Ryan Gonzalez wrote: > >> So... >> >> There was a recent discussion here on porting Python to Android. Well, >> for those of you who saw too many unread messages and marked the whole >> thread as read like I usually do, I helped Cyd figure out some patches that >> make it work. Cyd then opened Issue 23496 >> . Now, I'm going to try to redo the >> patches against HEAD (or tip in Mercurial language). >> >> Which leads me to the question. See, of course, the patches should only >> be enabled if Python is being built targeting Android, but I'm not sure how >> that should be detected. >> >> I know that the Android target triple is arm-linux-androideabi. Should I >> test for the __ANDROID__ macro in the source file, or should I modify >> configure.ac to detect Android and define its own macro? Also, if a >> configure.ac edit is preferred, where should it be? It's slightly >> painful to scan through all 4000 lines of Python's configure script. ;) >> >> -- >> Ryan >> If anybody ever asks me why I prefer C++ to C, my answer will be simple: >> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was >> nul-terminated." >> Personal reality distortion fields are immune to contradictory evidence. >> - srean >> Check out my website: http://kirbyfan64.github.io/ >> > > > > -- > Ryan > If anybody ever asks me why I prefer C++ to C, my answer will be simple: > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was > nul-terminated." > Personal reality distortion fields are immune to contradictory evidence. - > srean > Check out my website: http://kirbyfan64.github.io/ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rmsr%40lab.net > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Fri Feb 27 02:20:06 2015 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 26 Feb 2015 20:20:06 -0500 Subject: [Python-Dev] ubuntu buildbot In-Reply-To: References: <1416799084.1792952.194543661.2FC7FCAA@webmail.messagingengine.com> <1416845241.1998368.194775069.7DEEE5BE@webmail.messagingengine.com> Message-ID: <1425000006.2827958.233050233.5FA5CECD@webmail.messagingengine.com> On Wed, Feb 25, 2015, at 22:38, David Bolen wrote: > On Mon, Nov 24, 2014 at 11:07 AM, Benjamin Peterson > wrote: > > > > On Mon, Nov 24, 2014, at 00:33, David Bolen wrote: > >> Yeah, it definitely needs it. Historically it was intentional as my own > >> servers were all on 8.04, but the last of those moved 12.04 last year. > >> > >> I think there's already a 12.04 buildbot, so perhaps 14.04 would be > >> better? I do prefer sticking with an LTS. > > > > 14.04 would be good. > > > >> It'll need to move to 64-bit given the hosting environment, at least for > >> the kernel. I could do 32-bit userspace via multiarch if keeping a 32-bit > >> buildbot in the mix is attractive. > > > > It'd be nice to keep a 32-bit bot around. > > Took a bit longer than anticipated, but the slave upgrade is complete. > > The bolen-ubuntu slave is now a 32-bit Ubuntu 14.04.2 LTS system. > I've re-run the most recent 2.7, 3.4 and 3.x builds which all pass > (though with a few new compiler warnings in some cases). Thank you very much! From ethan at stoneleaf.us Fri Feb 27 02:19:56 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 Feb 2015 17:19:56 -0800 Subject: [Python-Dev] Working on issue 23496: should I use a macro test or an edit to configure.ac? In-Reply-To: References: Message-ID: <54EFC63C.60808@stoneleaf.us> On 02/26/2015 05:13 PM, Ryan Smith-Roberts wrote: > I'm not an official cpython developer but ifdef __ANDROID__ is quite in line with > other per-platform support (__FreeBSD__, __linux__, etc), as well as already being > in use in Modules/_posixsubprocess.c. Is __ANDROID__ not being defined when it > should be? I believe the issue is adding in Android support, and how best to do it. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From larry at hastings.org Wed Feb 25 14:07:08 2015 From: larry at hastings.org (Larry Hastings) Date: Wed, 25 Feb 2015 05:07:08 -0800 Subject: [Python-Dev] [RELEASED] Python 3.4.3 is now available Message-ID: <54EDC8FC.5040003@hastings.org> On behalf of the Python development community and the Python 3.4 release team, I'm pleased to announce the availability of Python 3.4.3. Python 3.4.3 has many bugfixes and other small improvements over 3.4.2. You can find it here: https://www.python.org/downloads/release/python-343/ The release slipped by two or three days, depending on what time zone you're in. This is my fault--I apologize for the inconvenience. Cheers, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmsr at lab.net Fri Feb 27 02:52:35 2015 From: rmsr at lab.net (Ryan Smith-Roberts) Date: Thu, 26 Feb 2015 17:52:35 -0800 Subject: [Python-Dev] Working on issue 23496: should I use a macro test or an edit to configure.ac? In-Reply-To: References: Message-ID: On Thu, Feb 26, 2015 at 5:13 PM, Ryan Smith-Roberts wrote: > I'm not an official cpython developer but ifdef __ANDROID__ is quite in line > with other per-platform support (__FreeBSD__, __linux__, etc), as well as > already being in use in Modules/_posixsubprocess.c. Is __ANDROID__ not being > defined when it should be? Might as well spend the time to answer my own question: Some Googling indicates that __ANDROID__ is baked into the NDK toolchain, so it's definitely what one would use to wrap Android-specific code. "__ANDROID__ shall always be defined by the toolchain, without a need for special flags though, so please rely on that instead in your code." - David Turner, Android NDK Lead, 2010. From ethan at stoneleaf.us Fri Feb 27 03:11:13 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 Feb 2015 18:11:13 -0800 Subject: [Python-Dev] Working on issue 23496: should I use a macro test or an edit to configure.ac? In-Reply-To: References: Message-ID: <54EFD241.1060502@stoneleaf.us> On 02/26/2015 05:52 PM, Ryan Smith-Roberts wrote: > Might as well spend the time to answer my own question: Thanks! Much appreciated. :) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rymg19 at gmail.com Fri Feb 27 04:13:08 2015 From: rymg19 at gmail.com (Ryan) Date: Thu, 26 Feb 2015 21:13:08 -0600 Subject: [Python-Dev] Working on issue 23496: should I use a macro test or an edit to configure.ac? In-Reply-To: References: Message-ID: Thank you so much! Ryan Smith-Roberts wrote: >On Thu, Feb 26, 2015 at 5:13 PM, Ryan Smith-Roberts >wrote: >> I'm not an official cpython developer but ifdef __ANDROID__ is quite >in line >> with other per-platform support (__FreeBSD__, __linux__, etc), as >well as >> already being in use in Modules/_posixsubprocess.c. Is __ANDROID__ >not being >> defined when it should be? > >Might as well spend the time to answer my own question: > >Some Googling indicates that __ANDROID__ is baked into the NDK >toolchain, so it's definitely what one would use to wrap >Android-specific code. > >"__ANDROID__ shall always be defined by the toolchain, without a need >for special flags though, so please rely on that instead in your >code." - David Turner, Android NDK Lead, 2010. >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Feb 27 09:02:28 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 27 Feb 2015 18:02:28 +1000 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: <54EF8409.80906@stoneleaf.us> Message-ID: On 27 Feb 2015 07:12, "Brett Cannon" wrote: > > > > On Thu, Feb 26, 2015 at 3:38 PM Ethan Furman wrote: >> >> On 02/26/2015 12:19 PM, Guido van Rossum wrote: >> >> > As a follow-up, Joshua updated the PEP to remove *comprehensions, and it is now accepted. >> >> Congratulations Thomas, Joshua, and Neil!! > > > I'll add a "thanks" to everyone involved with the PEP since it was an involved one implementation-wise and discussion-wise. Likewise! It was a hard slog to get there, but this will be a nice quality of life improvement for 3.5+ users. Thanks to everyone involved in moving it forward :) Regards, Nick. > > -Brett > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoffspear at gmail.com Fri Feb 27 13:51:57 2015 From: geoffspear at gmail.com (Geoffrey Spear) Date: Fri, 27 Feb 2015 07:51:57 -0500 Subject: [Python-Dev] [RELEASED] Python 3.4.3 is now available In-Reply-To: <54EDC8FC.5040003@hastings.org> References: <54EDC8FC.5040003@hastings.org> Message-ID: The download button (for Windows anyway) on the python.org Download dropdown menu is broken; there's a small gray square where the 3.4.3 button should be, with the 2.7.9 one working fine. On Wed, Feb 25, 2015 at 8:07 AM, Larry Hastings wrote: > > > On behalf of the Python development community and the Python 3.4 release > team, I'm pleased to announce the availability of Python 3.4.3. Python > 3.4.3 has many bugfixes and other small improvements over 3.4.2. > > You can find it here: > > https://www.python.org/downloads/release/python-343/ > > > The release slipped by two or three days, depending on what time zone > you're in. This is my fault--I apologize for the inconvenience. > > Cheers, > > > */arry* > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/geoffspear%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Feb 27 14:59:52 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 27 Feb 2015 14:59:52 +0100 Subject: [Python-Dev] [RELEASED] Python 3.4.3 is now available In-Reply-To: References: <54EDC8FC.5040003@hastings.org> Message-ID: Hum, it's probably an attack of the Python 2 mafia who fight against Python 3! Victor 2015-02-27 13:51 GMT+01:00 Geoffrey Spear : > The download button (for Windows anyway) on the python.org Download dropdown > menu is broken; there's a small gray square where the 3.4.3 button should > be, with the 2.7.9 one working fine. > > On Wed, Feb 25, 2015 at 8:07 AM, Larry Hastings wrote: >> >> >> >> On behalf of the Python development community and the Python 3.4 release >> team, I'm pleased to announce the availability of Python 3.4.3. Python >> 3.4.3 has many bugfixes and other small improvements over 3.4.2. >> >> You can find it here: >> >> https://www.python.org/downloads/release/python-343/ >> >> >> The release slipped by two or three days, depending on what time zone >> you're in. This is my fault--I apologize for the inconvenience. >> >> Cheers, >> >> >> /arry >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/geoffspear%40gmail.com >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From bcannon at gmail.com Fri Feb 27 15:40:02 2015 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 27 Feb 2015 14:40:02 +0000 Subject: [Python-Dev] Update to PEP 11 to clarify garnering platform support References: Message-ID: On Fri, Feb 20, 2015 at 1:47 PM Brett Cannon wrote: > I just realized I actually never committed this change. Assuming no new > objections I'll commit this in the near future (promise this time =). > My proposed changes have now been committed. Thanks to everyone who provided feedback! This should hopefully make it much clearer what it takes to accept platform-specific patches (i.e., core dev willing to maintain the compatibility and a stable buildbot for the platform). For those trying to get Python working on Android, this will mean a conversation will be necessary about how to get a buildbot or some form or regular testing set up in order to accept Android-specific patches (along with a core dev willing to keep an eye on the compatibility). -Brett > > > On Fri May 16 2014 at 1:51:00 PM Brett Cannon wrote: > >> Here is some proposed wording. Since it is more of a clarification of >> what it takes to garner support -- which is just a new section -- rather >> than a complete rewrite I'm including just the diff to make it easier to >> read the changes. >> >> >> *diff -r 49d18bb47ebc pep-0011.txt* >> >> *--- a/pep-0011.txt Wed May 14 11:18:22 2014 -0400* >> >> *+++ b/pep-0011.txt Fri May 16 13:48:30 2014 -0400* >> >> @@ -2,22 +2,21 @@ >> >> Title: Removing support for little used platforms >> >> Version: $Revision$ >> >> Last-Modified: $Date$ >> >> -Author: martin at v.loewis.de (Martin von L?wis) >> >> +Author: Martin von L?wis , >> >> + Brett Cannon >> >> Status: Active >> >> Type: Process >> >> Content-Type: text/x-rst >> >> Created: 07-Jul-2002 >> >> Post-History: 18-Aug-2007 >> >> + 16-May-2014 >> >> >> >> >> >> Abstract >> >> -------- >> >> >> >> -This PEP documents operating systems (platforms) which are not >> >> -supported in Python anymore. For some of these systems, >> >> -supporting code might be still part of Python, but will be removed >> >> -in a future release - unless somebody steps forward as a volunteer >> >> -to maintain this code. >> >> +This PEP documents how an operating system (platform) garners >> >> +support in Python as well as documenting past support. >> >> >> >> >> >> Rationale >> >> @@ -37,16 +36,53 @@ >> >> change to the Python source code will work on all supported >> >> platforms. >> >> >> >> -To reduce this risk, this PEP proposes a procedure to remove code >> >> -for platforms with no Python users. >> >> +To reduce this risk, this PEP specifies what is required for a >> >> +platform to be considered supported by Python as well as providing a >> >> +procedure to remove code for platforms with little or no Python >> >> +users. >> >> >> >> +Supporting platforms >> >> +-------------------- >> >> + >> >> +Gaining official platform support requires two things. First, a core >> >> +developer needs to volunteer to maintain platform-specific code. This >> >> +core developer can either already be a member of the Python >> >> +development team or be given contributor rights on the basis of >> >> +maintaining platform support (it is at the discretion of the Python >> >> +development team to decide if a person is ready to have such rights >> >> +even if it is just for supporting a specific platform). >> >> + >> >> +Second, a stable buildbot must be provided [2]_. This guarantees that >> >> +platform support will not be accidentally broken by a Python core >> >> +developer who does not have personal access to the platform. For a >> >> +buildbot to be considered stable it requires that the machine be >> >> +reliably up and functioning (but it is up to the Python core >> >> +developers to decide whether to promote a buildbot to being >> >> +considered stable). >> >> + >> >> +This policy does not disqualify supporting other platforms >> >> +indirectly. Patches which are not platform-specific but still done to >> >> +add platform support will be considered for inclusion. For example, >> >> +if platform-independent changes were necessary in the configure >> >> +script which was motivated to support a specific platform that would >> >> +be accepted. Patches which add platform-specific code such as the >> >> +name of a specific platform to the configure script will generally >> >> +not be accepted without the platform having official support. >> >> + >> >> +CPU architecture and compiler support are viewed in a similar manner >> >> +as platforms. For example, to consider the ARM architecture supported >> >> +a buildbot running on ARM would be required along with support from >> >> +the Python development team. In general it is not required to have >> >> +a CPU architecture run under every possible platform in order to be >> >> +considered supported. >> >> >> >> Unsupporting platforms >> >> ---------------------- >> >> >> >> -If a certain platform that currently has special code in it is >> >> -deemed to be without Python users, a note must be posted in this >> >> -PEP that this platform is no longer actively supported. This >> >> +If a certain platform that currently has special code in Python is >> >> +deemed to be without Python users or lacks proper support from the >> >> +Python development team and/or a buildbot, a note must be posted in >> >> +this PEP that this platform is no longer actively supported. This >> >> note must include: >> >> >> >> - the name of the system >> >> @@ -69,8 +105,8 @@ >> >> forward and offer maintenance. >> >> >> >> >> >> -Resupporting platforms >> >> ----------------------- >> >> +Re-supporting platforms >> >> +----------------------- >> >> >> >> If a user of a platform wants to see this platform supported >> >> again, he may volunteer to maintain the platform support. Such an >> >> @@ -101,7 +137,7 @@ >> >> release is made. Developers of extension modules will generally need >> >> to use the same Visual Studio release; they are concerned both with >> >> the availability of the versions they need to use, and with keeping >> >> -the zoo of versions small. The Python source tree will keep >> >> +the zoo of versions small. The Python source tree will keep >> >> unmaintained build files for older Visual Studio releases, for which >> >> patches will be accepted. Such build files will be removed from the >> >> source tree 3 years after the extended support for the compiler has >> >> @@ -223,6 +259,7 @@ >> >> ---------- >> >> >> >> .. [1] http://support.microsoft.com/lifecycle/ >> >> +.. [2] http://buildbot.python.org/3.x.stable/ >> >> >> >> Copyright >> >> --------- >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Feb 27 17:28:45 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 27 Feb 2015 08:28:45 -0800 Subject: [Python-Dev] Update to PEP 11 to clarify garnering platform support In-Reply-To: References: Message-ID: <54F09B3D.5070103@stoneleaf.us> On 02/27/2015 06:40 AM, Brett Cannon wrote: > My proposed changes have now been committed. Thanks to everyone who provided feedback! Thanks, Brett! > For those trying to get Python working on Android, this will mean a conversation will be necessary about how to get a > buildbot or some form or regular testing set up in order to accept Android-specific patches (along with a core dev > willing to keep an eye on the compatibility). Noted. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From thomas at python.org Fri Feb 27 18:01:11 2015 From: thomas at python.org (Thomas Wouters) Date: Fri, 27 Feb 2015 09:01:11 -0800 Subject: [Python-Dev] PEP 448 review In-Reply-To: <54EF8409.80906@stoneleaf.us> References: <54EF8409.80906@stoneleaf.us> Message-ID: On Thu, Feb 26, 2015 at 12:37 PM, Ethan Furman wrote: > On 02/26/2015 12:19 PM, Guido van Rossum wrote: > > > As a follow-up, Joshua updated the PEP to remove *comprehensions, and it > is now accepted. > > Congratulations Thomas, Joshua, and Neil!! > Wot, me? No, no, all credit goes to Joshua and Neil. All I did was propose the change too late for Python 3, then procrastinate until someone else picked it up. (My procrastination goes so far that I *still* mean to review the latest patch, as soon as I get round tuits...) I do stand behind the idea, though, and I'm happy with the PEP and with the final decision. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Feb 27 18:08:16 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 27 Feb 2015 18:08:16 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20150227170816.E729156216@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-02-20 - 2015-02-27) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4788 (+18) closed 30510 (+26) total 35298 (+44) Open issues with patches: 2245 Issues opened (34) ================== #23458: [2.7] random: make the file descriptor non-inheritable (on POS http://bugs.python.org/issue23458 reopened by haypo #23494: adding timedelta to datetime object is not timezone aware http://bugs.python.org/issue23494 opened by gilsho #23495: The writer.writerows method should be documented as accepting http://bugs.python.org/issue23495 opened by Steven.Barker #23496: Steps for Android Native Build of Python 3.4.2 http://bugs.python.org/issue23496 opened by chaselton #23498: Expose http.cookiejar.split_header_words() http://bugs.python.org/issue23498 opened by vadmium #23500: Argument Clinic: multiple macro definition http://bugs.python.org/issue23500 opened by serhiy.storchaka #23501: Argument Clinic: generate code into separate files by default http://bugs.python.org/issue23501 opened by serhiy.storchaka #23502: pprint: added support for mapping proxy http://bugs.python.org/issue23502 opened by serhiy.storchaka #23503: undefined behavior in Objects/obmalloc.c http://bugs.python.org/issue23503 opened by dwight.guth #23504: Add __all__ into types http://bugs.python.org/issue23504 opened by serhiy.storchaka #23505: Urlparse insufficient validation leads to open redirect http://bugs.python.org/issue23505 opened by yaaboukir #23507: Tuple creation is too slow http://bugs.python.org/issue23507 opened by serhiy.storchaka #23508: Document & unittest the subprocess.getstatusoutput() status va http://bugs.python.org/issue23508 opened by gregory.p.smith #23509: Speed up Counter operators http://bugs.python.org/issue23509 opened by joern #23510: multiprocessing bug SyncManager and 'with' http://bugs.python.org/issue23510 opened by Cezary.Wagner #23512: List of builtins is not alphabetical on https://docs.python.or http://bugs.python.org/issue23512 opened by edsouza #23513: Add support for classes/object model in multiprocessing/pickle http://bugs.python.org/issue23513 opened by Cezary.Wagner #23514: multiprocessing documentation - little more examples for paral http://bugs.python.org/issue23514 opened by Cezary.Wagner #23516: pip: urllib3 does not encode userinfo section of URL with auth http://bugs.python.org/issue23516 opened by leotan #23517: datetime.utcfromtimestamp parses timestamps incorrectly http://bugs.python.org/issue23517 opened by tbarbugli #23520: test_os failed (python-3.4.3) http://bugs.python.org/issue23520 opened by avoevodkin #23521: OverflowError from timedelta * float in datetime.py http://bugs.python.org/issue23521 opened by belopolsky #23522: Misleading note in Statistics module documentation http://bugs.python.org/issue23522 opened by Journeyman08 #23524: Use _set_thread_local_invalid_parameter_handler in posixmodule http://bugs.python.org/issue23524 opened by steve.dower #23525: isbuiltin, isroutine, etc. http://bugs.python.org/issue23525 opened by abarnert #23526: Silence resource warnings in test_httplib http://bugs.python.org/issue23526 opened by ashkop #23527: test_smtpnet uses incorrect port for STARTTLS http://bugs.python.org/issue23527 opened by ashkop #23528: Limit decompressed data when reading from GzipFile http://bugs.python.org/issue23528 opened by vadmium #23529: Limit decompressed data when reading from LZMAFile and BZ2File http://bugs.python.org/issue23529 opened by vadmium #23530: os and multiprocessing.cpu_count do not respect cpuset/affinit http://bugs.python.org/issue23530 opened by jtaylor #23531: SSL operations cause entire process to hang http://bugs.python.org/issue23531 opened by johnkw #23532: add example of 'first match wins' to regex "|" documentation? http://bugs.python.org/issue23532 opened by Rick Otten #23534: `test_longdouble` fails on Mac when using system libffi (versi http://bugs.python.org/issue23534 opened by Maxime Belanger #23536: Add explicit information on config file format not supporting http://bugs.python.org/issue23536 opened by piotr.dobrogost Most recent 15 issues with no replies (15) ========================================== #23536: Add explicit information on config file format not supporting http://bugs.python.org/issue23536 #23527: test_smtpnet uses incorrect port for STARTTLS http://bugs.python.org/issue23527 #23525: isbuiltin, isroutine, etc. http://bugs.python.org/issue23525 #23522: Misleading note in Statistics module documentation http://bugs.python.org/issue23522 #23520: test_os failed (python-3.4.3) http://bugs.python.org/issue23520 #23512: List of builtins is not alphabetical on https://docs.python.or http://bugs.python.org/issue23512 #23504: Add __all__ into types http://bugs.python.org/issue23504 #23503: undefined behavior in Objects/obmalloc.c http://bugs.python.org/issue23503 #23502: pprint: added support for mapping proxy http://bugs.python.org/issue23502 #23501: Argument Clinic: generate code into separate files by default http://bugs.python.org/issue23501 #23498: Expose http.cookiejar.split_header_words() http://bugs.python.org/issue23498 #23487: argparse: add_subparsers 'action' broken http://bugs.python.org/issue23487 #23485: PEP 475: handle EINTR in the select and selectors module http://bugs.python.org/issue23485 #23470: OpenBSD buildbot uses wrong stdlib http://bugs.python.org/issue23470 #23469: Delete Misc/*.wpr files http://bugs.python.org/issue23469 Most recent 15 issues waiting for review (15) ============================================= #23529: Limit decompressed data when reading from LZMAFile and BZ2File http://bugs.python.org/issue23529 #23528: Limit decompressed data when reading from GzipFile http://bugs.python.org/issue23528 #23527: test_smtpnet uses incorrect port for STARTTLS http://bugs.python.org/issue23527 #23526: Silence resource warnings in test_httplib http://bugs.python.org/issue23526 #23524: Use _set_thread_local_invalid_parameter_handler in posixmodule http://bugs.python.org/issue23524 #23521: OverflowError from timedelta * float in datetime.py http://bugs.python.org/issue23521 #23509: Speed up Counter operators http://bugs.python.org/issue23509 #23507: Tuple creation is too slow http://bugs.python.org/issue23507 #23504: Add __all__ into types http://bugs.python.org/issue23504 #23502: pprint: added support for mapping proxy http://bugs.python.org/issue23502 #23501: Argument Clinic: generate code into separate files by default http://bugs.python.org/issue23501 #23496: Steps for Android Native Build of Python 3.4.2 http://bugs.python.org/issue23496 #23492: Argument Clinic: improve generated parser for 1-argument funct http://bugs.python.org/issue23492 #23491: PEP 441 - Improving Python Zip Application Support http://bugs.python.org/issue23491 #23488: Random objects twice as big as necessary on 64-bit builds http://bugs.python.org/issue23488 Top 10 most discussed issues (10) ================================= #23491: PEP 441 - Improving Python Zip Application Support http://bugs.python.org/issue23491 26 msgs #23496: Steps for Android Native Build of Python 3.4.2 http://bugs.python.org/issue23496 24 msgs #23521: OverflowError from timedelta * float in datetime.py http://bugs.python.org/issue23521 17 msgs #23524: Use _set_thread_local_invalid_parameter_handler in posixmodule http://bugs.python.org/issue23524 14 msgs #23493: optimize sort_keys in json module by using operator.itemgetter http://bugs.python.org/issue23493 12 msgs #23517: datetime.utcfromtimestamp parses timestamps incorrectly http://bugs.python.org/issue23517 12 msgs #23492: Argument Clinic: improve generated parser for 1-argument funct http://bugs.python.org/issue23492 8 msgs #20323: Argument Clinic: docstring_prototype output causes build failu http://bugs.python.org/issue20323 7 msgs #23314: Disabling CRT asserts in debug build http://bugs.python.org/issue23314 7 msgs #23458: [2.7] random: make the file descriptor non-inheritable (on POS http://bugs.python.org/issue23458 7 msgs Issues closed (25) ================== #5700: io.FileIO calls flush() after file closed http://bugs.python.org/issue5700 closed by serhiy.storchaka #6639: turtle: _tkinter.TclError: invalid command name ".10170160" http://bugs.python.org/issue6639 closed by serhiy.storchaka #15955: bz2, lzma: add option to limit output size http://bugs.python.org/issue15955 closed by pitrou #20780: Shadowed (duplicate name but different body) test in test_stat http://bugs.python.org/issue20780 closed by benjamin.peterson #21934: OpenBSD has no /dev/full device http://bugs.python.org/issue21934 closed by serhiy.storchaka #22113: memoryview and struct.pack_into http://bugs.python.org/issue22113 closed by serhiy.storchaka #22435: socketserver.TCPSocket leaks socket to garbage collector if se http://bugs.python.org/issue22435 closed by berker.peksag #23152: fstat64 required on Windows http://bugs.python.org/issue23152 closed by steve.dower #23215: MemoryError with custom error handlers and multibyte codecs http://bugs.python.org/issue23215 closed by serhiy.storchaka #23245: urllib2: urlopen() gets exception(kwargs bug?) http://bugs.python.org/issue23245 closed by Douman #23374: pydoc 3.x raises UnicodeEncodeError on sqlite3 package http://bugs.python.org/issue23374 closed by serhiy.storchaka #23476: SSL cert verify fail for "www.verisign.com" http://bugs.python.org/issue23476 closed by pitrou #23489: atexit handlers are not executed when using multiprocessing.Po http://bugs.python.org/issue23489 closed by davin #23490: allocation (and overwrite) of a 0 byte buffer http://bugs.python.org/issue23490 closed by serhiy.storchaka #23497: textwrap.dedent doesn't work properly with embedded strings co http://bugs.python.org/issue23497 closed by ned.deily #23499: Minor grammar issue in mimetypes.rst http://bugs.python.org/issue23499 closed by ned.deily #23506: Data Structure of Dict not changing its value as expected http://bugs.python.org/issue23506 closed by zach.ware #23511: Broken code example in email module documentation http://bugs.python.org/issue23511 closed by berker.peksag #23515: Bad logic in timsort's merge_collapse http://bugs.python.org/issue23515 closed by python-dev #23518: Misplaced diagnostic caret for some SyntaxErrors http://bugs.python.org/issue23518 closed by zach.ware #23519: using asyncio.iscoroutinefunction() on a functools.partial obj http://bugs.python.org/issue23519 closed by gvanrossum #23523: cmath.atanh has wrong output http://bugs.python.org/issue23523 closed by benjamin.peterson #23533: MacOSX Python child process running urlopen crashed when tkMes http://bugs.python.org/issue23533 closed by ned.deily #23535: os.path.join() wrong concatenation of "C:" on Windows http://bugs.python.org/issue23535 closed by eric.smith #23537: BaseSubprocessTransport includes two unused methods http://bugs.python.org/issue23537 closed by haypo From guido at python.org Fri Feb 27 18:30:21 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 27 Feb 2015 09:30:21 -0800 Subject: [Python-Dev] [RELEASED] Python 3.4.3 is now available In-Reply-To: References: <54EDC8FC.5040003@hastings.org> Message-ID: There's a GitHub tracker for the website: https://github.com/python/pythondotorg/issues Please report issues with the website there. On Fri, Feb 27, 2015 at 5:59 AM, Victor Stinner wrote: > Hum, it's probably an attack of the Python 2 mafia who fight against > Python 3! > > Victor > > 2015-02-27 13:51 GMT+01:00 Geoffrey Spear : > > The download button (for Windows anyway) on the python.org Download > dropdown > > menu is broken; there's a small gray square where the 3.4.3 button should > > be, with the 2.7.9 one working fine. > > > > On Wed, Feb 25, 2015 at 8:07 AM, Larry Hastings > wrote: > >> > >> > >> > >> On behalf of the Python development community and the Python 3.4 release > >> team, I'm pleased to announce the availability of Python 3.4.3. Python > >> 3.4.3 has many bugfixes and other small improvements over 3.4.2. > >> > >> You can find it here: > >> > >> https://www.python.org/downloads/release/python-343/ > >> > >> > >> The release slipped by two or three days, depending on what time zone > >> you're in. This is my fault--I apologize for the inconvenience. > >> > >> Cheers, > >> > >> > >> /arry > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> > https://mail.python.org/mailman/options/python-dev/geoffspear%40gmail.com > >> > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Feb 27 20:38:26 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 27 Feb 2015 11:38:26 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) Message-ID: I think it's time to accept PEP 485. I've re-read it once more, and it looks like the text is in great shape. (My only recommendation would be to update the Abstract to state that we're specifically adding math.isclose().) A wording question: "This implementation has a flag that lets the user select which relative tolerance test to apply -- this PEP does not suggest that that be retained, but rather than the weak test be selected." -- I think this was meant to say "... rather *that* the weak test be selected", right? (It would be nice if the sample implementation defaulted to the choice in the PEP.) However, those are just minor edits, and I hereby approve the PEP. Thanks Chris and everyone else for the fruitful discussion (and thanks especially to Chris for eventually ending the bikeshedding and writing a PEP that explains each of the choices). Congrats! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From willingc at willingconsulting.com Fri Feb 27 19:51:50 2015 From: willingc at willingconsulting.com (Carol Willing) Date: Fri, 27 Feb 2015 10:51:50 -0800 Subject: [Python-Dev] [RELEASED] Python 3.4.3 is now available In-Reply-To: References: <54EDC8FC.5040003@hastings.org> Message-ID: <54F0BCC6.7060507@willingconsulting.com> Appears to be fixed now on the MacOSx tab. On 27/02/2015 09:30, Guido van Rossum wrote: > There's a GitHub tracker for the website: > https://github.com/python/pythondotorg/issues > > Please report issues with the website there. > > On Fri, Feb 27, 2015 at 5:59 AM, Victor Stinner > > wrote: > > Hum, it's probably an attack of the Python 2 mafia who fight > against Python 3! > > Victor > > 2015-02-27 13:51 GMT+01:00 Geoffrey Spear >: > > The download button (for Windows anyway) on the python.org > Download dropdown > > menu is broken; there's a small gray square where the 3.4.3 > button should > > be, with the 2.7.9 one working fine. > > > > On Wed, Feb 25, 2015 at 8:07 AM, Larry Hastings > > wrote: > >> > >> > >> > >> On behalf of the Python development community and the Python > 3.4 release > >> team, I'm pleased to announce the availability of Python > 3.4.3. Python > >> 3.4.3 has many bugfixes and other small improvements over 3.4.2. > >> > >> You can find it here: > >> > >> https://www.python.org/downloads/release/python-343/ > >> > >> > >> The release slipped by two or three days, depending on what > time zone > >> you're in. This is my fault--I apologize for the inconvenience. > >> > >> Cheers, > >> > >> > >> /arry > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> > https://mail.python.org/mailman/options/python-dev/geoffspear%40gmail.com > >> > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/willingc%40willingconsulting.com -- *Carol Willing* Developer | Willing Consulting https://willingconsulting.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Feb 27 21:07:48 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 27 Feb 2015 12:07:48 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: Thank you Guido. It'll be nice to see this all come to something. Thanks to all who contributed to the discussion -- despite this being a pretty simple function, I learned a lot and far more fully appreciate the nuance of all of this. I'll edit the text as you suggest, and then work on a patch -- I'm sure I'll have questions for Python-dev when I actually do that, but I'll get started on my own and see how far I get. -Chris On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum wrote: > I think it's time to accept PEP 485. I've re-read it once more, and it > looks like the text is in great shape. (My only recommendation would be to > update the Abstract to state that we're specifically adding math.isclose().) > > A wording question: "This implementation has a flag that lets the user > select which relative tolerance test to apply -- this PEP does not suggest > that that be retained, but rather than the weak test be selected." -- I > think this was meant to say "... rather *that* the weak test be selected", > right? (It would be nice if the sample implementation defaulted to the > choice in the PEP.) > > However, those are just minor edits, and I hereby approve the PEP. Thanks > Chris and everyone else for the fruitful discussion (and thanks especially > to Chris for eventually ending the bikeshedding and writing a PEP that > explains each of the choices). Congrats! > > -- > --Guido van Rossum (python.org/~guido) > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From gherron at digipen.edu Sat Feb 28 01:40:32 2015 From: gherron at digipen.edu (Gary Herron) Date: Fri, 27 Feb 2015 16:40:32 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: <54F10E80.6030605@digipen.edu> On 02/27/2015 12:07 PM, Chris Barker wrote: > Thank you Guido. > > It'll be nice to see this all come to something. > > Thanks to all who contributed to the discussion -- despite this being > a pretty simple function, I learned a lot and far more fully > appreciate the nuance of all of this. > > I'll edit the text as you suggest, and then work on a patch -- I'm > sure I'll have questions for Python-dev when I actually do that, but > I'll get started on my own and see how far I get. > > -Chris There's another typo: The "Large tolerances" section, references a "string test". Perhaps that should be "strong test", but that would seem to contradict the sentence which follows. -- Dr. Gary Herron Department of Computer Science DigiPen Institute of Technology (425) 895-4418 > > > > > On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum > wrote: > > I think it's time to accept PEP 485. I've re-read it once more, > and it looks like the text is in great shape. (My only > recommendation would be to update the Abstract to state that we're > specifically adding math.isclose().) > > A wording question: "This implementation has a flag that lets the > user select which relative tolerance test to apply -- this PEP > does not suggest that that be retained, but rather than the weak > test be selected." -- I think this was meant to say "... rather > *that* the weak test be selected", right? (It would be nice if the > sample implementation defaulted to the choice in the PEP.) > > However, those are just minor edits, and I hereby approve the PEP. > Thanks Chris and everyone else for the fruitful discussion (and > thanks especially to Chris for eventually ending the bikeshedding > and writing a PEP that explains each of the choices). Congrats! > > -- > --Guido van Rossum (python.org/~guido ) > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/gherron%40islandtraining.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 28 15:18:09 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Mar 2015 00:18:09 +1000 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: On 28 Feb 2015 06:16, "Chris Barker" wrote: > > Thank you Guido. > > It'll be nice to see this all come to something. > > Thanks to all who contributed to the discussion -- despite this being a pretty simple function, I learned a lot and far more fully appreciate the nuance of all of this. The simple enhancements are often the hardest to get consensus on, but also often the most beneficial in the long run. Thanks for pushing this one through to completion! Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: